JP2018005663A - Information processing unit, display system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing unit capable of guiding a user to a location for operating an operation target.SOLUTION: An information processing unit 30 acquires positional information on a position of a user from a device 10 for detecting a position of the user in a periphery of a display device. The information processing unit includes position display means 42 configured to convert the positional information to a display position in the display device, and to display user position display information representing the position of the user at the display position. The position display means displays, on the display device, guide position display information for guiding the user to a location for operating a content displayed on the display device according to a movement of the user.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a display system, and a program.

  There is known a system for displaying information on a display installed at a place where people gather or pass. An information processing device is connected to the display via a network or the like, and the information processing device displays useful information such as advertisements, weather, news, etc. by displaying still images and moving images on the display. it can. Such a system or mechanism is referred to as digital signage. The displays are installed outdoors, in stores, public spaces, transportation facilities, passages, and the like, and can provide detailed and new information to visitors, visitors, facility users (hereinafter simply referred to as users), and the like.

  By the way, the main function is to display information unilaterally especially in the large-screen digital signage among the current digital signage. However, when viewed from the user, the information changes from moment to moment, so even if you are interested in the displayed information, you may not be able to browse slowly. Further, when viewed from the information provider, it is impossible to grasp what information the user has shown to interest because no reaction can be obtained from the user simply by flowing the information. In addition, it is difficult even if the user wants to display more detailed information using the digital signage information as a window, or the provider wants to provide detailed information on the information that the user is interested in. In other words, interactive (bidirectional) operation is difficult with conventional digital signage.

  In order to realize interactive operation, an input device for the user to input operations needs to be prepared near the display. However, it is difficult to secure an installation space, and even if there is an installation space, the input device may be damaged. There is a risk of theft. Therefore, an operation mode in which a digital signage detects a user's gesture and receives an operation is studied (for example, see Patent Document 1). Patent Document 1 discloses a pointing control device that moves a cursor displayed on a display surface based on a detected user movement and adjusts the movement of the cursor based on a distance from the user.

  However, Patent Document 1 has a problem that the user cannot grasp where the user can operate the digital signage. This will be described below. Since digital signage is installed where many users come and go, only one person can see digital signage at the same time. In other words, digital signage is likely to be seen by multiple users at the same time, and if all users who can see digital signage can operate digital signage, it becomes difficult for each user to perform a desired operation. . In addition, if all users can operate, the user's unconscious operation that has just passed may interfere with the operation of the user who really wants to operate digital signage.

  For such inconvenience, a method in which the motion detection device detects the motion of only the user standing at a specific position is conceivable. If the specific position is not so wide, the number of users whose motion is detected by the motion detection device can be limited to a predetermined number (for example, one person).

  However, in order for a user to grasp a specific position using a physical index (marking such as tape or paint), an administrator of the digital signage needs to prepare the index separately. In addition, it is necessary to prepare an index in accordance with a position where the motion detection device can detect the user's motion, and setting takes time and cost.

  An object of the present invention is to provide an information processing apparatus capable of guiding a user to a place where an operation target is operated.

  The present invention is an information processing apparatus that acquires position information related to a user's position from a device that detects a position of a user around the display device, the position information being converted into a display position on the display device, and the display Position display means for displaying user position display information representing the position of the user at the position, the position display means at the position where the user operates the content displayed on the display device according to the operation of the user; Guidance position display information for guiding the user is displayed on the display device.

  An information processing apparatus capable of guiding a user to a place where an operation target is operated can be provided.

It is an example of the figure explaining control of the position of the display and cursor which an information processing system has. 1 is an example of a system configuration diagram of an information processing system. It is an example of the hardware block diagram of information processing apparatus. It is an example of the hardware block diagram of an operation | movement detection apparatus. It is a figure which shows the joint from which a three-dimensional coordinate is obtained. It is an example of the figure explaining the relationship between application software and a gesture operation reception program. It is an example of the functional block diagram in which the function of the gesture operation reception program was shown in the block shape. It is an example of the figure explaining the detection range and operation position of a person with respect to a motion detection apparatus. It is an example of the figure which shows a detection range in three dimensions. It is an example of the figure explaining an operation area. It is an example of the figure explaining an operation area in detail. It is an example of the figure explaining a user's non-operation operation | movement. It is an example of the figure which looked at the cube of FIG. 12 from the side. It is an example of the figure which shows the coordinate of a wrist joint or a wrist joint, and the position of a cursor in an operation area. It is the figure which looked at the three-dimensional coordinate of the wrist joint or wrist joint of the user U from the side. It is an example of the figure explaining the modification of an operation area. It is an example of the figure explaining the detection method of the speed of a wrist joint or a wrist joint. It is an example of the figure explaining the detection method of an event. It is an example of the figure explaining coordinate conversion in case speed is less than a threshold value. It is an example of the figure which shows the operation reception assistance icon when a user is detected by the motion detection apparatus. It is an example of the figure explaining the calculation method of the position of the person icon in an operation reception assistance icon. It is an example of the figure which shows the operation reception assistance icon when a user raises a hand in an operation position. It is an example of the figure which shows a guide mesh in case the user is raising the hand in the operation position, but the wrist joint or the speed of a wrist joint is less than a threshold value. It is an example of the flowchart figure which shows the procedure until a gesture operation reception program gives an operation right to a user. It is an example of the flowchart figure which shows the procedure in which a gesture operation reception program receives a user's operation and operates application software. It is a figure which shows an example of the sensor information which an information recording part records for analysis, such as face orientation. It is an example of the figure which shows the screenshot which the information recording part acquired when the sensor information of FIG. 26 was acquired. It is an example of the figure explaining the user who is not detected. It is an example of the figure explaining the example of analysis of the sensor information which the information recording part memorize | stored in sensor information DB.

  Hereinafter, an information processing system 100 for implementing the present invention and a position information creation method performed by the information processing system 100 will be described with reference to the drawings.

<Schematic features of information processing system>
The general features of the information processing system 100 will be described with reference to FIG. FIG. 1 is an example of a diagram for explaining user guidance to the operation position 52.

  As shown in FIG. 1A, the display 310 displays a content list 101 and a moving image 102 as digital signage, and various operation buttons 103 (play, stop, etc. of the moving image 102) for operation input. It is displayed. In addition, the motion detection device 10 is installed below the display 310. The motion detection device 10 detects a user who has entered a human detection range 51 within a certain angular range and within a certain distance. Further, the motion detection device 10 detects the user's standing position and motion at the operation position 52.

  The display 310 always displays an operation reception support icon 81. Alternatively, it may be displayed only when the motion detection device 10 detects a user. On the operation reception support icon 81, an operation position instruction display 83 described later is displayed. When the motion detection apparatus 10 detects the user U, the information processing apparatus 30 displays a person icon 82 representing the user U on the operation reception support icon 81. The person icon 82 displays the person icon 82 at a position in the operation reception support icon 81 corresponding to the position of the user in the detection range 51.

  Therefore, the user watching the display 310 can grasp that he / she has been detected. Further, it can be understood that the person icon 82 moves when the user moves.

  In order to operate the digital signage, the user knows from the guidelines that the person icon 82 must be moved to the operation position instruction display 83. The user moves within the detection range 51 so that the person icon 82 whose position changes with his / her movement enters the operation position instruction display 83. When the person icon 82 is present on the operation position instruction display 83, the user is present at the operation position 52.

  In this way, the user moves so that the person icon 82 moves to the operation position instruction display 83, so that the user can naturally move to the operation position 52 where the motion detection device 10 detects the user's motion. Therefore, the information processing apparatus 30 can guide the user to the operation position 52 by detecting the position of the user and converting the position into a position on the operation reception support icon 81 and displaying it.

  Therefore, the information processing system according to the present embodiment allows the user to operate the digital signage wherever the user moves without the physical sign (marking such as tape or paint) being applied to the operation position 52 by the digital signage administrator or the like. It is possible to make the user grasp this. However, when there is a physical index, the user can easily grasp the operation position 52.

<Terminology>
The position information related to the user's position includes information for specifying the user's position, information representing the user's position, and the like. For example, the relative position with respect to the motion detection device 10 may be referred to as the relative position with respect to the display device if the position of the motion detection device 10 and the display device is fixed. The position information may be an absolute position including coordinates.

  The display position in the display device refers to a position on a display device such as a display. Specifically, the display position is indicated by a cursor or the like.

  The content displayed on the display device refers to all information that is displayed and can be visually judged. If audio flows with the content, the audio is also included in the content. In this embodiment, the term “content” will be used.

  The user position display information is display information indicating the user or the position of the user on the display device. In the present embodiment, the term human icon 82 will be used.

  The guidance position display information is display information for guiding the user to a predetermined position on the display device. Or it is display information showing a user's desirable position. In this embodiment, the term “operation position instruction display 83” is used.

  The operation information is information representing the user's operation. Or it is the information converted into the form which a user's operation | movement can interpret into an information processing apparatus. Or it is the information for inputting a user's operation into an information processor by non-contact. Specifically, the position, posture, movement of limbs, and the like are described in the present embodiment in terms of sensor information.

  The condition when the predetermined speed condition is met refers to a condition for detecting the user's intention regarding the control of the cursor position. This intention is an intention to control the position with high accuracy. The case where the conditions are met includes, for example, that the speed is less than a threshold and the speed is within a predetermined range. In the following, the case where the speed is less than the threshold value will be described as an example.

<System configuration example>
FIG. 2 is an example of a system configuration diagram of the information processing system 100. As illustrated in FIG. 2A, the information processing system 100 includes a display 310, an operation detection device 10, and an information processing device 30. The motion detection device 10 is connected to the information processing device 30 by a cable such as a USB cable or IEEE1394, and the information processing device 30 and the display 310 are DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (registered trademark, High. -It is connected via a cable that can communicate with standards such as Definition Multimedia Interface (VGA) and Video Graphics Array (VGA). Note that the motion detection device 10 and the information processing device 30 may communicate with each other wirelessly such as a wireless LAN, Bluetooth (registered trademark), or ZigBee (registered trademark). Similarly, the information processing apparatus 30 and the display 310 may communicate wirelessly such as a wireless LAN or HDMI wireless standard.

  The information processing apparatus 30 can use general-purpose devices such as a PC (Personal Computer), a tablet terminal, a smartphone, a mobile phone, and a PDA (Personal Digital Assistant). An OS (Operating System) operates in the information processing apparatus 30 and has a function of detecting the position of a pointing device such as a mouse or a touch panel and displaying the cursor 3 at this position. Further, the OS detects an event (left click, right click, double click, mouse over, mouse down, mouse up, etc.) performed by the pointing device. The information processing apparatus 30 operates various application software on the OS. Since the OS notifies the application software of the position and event of the pointing device, the application software operates based on the event performed at the position of the pointing device.

  In the present embodiment, the motion detection apparatus 10 and a gesture operation reception program described later operate as a pointing device. The gesture operation reception program is a program that acquires sensor information from the motion detection device 10 and converts it to the position of the cursor 3 or detects an event, and operates on the information processing device 30. The gesture operation reception program 110 notifies the application software 120 of the position of the cursor 3 and the event, so that the application software 120 is not aware of the pointing device (no need to change due to the pointing device being the motion detection device 10). ) Can work.

  The display 310 is a liquid crystal display device, a display device using an organic EL as a light emission principle, a display device using plasma light emission, or the like. The application software 120 creates a screen called CUI (Character User Interface) or GUI (Graphical User Interface) and displays it on the display 310.

  As the display 310, any device provided with the display 310 may be used. For example, a display provided in an electronic blackboard, a notebook PC, or the like can be used as the display 310.

  The motion detection device 10 is a sensor that detects a human motion. For example, when a space is imaged with a stereo camera, parallax is obtained for each pixel of the image data, and distance information for each pixel is obtained. In addition, it is possible to detect a person and recognize a person's movement by recognizing the pattern of the image data. Specifically, three-dimensional coordinates such as the palm of the person, wrist, and shoulder, and posture can be obtained. Kinect (registered trademark, hereinafter abbreviated as a registered trademark) is known as a sensor suitable for detecting such human movements. Details of Kinect will be described later. Note that the motion detection apparatus 10 may not be a kinetic, but may be a sensor having a function similar to that of a kinetic, a stereo camera, a camera that detects a distance using TOF (Time Of Flight), or the like.

  Note that the installation position of the motion detection device 10 in FIG. 2A is an example, and may be installed on the upper side, the left side, or the right side of the display 310. Further, the installation position in the information processing apparatus 30 is merely an example, and may be installed behind the display 310, for example. Moreover, you may connect with the operation | movement detection apparatus 10 via the network.

  As illustrated in FIG. 2B, the information processing system 100 may include a projector 5 instead of the display 310. In this case, the information processing system 100 includes the motion detection device 10, the projector 5, and the information processing device 30. The projector 5 is a display device that projects an image on a projection surface 6 such as a screen by an LCD (transmission type liquid crystal) method, an LCOS (reflection type liquid crystal) method, or a DLP (Digital Light Processing) method.

  The projector 5 and the information processing apparatus 30 are connected in the same manner as the connection mode between the display 310 and the information processing apparatus 30. The projector 5 is preferably capable of projecting from a so-called polar single focus. Thereby, it becomes easy to secure the space where the user U operates. The motion detection device 10 may be closer to the user U than the projector 5 or may be closer to the screen. When it is on the screen side, it may be fixed to the lower side, upper side, left side or right side of the screen. Moreover, when it exists in the user U side, it installs on the prepared stand. The motion detection device 10 is installed at the right end of the display 310 as an example, and may be installed at the left end. It is installed at the right end or the left end in order to avoid the waste heat of the projector 5. If the waste heat can be ignored, it may be installed at the center of the display 310.

  Note that the display 310 and the projector 5 may be used in combination. In this case, for example, when the projector 5 projects an arrow or the like toward the floor, the user can be guided to the operation position 52. In addition, an arrow can be displayed to indicate the traveling direction (for example, the elevator is over and the operation position 52 is over).

<About hardware configuration>
<< Hardware configuration of information processing device 30 >>
FIG. 3 is an example of a hardware configuration diagram of the information processing apparatus 30. The information processing apparatus 30 includes a CPU 301, a ROM 302, a RAM 303, and an auxiliary storage device 304. The private use server further includes an input unit 305, a display control unit 306, a network I / F 307, and an external device I / F 308. Each unit of the information processing apparatus 30 is connected to each other via the bus B.

  The CPU 301 executes various programs, OS, etc. stored in the auxiliary storage device 304. The ROM 302 is the nonvolatile memory 16. The ROM 302 stores programs, data, and the like necessary for the CPU 301 to execute various programs stored in the auxiliary storage device 304.

  The RAM 303 is a main storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). Various programs stored in the auxiliary storage device 304 when executed by the CPU 301 are expanded in the RAM 303, and the RAM 303 becomes a work area of the CPU 301.

  The auxiliary storage device 304 stores various programs executed by the CPU 301 and various databases used when the various programs are executed by the CPU 301. The auxiliary storage device 304 is a nonvolatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

  The input unit 305 is an interface for an operator to input various instructions to the information processing apparatus 30. For example, a keyboard, a mouse, a touch panel, a voice input device, and the like may be used, but may not be connected in this embodiment.

  In response to a request from the CPU 301, the display control unit 306 displays various information included in the information processing apparatus 30 on the display 310 in the form of a cursor 3, a menu, a window, characters, an image, or the like. The display control unit 306 is, for example, a graphic chip or a display 310 I / F.

  The network I / F 307 is a communication device that communicates with other devices via a network. The network I / F 307 is, for example, an Ethernet (registered trademark) card, but is not limited thereto.

  The external device I / F 308 is an interface for connecting various storage media 320 such as a USB cable or a USB memory. In the present embodiment, the motion detection device 10 is connected.

  Note that the information processing apparatus 30 is preferably compatible with so-called cloud computing, and the information processing apparatus 30 does not need to be housed in a single housing or provided as a single device. In this case, the configuration of the information processing apparatus 30 is configured by dynamically connecting / disconnecting hardware resources according to the load.

<< Hardware configuration of motion detection device 10 >>
FIG. 4 is an example of a hardware configuration diagram of the motion detection apparatus 10. The motion detection device 10 includes a CPU 11, a color camera 12, an infrared camera 13, an infrared projector 14, a main storage device 15, a nonvolatile memory 16, a microphone 17, an acceleration sensor 18, and an LED 19. The CPU controls the entire operation detection device 10 by expanding and executing the firmware stored in the nonvolatile memory 16 in the main storage device 15. The main storage device 15 is a high-speed volatile memory such as a DRAM, and the nonvolatile memory 16 is a flash memory or the like.

  As an example, the color camera 12 is an imaging device that captures a color image with a resolution of 1920 × 1080. That is, R, G, and B density information is obtained for each pixel. The infrared camera 13 and the infrared projector 14 function as one depth sensor (distance sensor). The infrared projector 14 projects pulse-modulated infrared light and images the infrared light projected by the infrared camera 13. The depth sensor measures the time it returns after reflecting infrared light, and obtains distance information for each projected infrared light. Note that the resolution of the distance information is 512 × 424, but is merely an example.

  In addition, the principle that the depth sensor measures the distance is merely an example, and the projected infrared pattern may be captured by the infrared camera 13 to acquire distance information from the distortion of the pattern.

  The microphone 17 is a device that detects surrounding sounds, and the CPU 11 estimates the direction of the sound source by comparing the phases of the sounds detected by the plurality of microphones 17. The acceleration sensor 18 is a sensor that detects the inclination of the motion detection device 10 with respect to the horizontal. The LED 19 displays the state of the motion detection device 10 by color or blinking.

<Information acquired by the motion detection device 10>
Next, information acquired by the motion detection device 10 will be described. Any information acquired by the motion detection device 10 can be used by the information processing device 30. However, it is not necessary to use all the information described below. Information acquired by the motion detection device 10 described below is referred to as “sensor information”.

1. Color image Image data including density information of R, G, and B for each pixel. The infrared camera 13 of the motion detection apparatus 10 captures a color image at a predetermined frame rate (for example, 30 fps).

2. Distance image This is image data in which distance information is obtained at a resolution of 512 × 424 as described above. The depth sensor of the motion detection device 10 captures a distance image at a predetermined frame rate (for example, 30 fps). Therefore, the distance to the target including the user U is obtained with a predetermined resolution.

3. Infrared image A gray-scale image captured by the infrared camera 13. The pulse projected from the infrared projector 14 can be imaged. Further, it is possible to detect a person (user U) that emits infrared rays.

4). The person shape data motion detection device 10 can create person shape data by extracting only a person region from a distance image. For example, a human imaging region is specified from an infrared image, or a region whose distance changes according to a frame is specified as a human imaging region. The shape data of the person is obtained by extracting the imaging region from the distance image. Therefore, the information processing apparatus 30 knows the pixel area where the user U is present. In addition, when a plurality of users U are detected, the motion detection device 10 detects the user U and the pixel area in association with each other. The information processing apparatus 30 can also cut out only a person area from a color image.

5. It is the coordinate information of the skeletal information joint. Sometimes called bone information or skeleton information. FIG. 5 shows a joint from which three-dimensional coordinates can be obtained. In addition, in FIG. 5, it is the figure which looked at the joint of the user from the front. The symbols and joint names in FIG. 5 will be described. In FIG. 5, the left half joint is the same as the right side, so the reference numerals are omitted.
a ... head joint, b ... neck joint, c ... shoulder center joint, d ... right shoulder joint, e ... right elbow joint, f ... right wrist joint, g ... right wrist joint, h ... right wrist joint, i ... right thumb joint , J: spine joint, k: central hip joint, l: right hip joint, m: right knee joint, n: right heel joint, o: right foot joint Note that the motion detection device 10 is a three-dimensional coordinate in the space of each joint. In addition to (X, Y, Z), the coordinates (x, y) of the joint on the color image (on the display 310) are provided. This is because the pixel area where the user U is present is clear. Further, since the three-dimensional coordinates of the “right hand joint”, “right thumb joint”, “left hand joint”, and “left thumb joint” are known, the motion detection device 10 can determine whether the user U has opened or closed the palm.

6). Expression An expression of the user U based on face recognition. The motion detection device 10 is usually laughing, surprised, with left / right eyes closed or open, mouth open / closed, looking away from the screen, wearing glasses , Etc.

7). Face orientation The orientation of the face. The motion detection device 10 can detect the yaw angle, pitch angle, and roll angle.

8). Heart rate The estimated heart rate of the user U. The skin is estimated with a color image and an infrared image, and estimated from subtle changes in the skin due to blood flow.

<About the function of the information processing apparatus 30>
First, the relationship between the application software 120 and the gesture operation reception program 110 will be described with reference to FIG. Both the application software 120 and the gesture operation reception program 110 operate on the OS 130. The application software 120 only needs to be operable by position information or an event. For example, there are application software 120 for displaying advertisements, application software 120 for presentations, and the like. However, the present invention is not limited to these, and for example, a browser, a word processor, a game, and the like may be included.

  A device driver 140 of the motion detection apparatus 10 is installed in the information processing apparatus 30. The device driver 140 is software that processes a signal detected by the motion detection apparatus 10 into sensor information. The device driver 140 may be called an SDK (Software Development Kit). Arbitrary application software 120 can acquire sensor information created by the device driver 140 via an API (Application Interface) of the OS 130. In this embodiment, the gesture operation reception program 110 uses this API to acquire sensor information used for processing.

  The gesture operation reception program 110 acquires sensor information from the motion detection device 10 and performs various processes to control the gesture operation reception program 110 itself, and the application software 120 indicates the position and event of the cursor 3 for operation. Send. The application software 120 has a function of detecting the position and event of the cursor 3 input by operating a pointing device such as a mouse or a touch panel. In general, the OS 130 detects the position and event of the pointing device and provides them to the application software 120. In this embodiment, since a general-purpose pointing device is not used (not connected to the information processing apparatus 30), the gesture operation reception program 110 provides the application software 120 with the position of the cursor 3 and the event. The gesture operation reception program 110 can provide the position of the cursor 3 and the event to a plurality of application software 120.

  The providing method may follow the specifications of the OS 130 and the application software 120. For example, inter-process communication between the application software 120 provided by the OS 130 and the gesture operation reception program 110 may be used, or the gesture operation reception program 110 notifies the OS 130 of the position of the cursor 3 and the event, The method that 120 acquires may be used. With such a mechanism, it is not necessary for the application software 120 to change the acquisition of the position of the cursor 3 and the event from the gesture operation reception program 110, and the gesture operation reception program 110 is used together with the general-purpose application software 120. Can do.

  The gesture operation reception program 110 is one of application software, but may be implemented as a device driver.

<< Function of Gesture Operation Reception Program 110 >>
FIG. 7 is an example of a functional block diagram showing the functions of the gesture operation reception program 110 in a block form. The gesture operation reception program 110 includes an information acquisition unit 31, a person detection unit 32, an operation start detection unit 33, an operation area creation unit 34, a coordinate conversion unit 35, a UI control unit 36, a speed detection unit 37, an event determination unit 38, An information recording unit 43 and an output unit 39 are included. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 3 operating according to a command from the CPU 301 according to the program 304p expanded from the auxiliary storage device 304 to the RAM 303. . The program 304p includes an OS 130, a device driver 140, application software 120, and a gesture operation reception program 110.

  The information processing apparatus 30 has a sensor information DB 44 (database), and the information recording unit 43 records sensor information in the sensor information DB 44. The sensor information DB 44 is constructed in the auxiliary storage device 304 and the RAM 303 shown in FIG.

The information acquisition unit 31 is realized by the CPU 301 illustrated in FIG. 3 executing the gesture operation reception program 110 and the like, and acquires sensor information from the device driver 140. Since the motion detection device 10 periodically detects motion, the information acquisition unit 31 periodically acquires sensor information. Note that it is not necessary to acquire all sensor information, and only sensor information used for processing needs to be acquired. In the present embodiment, the following information is used.
The three-dimensional coordinates of the right shoulder joint d, the right wrist joint f, the right hand joint g, the right hand joint h, and the right thumb joint i are used. Since the user can operate with either the right hand or the left hand, the three-dimensional coordinates of the left shoulder joint, the left wrist joint, the left hand joint, the left hand joint, and the left thumb joint are also used.
3D coordinates for each user U (mainly used to display the person icon 82)
Face orientation (used to display the eyes of the person icon 82)
The human detection unit 32 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and the like, and the motion detection device 10 detects that a person has been detected based on the sensor information acquired by the information acquisition unit 31. To detect. It may be detected that the motion detection device 10 has detected a person or that the number of people has been detected, or if the motion detection device 10 has detected a person, human shape data or skeleton information can be obtained. It may be detected that a person has been detected. The human detection range 51 is determined by the specification of the motion detection device 10. For example, the horizontal angle is 70 degrees and the distance is in the range of 0.5 to 4.5 m. A person may be detected only in a narrower range than the specification of the motion detection device 10, and if the specification of the motion detection device 10 is changed, the range in which the person is detected can be changed. The number of people that can be detected at the same time is six, but this can be changed if the specifications of the motion detection device 10 change.

  The operation start detection unit 33 is realized by the CPU 301 illustrated in FIG. 3 executing the gesture operation reception program 110, and the user U intends to operate the application software 120 based on the sensor information acquired by the information acquisition unit 31. Is detected. In the present embodiment, the user U detects the intention to operate by “existing at the operation position 52 + raising a hand”. Details will be described later. In addition, the operation start detection unit 33 detects that the user U has indicated the intention to end the operation of the application software 120 based on the sensor information. In the present embodiment, the intention of ending the operation is detected by the user U “leaving the operation position 52” or “lowering his / her hand”.

  The operation area creation unit 34 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like. When the operation start detection unit 33 detects the intention of the operation, the user U performs an operation. An operation area 54 is created. The operation area 54 is a range (area) in the three-dimensional space where the user U can operate the information processing apparatus 30. If the right hand joint g or right wrist joint f of the user U is in the operation area 54, the three-dimensional coordinates of the right hand joint g or right wrist joint f are reflected in the position of the cursor 3, or a gesture is detected.

  The coordinate conversion unit 35 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 and the like, and based on the sensor information acquired by the information acquisition unit 31, the right joint g of the user U in the operation area 54 The three-dimensional coordinates of the right wrist joint f are converted into the position of the cursor 3.

  The speed detection unit 37 is realized by the CPU 301 illustrated in FIG. 3 executing the gesture operation reception program 110, and the right joint g of the user U in the operation area 54 based on the sensor information acquired by the information acquisition unit 31. And the speed of the right wrist joint f is detected. The right hand joint g and the right wrist joint f may detect either speed, but the user U operates around the elbow, so the joint close to the hand can move the cursor 3 faster. Conversely, a joint far from the hand can easily control a small amount of movement.

  The event determination unit 38 is realized by, for example, the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and based on the sensor information acquired by the information acquisition unit 31, whether or not the user U has generated an event is determined. to decide. Events are left click, right click, double click, mouse over, mouse down, mouse up, etc. as described above, but general gestures corresponding to these events are not defined. For this reason, the event determination unit 38 associates the movement of the user U with an event as a gesture in advance, and determines that the associated event has occurred when the user U performs a predetermined gesture. For example, when the user U performs a gesture of holding a hand in the operation area 54, the event determination unit 38 determines that the click is a left click.

  The information recording unit 43 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation receiving program 110, and stores sensor information in the sensor information DB 44 in time series for each user. As described above, since the motion detection apparatus 10 detects the user's position and the user's facial expression and face orientation, it records whether the user is looking at the display 310 to such an extent that the user can estimate which part of the display 310 the user is looking at. it can. In addition, the information recording unit 43 acquires a screen shot of the content displayed on the display 310 from the application software 120. By making the timing for recording the sensor information and the timing for acquiring the screen shot substantially the same, it becomes easier for the content provider to identify the content that the user is interested in.

  The output unit 39 is realized by, for example, the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and acquires the position of the cursor 3 converted by the coordinate conversion unit 35 and the event determined by the event determination unit 38. And output to the application software 120. Note that the output unit 39 also has a function of acquiring the status of the application software 120. For example, when the application software 120 displays a moving image on the entire screen, the user does not need to operate, so the information acquisition unit 31 does not acquire sensor information. In addition, when the user is currently operating, the output unit 39 notifies the application software 120 to that effect. As a result, the application software 120 does not shift to the full-screen video display during the user's operation, so that the user can be prevented from suddenly becoming unable to operate.

  The UI control unit 36 is realized by the CPU 301 illustrated in FIG. 3 executing the gesture operation reception program 110 or the like. The UI control unit 36 further includes a cursor display unit 41 and an icon display unit 42. The cursor display unit 41 displays the cursor 3 using the function of the OS 130, or displays the guide mesh 4 instead of the normal cursor 3. Further, the icon display unit 42 displays an operation reception support icon 81 on the display 310. The icon display unit 42 changes the display mode of the operation reception support icon 81 according to the state of the user U. The operation reception support icon 81 functions as a user interface that supports the user U so that the user U can easily operate the application software 120 by a gesture. For example, the user is guided to the operation position 52 so that the motion detection apparatus 10 can acquire all the skeleton information of the user. Further, the user who has the operation right is clearly indicated (the person icon 82 who has raised his hand has the operation right even if the plurality of person icons 82 are displayed). The operation right is a right to operate the application software 120, and only one person is allowed at the same time.

<Human detection range 51 and operation position 52>
The human detection range 51 and the operation position 52 will be described with reference to FIG. FIG. 8 is an example of a diagram illustrating a human detection range 51 and an operation position 52 for the motion detection device 10. As described above, the motion detection device 10 detects a person within a predetermined distance (for example, 0.5 to 4.5 m) within a predetermined horizontal angle range (for example, 70 degrees). Naturally, the operation position 52 is within the human detection range 51.

  The operation position 52 is, for example, near the center of the human detection range 51, but may be anywhere inside the human detection range 51. The reason why the operation position 52 is determined is to prevent any user U passing the human detection range 51 from raising his hand so that the application software 120 cannot be operated. Further, if the operation position 52 is too wide, the number of users U who can acquire the operation right increases, and it is difficult to determine whether or not the user can operate. Further, since the operation position 52 is determined, the user U can grasp that the application software 120 can be operated by moving to the operation position 52. It is preferable that the actual operation position 52 is clearly displayed in a color different from other places, or characters such as “Please operate here” are displayed.

  As will be described later, the user moves so that the person icon 82 enters the operation position instruction display 83 of the operation reception support icon 81. When the person icon 82 enters the operation position instruction display 83, the user is present at the operation position 52. Therefore, the user is guided to the operation position 52 by the person icon 82, the operation reception support icon 81, and the operation position instruction display 83.

  However, for example, a case where the user moves out of the operation position 52 during operation is also assumed. For example, the display 310 may be approached to look closely.

  FIG. 9 is an example of a diagram illustrating the detection range 51 in a three-dimensional manner. In FIG. 9A, the detection range 51 includes all the skeleton information of the user. Therefore, the motion detection device 10 can acquire all the skeleton information and send it to the gesture operation reception program 110. In other words, the user can operate the digital signage by operation.

  On the other hand, in FIG. 9B, a part of the user's body protrudes from the detection range 51. The motion detection apparatus 10 sends only the skeleton information that has been acquired to the gesture operation reception program 110. For example, each function (the operation start detection unit 33, the operation area creation unit 34, the coordinate conversion unit 35, the UI control unit 36, the speed detection unit 37, and the event determination unit 38) of the gesture operation reception program 110 uses the skeleton information of the hand. If used but not included in the sensor information, the skeleton information cannot be acquired. For this reason, it can be detected that desired skeleton information is not acquired.

  In such a case, the UI control unit 36 can move the user to the operation position 52 by displaying the message 540 on the display 310 as shown in FIG. 9C. In FIG. 9C, a message 540 “Please away from the screen” is displayed. Therefore, the UI control unit 36 can display the guidance and guide the user into the detection range 51 when the user's position is shifted so that important (necessary control) skeleton information is lost.

  FIG. 9 shows an example in which the user has approached the display 310 too much. However, when at least a part of the user's body including the joint protrudes from the detection range 51, the gesture operation reception program 110 is included in the detection range 51. User can be guided.

<Detection of operation intention>
The operation will detection and the operation area 54 will be described with reference to FIG. FIG. 10 is an example of a diagram for explaining the detection of the will of operation and the operation area 54. FIG. 10A shows an initial operation performed by the user U at the operation position 52. When the user U reaches the operation position 52, the user U first raises his hand. The user U already knows whether or not it is instructed to perform this operation. The intention to start the operation is detected because an operation unintended by the user is sent to the application software 120 when the operation is accepted from the movement of an arbitrary hand of the user U within the detection range 51.

  The operation start detection unit 33 detects the intention to start the operation when the right hand joint g becomes higher than the height of the head joint a. Note that the user U's hand may be right or left. The operation start detection unit 33 preferentially detects the right hand, ignores even if the left hand is raised after raising the right hand, and ignores even if the right hand is raised after raising the left hand.

  In addition, the reason that the operation start detection unit 33 intends to start the operation is that the right hand joint g is higher than the height of the head joint a. The user U generally does not extend his hand more than the head. Because. For example, when talking on a mobile phone or touching glasses, the hand remains lower than the head. For this reason, it becomes easy to suppress erroneous detection that there is an intention to operate due to an action of a person who does not intend to operate the application software 120.

<Operation area 54>
The operation area creation unit 34 creates an operation area 54 using the lengths of the right shoulder joint d and the head joint a (hereinafter referred to as A). As shown in FIG. 10B, a circle 53 having a length A as a diameter and a center on the right shoulder is created. Then, a circumscribed rectangle of the circle 53 is created as shown in FIG. This circumscribed rectangle is the operation area 54. The length A is about 30 cm, but it is only an example and may be a length that can be reached by the user U.

  In this manner, by creating the operation area 54 from the circle 53 centered above the shoulder, it becomes easier to image the user U's hand from the motion detection device 10. Although it depends on the height at which the motion detection device 10 is installed, when the motion detection device 10 is installed at the waist level of the user U, the motion detection device 10 is placed on the shoulder when the hand is as high as the shoulder. Cannot be captured. For this reason, the error of the length of the right shoulder joint d and the head joint a becomes large (the length A becomes short or large), and the shape of the operation area 54 becomes unstable.

FIG. 11 is an example of a diagram illustrating the operation area 54 in detail. The three-dimensional coordinate of the head joint a is P1 (X1, Y1, Z1), and the three-dimensional coordinate of the right shoulder joint d is P2 (X2, Y2, Z2). The horizontal direction is the X axis, the height direction is the Y axis, and the vertical direction of the paper is the Z axis. The length A is calculated as follows.
A = Y1-Y2
The coordinates of the center O of the circle 53 are (X2, Y2 + A / 2). Therefore, the operation area creation unit 34 may draw a circle 53 having a radius A / 2 with the center O as the center. Further, without obtaining the circle 53, the coordinates of the four vertices C1 to C4 of the operation area 54 can be calculated as follows. Although the Z coordinate is omitted, for example, Z1, Z2, or an average thereof is used.
C1 = (X2-A / 2, Y2 + A)
C2 = (X2 + A / 2, Y2 + A)
C3 = (X2 + A / 2, Y2)
C4 = (X2-A / 2, Y2)
By creating the operation area 54 in this way, the user U can easily control the position of the cursor 3. Since the user U keeps his hand up (when the user U is lowered, he loses the right to operate), and the user U inevitably moves based on the elbow. Since the position of the shoulder is substantially fixed, the operation area 54 created above the shoulder can be operated around the elbow. Therefore, the user U can operate the cursor 3 by moving the right wrist joint f and the right wrist joint g around the elbow without searching for where the operation area 54 is.

<Non-operating action without operation right>
When a user talks on a mobile phone or touches glasses, the user often raises his hand above the head. For this reason, it is more effective that the operation start detection unit 33 detects the non-operation operation of the user as follows.

  FIG. 12 is an example of a diagram illustrating a non-operation operation of the user. The non-operation operation is an operation similar to an operation (for example, raising a hand) indicating a user's intention to operate the digital signage. For example, an operation in which a user makes a call on a mobile phone, an operation of touching glasses, an operation of placing a hand on the forehead, an operation of stroking the head, and the like.

  As shown in FIG. 12A, the operation start detection unit 33 defines a cube 55 centered on the head joint a. This definition may be referred to as setting, and specifically means that the coordinates of the cube 55 are specified. The dimensions of the cube 55 are slightly larger than those of the head and the face, and are large enough to allow the user's hand to enter when touching the face. It may be variable for each user or may be fixed. Moreover, a rectangular parallelepiped, a spherical body, or an ellipsoid may be sufficient instead of a cube.

  When the user performs a non-operational operation, the user's hand often enters the cube. Therefore, the operation start detection unit 33 can determine whether or not the operation is a non-operation operation based on whether or not the hand enters the cube 55.

  As illustrated in FIG. 12B, when the right hand joint g does not enter the cube 55, the operation start detection unit 33 determines that it is not a non-operation operation. In this case, since the hand of the user U is raised, the operation start detection unit 33 gives the operation right.

  As illustrated in FIG. 12C, when the right hand joint g enters the cube 55, the operation start detection unit 33 determines that it is a non-operation operation. In this case, the operation start detection unit 33 does not grant the operation right because the hand is in the cube 55 even if the hand is raised.

  Therefore, it is possible to suppress erroneous detection that the user U who does not intend to operate is willing to operate.

  FIG. 13 is an example of a view of the cube 55 of FIG. 12 viewed from the side. Since it is the cube 55, the cube 55 has a depth in the distance direction (Z direction) with respect to the motion detection apparatus 10. Since the hand of the user U1 is not in the cube 55, it is determined that it is not a non-operation operation, and since the hand of the user U2 is in the cube 55, it is determined that it is a non-operation operation.

  As described above, since the determination is made with the cube 55, when the user raises his / her hand so as not to enter the depth of the cube 55, the operation start detection unit 33 gives the user the operation right, and the coordinate conversion unit 35 receives the right hand joint g or the like. Can be detected. On the other hand, when the user raises his / her hand so as to enter the depth of the cube 55, even if the user raises his / her hand, the operation start detection unit 33 does not allow the user to operate. Therefore, rather than simply determining whether or not there is a hand near the face on the XY plane, it is easier to detect the non-operation operation and the operation indicating the intention to operate by determining with the cube 55.

<Coordinate transformation>
Next, conversion from the wrist joint or wrist joint coordinates in the operation area 54 to the position of the cursor 3 will be described with reference to FIG. FIG. 14 is an example of a diagram showing the wrist joint or wrist joint coordinates and the position of the cursor 3 in the operation area 54. FIG. 14 illustrates a method for determining the position of the cursor 3 when the speed of the hand is equal to or greater than a threshold value.

  The coordinate conversion unit 35 associates the wrist joint or the three-dimensional coordinates of the wrist joint in the operation area 54 with the position of the cursor 3 on the display 310 on a one-to-one basis. First, the coordinate conversion unit 35 determines the vertex C1 of the upper left corner of the operation area 54 as the origin (0, 0). Then, the three-dimensional coordinates of the wrist joint or the wrist joint in the operation area 54 are converted into three-dimensional coordinates based on the origin (0, 0). Since the operation area 54 is a two-dimensional plane of the XY plane, the wrist joint or the Z coordinate of the wrist joint is not used. For this reason, the Z coordinate may always be regarded as constant and may be omitted from the calculation. Let R (X, Y) be the three-dimensional coordinate of the wrist joint or wrist joint with C1 as the origin.

Also, assume that the vertical resolution of the display 310 is H and the horizontal resolution is W. Further, the vertical and horizontal lengths of the operation area 54 are A. Therefore, the position S (x, y) of the cursor 3 on the display 310 can be calculated by simple proportional calculation.
x = W × A / X
y = H × A / Y
Of course, the wrist joint or wrist joint of the user U protrudes outside the operation area 54, but the coordinate conversion unit 35 ignores the wrist joint or wrist joint outside the operation area 54.

  The coordinate conversion unit 35 performs coordinate conversion every time the information acquisition unit 31 acquires sensor information. On the other hand, the operation area creation unit 34 updates the operation area 54 for each frame of the image data. Thereby, even if a user changes a motion by facing the front or a shoulder toward the motion detection device 10, an appropriate operation area 54 can be created.

  The operation area 54 is deleted when the operation start detection unit 33 detects the intention to end the operation (when the user U lowers his / her hand).

  FIG. 15 is a diagram of the three-dimensional coordinates of the wrist joint or wrist joint of the user U viewed from the side. That is, the horizontal direction in FIG. 15 corresponds to the Z axis. The user U1 in FIG. 15 moves his arm near his body, and the user U2 moves his arm in front of his body. That is, the Z coordinates of the wrist joints or wrist joints of the users U1 and U2 are different. However, the Y-coordinates of the wrist joints or wrist joints of the users U1 and U2 are substantially the same (although not apparent from FIG. 15, the X-coordinates are also substantially the same).

  Thus, even if the Z coordinates are different, the coordinate conversion unit 35 does not use the Z coordinates when converting the wrist joint or the wrist joint coordinates to the position of the cursor 3, so the position of the cursor 3 operated by the users U1 and U2 The (trajectory) is the same.

<< Modification of Operation Area 54 >>
In general, the aspect ratio of the display 310 is not 1: 1 but longer in the horizontal direction. On the other hand, the vertical and horizontal lengths of the operation area 54 are both length A. Therefore, when the user U moves his / her hand in the horizontal direction, the cursor 3 moves greatly in the horizontal direction, and thus the user U may have difficulty controlling the position of the cursor 3 in the horizontal direction. Therefore, it is effective to change the vertical and horizontal lengths of the operation area 54 according to the aspect ratio of the display 310. Note that the aspect ratio of the display 310 is input to the information processing apparatus 30.

  FIG. 16 is an example of a diagram illustrating a modified example of the operation area 54. FIG. 16A shows an example of the operation area 54 when the aspect ratio of the display 310 is horizontally long. The operation area creation unit 34 creates the operation area 54 by adding the extended areas 54a to the left and right of the square operation area 54 described above. The length α of the extended region 54a is, for example, 10 cm. This is because if the length A is 30 cm, the hand of the user U can reach within a range of about 50 cm. Therefore, 10 cm is only an example, and the length A may be taken into consideration within the reachable range. For example, it may be variable according to the height of the user U.

  FIG. 16B shows an example of the operation area 54 when the aspect ratio of the display 310 is vertically long. α may be the same as in FIG. Thus, the operability of the user U is improved by changing the vertical and horizontal lengths of the operation area 54 in accordance with the aspect ratio of the display 310.

<Speed detection>
The speed detection will be described with reference to FIG. FIG. 17 is an example of a diagram illustrating a wrist joint or wrist joint speed detection method. The speed detection unit 37 calculates acceleration using the wrist joint or the three-dimensional coordinates of the past several cycles of the wrist joint. For example, the wrist joint or velocity V ₁ of the wrist of the time t1 can be obtained from the three-dimensional coordinates of the wrist joint or wrist of the time t0 and t1. Note that the period at which sensor information is acquired is denoted by Δt. As described above, the speed V1 at time t1 is calculated using only the X and Y components.
Vx = (X _t1 −X _to ) / Δt, Vy = (Y _t1 −Y _to ) / Δt,
V ₁ = √ (Vx ² + Vy ² )
Similarly, the wrist joint or wrist joint velocity V2 at time t2 can be obtained from the three-dimensional coordinates of the wrist joint or wrist joint at times t1 and t2. Since the acceleration α at time t2 is a value obtained by dividing the difference between the speeds V1 and V2 by the period Δt, it can be calculated as follows.
α = (V ₂ −V ₁ ) / Δt
Therefore, if there are three-dimensional coordinates in the past three periods, the speed detector 37 can calculate the acceleration of the wrist joint or the wrist joint for each period.

  In this embodiment, speed and acceleration are used as an index of hand speed. The threshold compared with these speeds shall be determined experimentally. For example, the threshold is a speed that is slightly faster than the stop (for example, 1 to 10 [cm / sec]). The reason for trying to click the operation button 103 is that the speed of the wrist joint or the wrist joint is often slow.

  In addition, when using speed as the speed of a wrist joint or a wrist joint, the speed detection part 37 can calculate speed if the past two periods of three-dimensional coordinates are used. Further, the acceleration may be further calculated as an index of speed. In this case, it can be calculated by using the three-dimensional coordinates of the past four periods.

<Event detection>
The event detection method will be described with reference to FIG. FIG. 18A shows the joints and hand shapes included in the hand portion, and FIG. 18B shows the joints and hand shapes included in the hand portion in a state where the user U holds the hand. The event determination part 38 detects as an event that the user U grasps a hand and clicks the gesture which opens quickly as an example. In other words, at the time of grasping (goo), it becomes an event of left mouse button down, open (par) and left button up. This allows the user to click and drag while holding. In addition, when the time of holding is long, the event determination unit 38 detects a right button down event.

  As described above, the motion detection device 10 has the three-dimensional coordinates of the right hand joint g, the right wrist joint f, the right hand joint h, the right thumb joint i (and the left wrist joint, the left wrist joint, the left hand joint, the left thumb joint). Is detected. As shown in the figure, when the user U holds the right hand, the three-dimensional coordinates of the right hand joint h and the right thumb joint i greatly move in a specific direction. As shown in the figure, when the X and Y coordinates are taken, the Y coordinate of the right hand joint h becomes smaller. In addition, the X coordinate of the right thumb joint i is also reduced.

  The event determination unit 38 determines that a left button down event has occurred when the Y coordinate of the right hand joint h decreases and the X coordinate of the right thumb joint i also decreases. When the Y coordinate of the right hand joint h increases and the X coordinate of the right thumb joint i also increases, it is determined that a left button down event has occurred. When the Y coordinate of the right hand joint h becomes small or the X coordinate of the right thumb joint i also becomes small, it may be determined that an event of a left button down has occurred.

  In addition, since the motion detection device 10 detects a gesture called so-called Janken (goo, choki, par), it may detect a gesture of grasping using this gesture. For example, when a gesture of goo or choki is detected, it is determined that an event of left button down has occurred.

  Further, the event determination unit 38 may detect a gesture of gripping using the Z coordinate instead of using only the two-dimensional coordinate out of the three-dimensional coordinates. When the user U clicks the application software 120, the user U moves his / her hand forward (or backward), for example. The event determination unit 38 detects a change in the three-dimensional coordinates in the Z-axis direction at a predetermined speed or higher, such as the right hand joint g, and determines that a click event has occurred.

  However, if the user U tries to move his / her hand in the Z direction, the X and Y coordinates also move easily, and the position of the cursor 3 may move from the position to be clicked. For this reason, as in this embodiment, a gesture of grasping a hand in which the X coordinate and the Y coordinate are difficult to change is effective. Therefore, gestures other than mainly holding a hand such as moving a finger and opening may be associated with a click event.

<Coordinate conversion when speed is less than threshold>
The coordinate conversion when the speed is less than the threshold will be described with reference to FIG. FIG. 19 is an example of a diagram illustrating coordinate conversion when the speed is less than the threshold value. When the speed detection unit 37 determines that the wrist joint or wrist joint speed is less than the threshold, the coordinate conversion unit 35 and the UI control unit 36 operate as follows.
(i) The cursor display unit 41 displays the guide mesh 4 at the position of the cursor 3.
(ii) It is determined whether or not the three-dimensional coordinates have changed by more than a threshold value (whether the wrist joint has moved by a magnitude greater than the threshold value).
(iii) When the three-dimensional coordinate changes by a threshold value or more, it is determined which component is larger in the X direction or the Y direction.
(iv) Move the grid corresponding to the amount of movement along the guide mesh 4 in the X direction or Y direction with the larger component.

  (i) is demonstrated. When the cursor display unit 41 acquires information indicating that the speed detection unit 37 determines that the speed is less than the threshold, the cursor display unit 41 starts displaying the guide mesh 4. As shown in FIG. 19A, the cursor display unit 41 associates the position of the cursor 3 acquired from the coordinate conversion unit 35 with a predetermined intersection of the guide mesh 4, arranges the cursor 3 at the intersection, and guides the mesh 4. Is displayed. The predetermined intersection is determined in advance at approximately the center of the guide mesh 4. Therefore, the cursor 3 is displayed at the approximate center of the guide mesh 4. The cursor 3 displayed with the guide mesh 4 has a different shape from the cursor 3 when the speed is equal to or higher than the threshold value. In FIG. 19A, the vertical and horizontal double-ended arrows intersect vertically.

  The cursor 3 moves on the intersection of the guide mesh 4. The guide mesh 4 is deleted when the speed exceeds a threshold value. While the speed is less than the threshold value, the guide mesh 4 may move or may be fixed.

  Each square of the guide mesh 4 may be a square, but the length and width may be determined at the same ratio as the aspect ratio of the display 310.

  Next, the determination as in (ii) is to prevent the position of the cursor 3 from being changed by a slight hand movement unintended by the user U. Further, when the user U performs a gesture that the user U wants to click while the cursor 3 is on, for example, the operation button 103 of the application software 120, the hand often moves other than the gesture by the gesture. In this case, when the gesture is performed, the cursor 3 is moved outside the operation button 103, and the user U cannot click the operation button 103. Therefore, by determining as in (ii), it is possible to suppress the position of the cursor 3 from being changed by a slight hand movement that is not intended by the user U.

In FIG. 19B, it is assumed that the three-dimensional coordinates have changed by a threshold value or more. That is, it is assumed that the coordinates of the wrist joint or the wrist joint are moved from the coordinates (X1, Y1) to (X2, Y2). The cursor display unit 41 calculates respective movement amounts ΔX and ΔY in the X direction and the Y direction.
ΔX = X2-X1
ΔY = Y2−Y1
The cursor display unit 41 determines whether at least one of ΔX and ΔY is equal to or greater than a threshold value. The threshold value is determined in consideration of a general wrist joint or a movement amount of the wrist joint, which may fluctuate even when the user U tries to stop, and an error of the motion detection device 10. For example, it may be several centimeters, but a person in charge or the like can appropriately determine experimentally.

  (iii) will be described. When at least one of the movement amounts ΔX and ΔY is equal to or greater than the threshold value, the cursor display unit 41 determines which of ΔX and ΔY is greater. In FIG. 19B, ΔX> ΔY.

(iv) will be described. The cursor display unit 41 determines the position of the cursor 3 for only the X2 of the wrist joint or the wrist joint coordinates (X2, Y2). The position in the y direction in the guide mesh at the position of the cursor 3 uses the previous value as it is. Since the x coordinate of the position of the cursor 3 converted by the coordinate conversion unit 35 is not necessarily on the intersection of the mesh guide, the cursor display unit 41 corrects the x coordinate on the intersection of the mesh guide. For example, the correction is performed as follows.
-Correct to the intersection of the mesh guide with the closest x-coordinate of the cursor 3-Correct to the intersection of the mesh guide next to the x-coordinate of the cursor 3-The mesh guide before the x-coordinate of the position of the cursor 3 Correction to the intersection point By correcting in this way, the cursor display unit 41 can display the cursor 3 displayed together with the mesh guide on the intersection point. In FIG. 19C, the cursor 3 is moved and displayed at the intersection point moved by two squares to the right from the position of the cursor 3 immediately before.

<Wrist joint or wrist joint switching>
There are a case where the user U wants to move the cursor 3 greatly and a case where the user U wants to move the cursor 3 small, such as near the operation button 103. Further, since the user U moves his hand around the elbow, the user U moves faster as the joints on the outer side of the hand move. Therefore, the coordinate conversion unit 35 switches the joint (part of the user's body) to be converted into the position of the cursor 3 according to whether or not the speed detection unit 37 has detected a speed equal to or higher than the threshold value.
A. If the speed is greater than or equal to the threshold, the three-dimensional coordinates of the right hand joint g are converted to the position of the cursor 3.
B. When the speed is less than the threshold value, the three-dimensional coordinates of the right wrist joint f are converted into the position of the cursor 3.

  Thereby, when it is desired to move the cursor 3 greatly, the user U can easily move the cursor 3 quickly, and the cursor 3 can be easily moved to an intended position using the wrist near the operation buttons 103 and the like. In the case of A, the three-dimensional coordinates of the right hand joint h and the right thumb joint i may be used. Since it is further from the elbow, the user can operate the cursor 3 faster.

<Operation reception support icon>
The operation reception support icon 81 displayed by the icon display unit 42 will be described with reference to FIGS. FIG. 20 is an example of a diagram illustrating the operation reception support icon 81 when the user U is detected by the motion detection device 10. The icon display unit 42 first displays the operation reception support icon 81 on the display 310 in a large size. The operation reception support icon 81 has a shape that is substantially similar to the detection range 51 of the motion detection device 10.

  The operation reception support icon 81 may be always displayed, or may be displayed only when a person is detected, but is displayed large when it is detected. Thereby, the user U can recognize that it was detected by the motion detection device 10. In addition, there is an effect that the user's eyes are directed to the display 310. The icon display unit 42 may not only display the operation reception support icon 81 in a large size but also display it by blinking or changing the color. In addition, the operation acceptance support icon 81 may be displayed on the projector 5 separate from the display 310 to perform followogram projection (stereoscopic view).

  In FIG. 20, the user U enters the detection range 51 from the right rear of the detection range 51 of the motion detection device 10. Since the information acquisition unit 31 has acquired the three-dimensional coordinates of the user U, the operation reception support icon 81 displays the person icon 82 at a position corresponding to the position of the user U (referred to as an icon position). Since the person icon 82 is displayed at the position on the icon, the user U can grasp his / her position in the detection range 51. Further, since the information acquisition unit 31 acquires the face orientation of the user U, the icon display unit 42 displays the eye position estimated from the face orientation on the person icon 82.

  In the operation reception support icon 81, a location corresponding to the operation position 52 (hereinafter referred to as an operation position instruction display 83) is highlighted, and the user U makes the person icon 82 enter the operation position instruction display 83. Moving.

  FIG. 21 is an example of a diagram illustrating a method for calculating the position of the person icon 82 in the operation reception support icon 81. FIG. 21A shows the operation reception support icon 81, and FIG. 21B shows a top view of the detection range 51. The motion detection device 10 detects the coordinates of each joint of the user. For example, the user's X1 coordinate and Z1 coordinate can be specified with the center of the motion detection apparatus 10 as the origin O. That is, the relative position (first relative position) of the user with respect to the motion detection device 10 is specified. The distance D from the center O and the direction θ with respect to the front are also known.

  On the other hand, the operation reception support icon 81 has a shape similar to (or similar to) the detection range 51. The position of the operation reception support icon 81 corresponding to the position where the motion detection device 10 is installed (the center of the upper side in the figure) is set as the origin O. The icon display unit 42 converts the above relative position into a relative position (second relative position) with respect to the origin O of the operation reception support icon 81 and displays the person icon 82 at the converted relative position.

The distance that the detection range 51 detects the user is A [m]. On the other hand, the side length a [pixel] corresponding to the distance A of the operation reception support icon 81 is known. Therefore, the coordinates (x1, z1) of the person icon 82 in the operation reception support icon 81 can be calculated using the ratio of a / A.
x1 = X1 × a / A
z1 = Z1 × a / A
The icon display unit 42 updates the position of the person icon 82 every time the information acquisition unit 31 periodically acquires sensor information.

  FIG. 22 is an example of an operation reception support icon 81 when the user U raises his hand at the operation position 52. FIG. 22 shows an example of a screen when the wrist joint or the wrist joint speed is equal to or higher than a threshold value. The icon display unit 42 displays the operation reception support icon 81 in a small size when a predetermined time elapses after the user U enters the operation position 52 or when a predetermined time elapses after the user U raises his hand at the operation position 52. To do. This makes it easier for the user U to view the screen of the application software 120 displayed on the display 310.

  Further, when the user U raises his hand, the operation start detection unit 33 detects that the user U has raised his hand, so the icon display unit 42 raises the hand of the person icon 82. In other words, the person icon 82 with the right hand raised is displayed. Thereby, the operation right is given to the user U who raised his hand. The gesture operation reception program 110 does not receive an operation even if another user U raises his hand. In FIG. 22, another user U enters the detection range 51 from the left rear, but this user U cannot operate the application software 120. That is, the user U (only) who raised his hand earliest at the operation position 52 obtains the operation right.

  The user U creates the operation area 54 at the operation position 52 as described above, and moves the hand to display the cursor 3, or performs a gesture to generate an event.

  FIG. 23A is an example of a diagram illustrating a guide mesh when the user U raises his / her hand at the operation position 52 but the wrist joint or the wrist joint speed is less than a threshold value. In this case, the cursor display unit 41 displays the cursor 3 at the intersection of the guide mesh 4. Further, since the cursor 3 moves only vertically or horizontally, the user U can control the minute movement of the cursor 3 with reference to the square of the guide mesh 4.

  The guide mesh 4 may not be displayed. Even without the guide mesh 4, the cursor 3 moves only vertically or horizontally, and the user U can control the minute movement of the cursor 3. However, since the guide mesh 4 is displayed, the user U can grasp that the cursor 3 can be moved only in units of cells, so that it is expected that the user U feels easy to operate.

  Further, the horizontal line of the guide mesh 4 may not be horizontal, and the vertical line may not be vertical. For example, it suffices if vertical and horizontal straight lines intersect.

  Further, as shown in FIG. 23B, the cursor 3 may move in the radial direction on the circumference of a concentric circle.

<Operation procedure>
FIG. 24 is an example of a flowchart illustrating a procedure until the gesture operation reception program 110 grants an operation right to the user U. The processing in FIG. 24 is repeatedly executed while the information processing apparatus 30 is executing the gesture operation reception program 110, for example.

  First, it is determined whether or not the motion detection device 10 has detected a person by the person detection unit 32 (S10). If no person is detected (No in S10), the process in FIG. 24 ends.

  When a person is detected (Yes in S10), the icon display unit 42 displays the operation reception support icon 81 on the display 310 (S20). Thereafter, the icon display unit 42 displays the operation reception support icon 81 smaller after a certain time even if a person is detected.

  Next, the icon display unit 42 displays the person icon 82 on the icon reception position of the operation reception support icon 81 based on the position of the user U (S30). Since the information acquisition unit 31 periodically acquires sensor information, the icon display unit 42 updates the position of the person icon 82.

  The operation start detection unit 33 determines whether or not the position of the user U has entered the operation position 52 (S40). That is, it is determined whether or not the operation position 52 includes the X, Y, and Z coordinates of the user U.

  Until the user U enters the operation position 52 (No in S40), it is determined whether or not a person has been detected (S50). When no person is detected (Yes in S50), the process in FIG.

  While a person is detected (No in S50), it is determined whether or not the user U has repeatedly entered the operation position 52.

  Next, when a person enters the operation position 52 (Yes in S40), the operation start detection unit 33 determines whether or not the user U raised his hand (S60). For example, it is determined whether or not the hand is positioned at a certain (or higher) height than the user's waist.

  Until the user U raises his hand (No in S60), the determinations after step S40 are repeatedly executed.

  When the user U raises his / her hand (Yes in S60), the operation start detection unit 33 determines whether or not the hand is in the predetermined peripheral area (S70). The predetermined peripheral region refers to the inside of the cube 55, for example. The determination using the cube 55 is an example of determining whether or not the cube 55 is in a predetermined area. For example, it is confirmed that the right hand joint h and the right hand joint g are not around the head joint a. Thereby, it can suppress that the user U on the telephone operates the application software 120, for example. Note that the determination in step S70 is to prevent giving an operation right to a user who does not intend to operate, and is not essential. Further, in the determination of step S70, it is confirmed that the hand (the peripheral area near the body) of the rectangular parallelepiped (the rectangular parallelepiped including from the head to the waist slightly) created from the user's head to the waist is not included. It is preferable to judge. As a result, a region independent of the user's posture is created. Specifically, even in the case where the user holds a baggage such as a binder, not only the area around the head but also the area around the waist can be included in the non-operation area.

  When the determination in step S70 is Yes, the operation start detection unit 33 detects the intention to start the operation, and gives the operation right to the user U who raised his hand (S80). Therefore, even if another user U raises his hand at the operation position 52, the application software 120 can be prevented from being operated.

  Next, the icon display unit 42 displays the person icon 82 with the hand raised at the upper position of the icon (S90). Accordingly, the user U can determine that he / she has raised his / her hand and can start the operation.

  When the intention to start the operation is detected, the operation area creating unit 34 creates the operation area 54 (S100). Thereafter, the user U can operate the position of the cursor 3 with a wrist joint or a wrist joint, or can operate the application software 120 with a gesture. The gesture operation reception program 110 starts receiving an operation of the application software 120 by the user U.

  FIG. 25 is an example of a flowchart illustrating a procedure in which the gesture operation reception program 110 receives the operation of the user U and operates the application software 120. The process of FIG. 25 starts when the operation area 54 is created.

  First, the information acquisition unit 31 acquires sensor information (S110).

  The speed detection unit 37 determines whether the speed of the wrist joint is greater than or equal to a threshold based on the sensor information (S120).

  If the speed of the wrist joint is equal to or higher than the threshold (Yes in S120), the coordinate conversion unit 35 converts the three-dimensional coordinates of the wrist joint into the position of the cursor 3 (S130).

  The output unit 39 outputs the position of the cursor 3 to the application software 120 (S140).

  Further, the event determination unit 38 determines whether or not the user U has made a gesture for generating an event (S150). This is because the user U can perform a gesture even when the guide mesh 4 is not displayed.

  If an event is detected, the output unit 39 outputs the position of the cursor 3 and the event to the application software 120 (S160). Note that the position of the cursor 3 and the event need not be output simultaneously.

  Next, when the speed of the wrist joint is not equal to or higher than the threshold (No in S120), the cursor display unit 41 determines whether the guide mesh 4 is being displayed (S180). This is because it is not necessary to display the guide mesh 4 again while the guide mesh 4 is displayed.

  When the guide mesh 4 is not being displayed (No in S180), the cursor display unit 41 displays the guide mesh 4 with the cursor 3 displayed at the center (S190). The cursor display unit 41 acquires the position of the cursor 3 from the coordinate conversion unit 35. Thereafter, the process proceeds to step S140.

  When the guide mesh 4 is being displayed (Yes in S180), the cursor display unit 41 determines whether or not the three-dimensional coordinates of the wrist joint have moved by a threshold value or more (S200). That is, by focusing attention on the wrist joint instead of the wrist joint, the user U can easily control the minute movement of the cursor 3.

  When the determination in step S200 is Yes, the cursor display unit 41 updates the position of the cursor 3 of the guide mesh 4 (S210). That is, the cursor display unit 41 converts only the larger one of the movement amounts in the X direction and the Y direction in which the wrist joint has moved into the position of the cursor 3. The cursor display unit 41 corrects the position of the cursor 3 to the intersection of the guide mesh 4. Thereafter, the process proceeds to step S140, and the cursor 3 is displayed on the guide mesh 4.

  Further, the operation start detection unit 33 appropriately determines whether or not the user U is in the operation position 52 and whether or not the hand is raised (S170). This is to confirm the willingness to operate.

  If the determination in step S170 is Yes, the process returns to step S110 and the operation of the application software 120 is continued.

  If the determination in step S170 is No, the output unit 39 requests the application software 120 to hide the guide mesh and cursor (S220). Thus, the process of FIG. 25 ends. In this case, the operation area 54 is deleted and the user U loses the operation right.

<Recording and using sensor information>
Subsequently, recording and utilization of sensor information will be described with reference to FIGS. Conventional digital signage cannot be left as a history whether it is viewed by a user. Since the information processing apparatus 30 according to the present embodiment acquires the user's face orientation and the like from the motion detection apparatus 10, the content provider and the like can analyze how many users were browsing later. Further, since the information recording unit 43 records a screen shot of the content displayed on the display 310, it is possible to analyze what screen shot the user has shown. Since the screen shot has a large data size, it may be recorded only when the user's face is facing the display 310.

FIG. 26 shows an example of sensor information recorded by the information recording unit 43 for analysis of face orientation and the like. In FIG. 26, sensor information recorded for one user in the XML format is described. Hereinafter, such information will be described.
The <TrackingID> tag is identification information of a user assigned to a recognized user (including a user who does not have operation authority).
The <Glasses> tag indicates whether you are wearing glasses.
The <Happy> tag indicates whether you are smiling (it looks happy).
The <LeftEyeClosed> tag indicates whether the left eye is closed.
<RightEyeClosed> tag indicates whether the right eye is closed
The <LookingAway> tag indicates whether it is facing you.
The <MouthMoved> tag indicates whether the mouth is moving.
The <MouthOpen> tag indicates whether the mouth is open.
The <Yaw> tag indicates a lateral angle (yaw angle) of the face, and becomes a positive value when the neck is bent to the left side toward the motion detection device 10 and becomes a negative value when the neck is bent to the right.
The <Pitch> tag indicates an angle (pitch angle) in the vertical direction of the face, and is a positive value when facing upward toward the motion detection device 10, and a negative value when facing downward.
The <Roll> tag indicates the angle in which the face is tilted (roll angle), and is zero when the eyes and mouth are vertical.
The <PositionInfo> tag indicates the position of the user, and the <X> tag indicates the distance in the horizontal direction from the center of the motion detection apparatus 10 (right plus value, left minus value). The <Y> tag indicates the height, and the <Z> tag indicates the distance from the motion detection device 10.
The <LeftButtonDown> tag of the <MouseTrackingInfo> tag indicates whether the left button of the mouse is pressed, and the <RightButtonDown> tag indicates whether the right button of the mouse is pressed. The <MousePositionInfo> tag indicates the X coordinate and Y coordinate of the cursor.

  Note that <MouseTrackingInfo> and <MousePositionInfo> are recorded only for users with operational authority. The information recording unit 43 records such information for each user.

  FIG. 27 is an example of a diagram showing a screen shot 201 acquired by the information recording unit 43 when the sensor information of FIG. 26 is acquired. Therefore, content providers etc. measure <Waw>, <Pitch>, and <Roll> tags to determine where the user is looking at the content (where they are interested), and the gaze dwell time, etc. You can do it. In addition, the <MouseTrackingInfo> tag can be used to determine from the screen shot 201 whether the user is performing an operation or what the user is interested in.

  By the way, since the motion detection apparatus 10 uses distance information for user detection, the user may not be detected depending on the positions of a plurality of users. FIG. 28 is an example of a diagram illustrating users who are not detected. FIG. 28 shows that there are three users U <b> 1 to U <b> 3 in the detection range 51 of the motion detection device 10. However, since the user U1 and the user U3 overlap in the distance direction when viewed from the motion detection device 10, the motion detection device 10 cannot detect the user U3. That is, skeleton information cannot be acquired.

  However, since the motion detection apparatus 10 captures the color image 202 as described above, the user U3 is shown in the color image 202. Further, the distance information of the area where the user U3 is shown is also acquired. The information recording unit 43 determines that the person is a person, for example, by recognizing a face or silhouette from the color image 202. Thereby, the number of users in the detection range 51 can be acquired correctly.

  The area in which the user U3 is shown can be extracted as an area within a certain range of coordinates in the Z direction (for example, within the processing range behind the user U1). The detection time can be shortened by limiting to the area having the rear Z coordinate.

  In this way, the number of users that can be acquired at one time is limited only by the skeleton information (currently a maximum of six people), but by combining with the color image 202 and distance information, the number of users can be reduced even when people are gathering It can be detected. Also, the color image 202 before the user U3 is obstructed by U1 is compared with the color image after the obstruction, or the distance information before the user U3 is obstructed by U1 is compared with the distance information after the obstruction. You can also. When the number of users detected by the human detection unit 32 suddenly decreases, the information recording unit 43 estimates that the user is hiding, and can search by face recognition or the like from an area where the distance information is similar to that before hiding.

  FIG. 29 is an example of a diagram illustrating an example of analysis of sensor information stored in the sensor information DB 44 by the information recording unit 43. The information processing apparatus 30 totals the number of users for each time zone (10 minutes in the figure) from the sensor information stored in the sensor information DB 44. In FIG. 29, the height of the bar graph 205 indicates the number of users for each time period. Further, in each bar graph 205, the number of users in the time zone is displayed separately for the number 204 of users having the operation right and the number 203 of other users. Note that since the same <TrackingID> is assigned to the same user that is continuously detected, there is little possibility that the information processing apparatus 30 counts twice.

  In FIG. 29, a screen shot 201 of each time zone is displayed above the bar graph 205 like a timeline. Alternatively, a representative screen shot 201 for each time period may be displayed. The representative screen shot 201 is a screen shot at the beginning or end of the time period, a screen shot with the largest image change, or the like.

  Therefore, a content provider or the like can determine which content is a collection of many users. For example, in the case of a lecturer (school or prep school teacher, sponsor, etc.) with different contents in FIG. 29, it can be determined which lecturer is popular.

<Summary>
As described above, the information processing apparatus 30 according to the present embodiment improves the operability of the position of the cursor 3 by limiting the movement direction of the cursor 3 to a predetermined direction when the speed of movement is less than the threshold value. be able to. In addition, when the speed of movement is less than the threshold, the cursor does not move unless the right wrist joint f is moved beyond the threshold, and therefore it is easy to prevent the cursor from moving when the user performs a click gesture. Further, depending on whether the speed of operation is equal to or higher than the threshold value, the user can be operated using different joints, so that the user operability can be improved. In addition, since the vertical and horizontal lengths of the operation area 54 are appropriately changed according to the aspect ratio of the display 310, the operability of the cursor 3 by the user can be improved. Further, even when a plurality of users are detected, the operation right can be given to the user who first indicated the intention of the operation. Further, by displaying the operation reception support icon 81, the user can easily move to the operation position 52, and can easily understand that the user is operating.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, the information processing apparatus 30 may operate using a server client method. In this case, the information processing apparatus 30 transmits the sensor information to the server, and receives the cursor position and the event from the server. Then, it is sent to the application software 120. With such a display system having the information processing apparatus 30 and the server, the user can operate the digital signage.

  Further, although it has been described that the gesture operation reception program mainly displays the cursor 3, the cursor 3 may be displayed as appropriate using the function of the OS 130. Further, the application software 120 may display the cursor 3.

  Further, instead of making the coordinates in the operation area 54 and the cursor position correspond one-to-one, the user may operate the cursor position by the amount of movement of the hand. For example, when moving with the cursor, hold the hand and move it at a certain speed. The coordinate conversion unit converts the movement amount of the hand into the movement amount of the cursor and updates the position of the last cursor. Therefore, in this case, the operation area 54 may not be provided.

  Further, although the left click gesture has been described in the present embodiment, it is also possible to input the right click gesture by the information processing apparatus 30 distinguishing and detecting the right hand and the left hand. In addition, the cursor can be moved based on the three-dimensional coordinates of the joints such as feet or the face orientation. In addition to holding a hand, various human gestures can be associated with events.

Further, although the operation of raising the hand has been described as the operation of obtaining the operation right, the operation right may be given by the following operation.
When a voice or sound (hand clapping) is detected by the microphone 17 When a voice or sound (hand clapping) detected by the operation start detection unit 33 is determined to be a specific rhythm When the operation start detection unit 33 detects a signal transmitted from a wireless device such as a smartphone that the user has ・ Analysis of image data When the operation start detection unit 33 detects that the hand has been raised ・ When the operation start detection unit 33 detects that the shape of the hand is in a specific shape that matches a predetermined condition (goo, choki, par, etc.) )
When the operation start detection unit 33 detects that the user is standing (the operation right can be given when a person sitting in the venue stands at a seminar or the like)
When the operation start detection unit 33 detects that the facial expression is a specific facial expression that matches a predetermined condition (for example, laughing)
When the operation start detection unit 33 detects that the user's line of sight (face orientation) is heading toward a specific area. The configuration example of FIG. These are divided according to the main functions. The present invention is not limited by the way of dividing the processing unit or the name. The processing of the information processing apparatus 30 can be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  The icon display unit 42 is an example of a position display unit, the information acquisition unit 31 is an example of an operation information acquisition unit, and the coordinate conversion unit 35 and the event determination unit 38 are examples of an operation reception unit.

DESCRIPTION OF SYMBOLS 3 Cursor 4 Guide mesh 10 Motion detection apparatus 30 Information processing apparatus 31 Information acquisition part 32 Person detection part 33 Operation start detection part 34 Operation area creation part 35 Coordinate conversion part 36 UI control part 37 Detection part 38 Event judgment part 39 Output part 41 Cursor display section 42 Icon display section 43 Information recording section 54 Operation area 81 Operation reception support icon 100 Information processing system

Japanese Patent Laying-Open No. 2015-176451

Claims (10)

An information processing apparatus that acquires position information related to a user's position from an apparatus that detects the position of a user around the display device,
Converting the position information into a display position in the display device, and displaying a user position display information representing a user position at the display position;
The position display means is an information processing apparatus that displays guidance position display information on the display device for guiding the user to a position where the user operates the content displayed on the display device according to the operation of the user. In addition to the position information, the device detects operation information related to a user operation,
Having operation information acquisition means for acquiring the operation information;
The information processing apparatus according to claim 1, wherein the guidance position display information represents a position of a user from which the apparatus can detect the operation information. Operation receiving means for receiving an operation of application software operating on the information processing apparatus based on the operation information;
The information processing apparatus according to claim 2, wherein the guidance position display information represents a position where a user can operate the application software. When the user position display information is displayed in the guidance position display information, and it is detected by the operation information that the user raised his hand,
The information processing apparatus according to claim 2, wherein the position display unit changes the shape of the user position display information to a shape in which a user raises his / her hand.   The information processing apparatus according to claim 1, wherein the position display unit displays a detection range in which the apparatus can detect a user on the display apparatus. If the device detects a user,
The information processing apparatus according to claim 5, wherein the position display unit displays the user position display information in the detection range. The position information represents a first relative position to the device;
The position display means converts the first relative position into a second relative position with respect to a predetermined position in the detection range, and displays the user position display information at the second relative position in the detection range. The information processing apparatus according to claim 5 or 6. A display system having one or more information processing devices for guiding a user based on position information related to a user position acquired from a device that detects a position of a user around the display device,
Converting the position information into a display position in the display device, and displaying a user position display information representing a user position at the display position;
The said position display means is a display system which displays the guidance position display information for guide | inducing the said user to the position which operates the content displayed on the said display apparatus according to the said user's operation | movement on the said display apparatus. A device for detecting a position of a user around the display device;
Action information acquisition means for acquiring position information relating to the position of the user from the device;
A program executed by an information processing device functioning as a position display unit that converts the position information into a display position in the display device and displays user position display information representing a user position at the display position;
The display device displays guidance position display information for guiding the user to a position where the user operates the content displayed on the display device according to the user's operation by the position display means. Display system. When installed in an information processing device, the information processing device is
Operation information acquisition means for acquiring position information relating to the position of the user from a device for detecting the position of the user around the display device;
A program for converting the position information into a display position in the display device and functioning as position display means for displaying user position display information representing the position of the user at the display position, wherein the position display means A program for displaying guidance position display information on the display device for guiding the user to a position where the content displayed on the display device is operated according to the user's operation.

Images