JP6834197B2 - Information processing equipment, display system, program - Google Patents

Description

The present invention relates to information processing devices, display systems and programs.

A system that displays information on a display installed in a place where people gather or pass is known. An information processing device is connected to the display via a network or the like, and the information processing device displays useful information such as advertisements, weather, news, etc. by displaying still images and moving images on the display. it can. Such a system or mechanism is called digital signage. The display is installed outdoors, in stores, in public spaces, in public spaces, in aisles, etc., and can provide detailed and new information to visitors, visitors, facility users (hereinafter, simply referred to as users), and the like.

By the way, among the current digital signage, especially the large screen digital signage has the main function of displaying information unilaterally. However, from the user's point of view, since the information changes from moment to moment, it may not be possible to browse slowly even if one is interested in the displayed information. Further, from the viewpoint of the information provider, it is not possible to grasp what kind of information the user is interested in because the reaction from the user cannot be obtained only by flowing the information. Further, it is difficult for the user to display more detailed information by using the digital signage information as a window, or for the provider to provide detailed information of the information that the user is interested in. That is, it has been difficult to perform interactive operations with conventional digital signage.

In order to realize interactive operation, it is sufficient to have an input device near the display for the user to input the operation, but it is difficult to secure the installation space, or even if there is an installation space, the input device may be damaged. There is a risk of theft. Therefore, an operation mode in which the digital signage detects the user's gesture and accepts the operation is examined (see, for example, Patent Document 1). Patent Document 1 discloses a pointing control device that moves the cursor displayed on the display surface from the detected movement of the user and adjusts the movement of the cursor based on the distance to the user.

However, Patent Document 1 has a problem that it is not possible for the user to know where to move to operate the digital signage. This will be described below. Since digital signage is installed in a place where many users come and go, it is unlikely that only one person will see the digital signage at the same time. That is, the digital signage is likely to be viewed by a plurality of users at the same time, and if all the users who can view the digital signage can operate the digital signage, it becomes difficult for each user to perform the desired operation. .. In addition, if all users can operate the digital signage, the unconscious movement of the passing user may interfere with the operation of the user who really wants to operate the digital signage.

To deal with such inconvenience, a method in which the motion detection device detects the motion of only the user standing at a specific position can be considered. If the specific position is not so wide, the motion detection device can limit the number of users (for example, one person) to detect the motion.

However, in order for the user to grasp a specific position by a physical index (marking such as tape or paint), a digital signage administrator or the like needs to prepare a separate index. In addition, it is necessary to prepare the index according to the position where the motion detection device can detect the user's motion, which takes time and cost for setting.

In view of the above problems, it is an object of the present invention to provide an information processing device capable of guiding a user to a place where an operation target is operated.

The present invention is an information processing device that acquires position information related to a user's position from a device that detects the user's position and operation information related to the user's operation around the display device, and the position is at the display position on the display device. It has a position display means for converting information and displaying user position display information indicating the user's position at the display position, and the position display means changes the content displayed on the display device by the user into the operation of the user. The guidance position display information for guiding the user to the position to be operated according to the operation is displayed on the display device, and the position display means displays the detection range in which the device can detect the user on the display device, and the operation. It has an operation information acquisition means for acquiring information and an operation reception means for receiving an operation of application software operating on the information processing device based on the operation information, and the user raises his / her hand from the device. When the operation information is received, the position display means displays the user position display information in the form of a person raising his / her hand, and the operation reception means gives an operation right only to the user who raises his / her hand, and another user. Does not accept the operation even if he raises his hand .

It is possible to provide an information processing device that can guide a user to a place where an operation target is operated.

This is an example of a diagram for explaining the control of the position of the display and the cursor of the information processing system. This is an example of a system configuration diagram of an information processing system. This is an example of a hardware configuration diagram of an information processing device. This is an example of the hardware configuration diagram of the motion detection device. It is a figure which shows the joint which can obtain the three-dimensional coordinates. This is an example of a diagram illustrating the relationship between the application software and the gesture operation reception program. This is an example of a functional block diagram in which the functions of the gesture operation reception program are shown in a block shape. This is an example of a diagram for explaining a person's detection range and an operation position with respect to the motion detection device. This is an example of a diagram showing the detection range in three dimensions. It is an example of the figure explaining the operation area. This is an example of a diagram for explaining the operation area in detail. It is an example of the figure explaining the non-operation operation of the user. This is an example of a side view of the cube of FIG. This is an example of a diagram showing the coordinates of the wrist joint or wrist joint and the position of the cursor in the operation area. It is the figure which looked at the three-dimensional coordinates of the wrist joint or the wrist joint of the user U from the side. It is an example of the figure explaining the modification of the operation area. It is an example of the figure explaining the method of detecting the speed of a wrist joint or a wrist joint. It is an example of the figure explaining the event detection method. It is an example of the figure explaining the coordinate transformation when the speed is less than the threshold value. This is an example of a diagram showing an operation reception support icon when a user is detected by the motion detection device. This is an example of a diagram illustrating a method of calculating the position of the person icon in the operation reception support icon. This is an example of a diagram showing an operation reception support icon when the user raises his / her hand at the operation position. This is an example of a diagram showing a guide mesh when the user raises his / her hand at the operation position but the speed of the wrist joint or the wrist joint is less than the threshold value. This is an example of a flowchart showing the procedure until the gesture operation reception program grants the operation right to the user. This is an example of a flowchart showing a procedure in which the gesture operation reception program accepts a user's operation and operates the application software. It is a figure which shows an example of the sensor information which an information recording part records for analysis such as a face orientation. It is an example of the figure which shows the screenshot which the information recording part acquired when the sensor information of FIG. 26 was acquired. It is an example of the figure explaining the user which is not detected. This is an example of a diagram for explaining an analysis example of sensor information stored in the sensor information DB by the information recording unit.

Hereinafter, the information processing system 100 for carrying out the present invention and the position information creation method performed by the information processing system 100 will be described with reference to the drawings.

<Outline features of information processing system>
The schematic features of the information processing system 100 will be described with reference to FIG. FIG. 1 is an example of a diagram for explaining guidance of the user to the operation position 52.

As shown in FIG. 1A, the display 310 displays a list 101 of contents as digital signage and a moving image 102, and various operation buttons 103 (playing, stopping, etc. of the moving image 102) for operation input are displayed. It is displayed. A motion detection device 10 is installed below the display 310. The motion detection device 10 detects a user who has entered the detection range 51 of a person within a certain angle range and within a certain distance. In addition, the motion detection device 10 detects the standing position and motion of the user at the operation position 52.

The operation reception support icon 81 is always displayed on the display 310. Alternatively, it may be displayed only when the motion detection device 10 detects the user. An operation position instruction display 83, which will be described later, is displayed on the operation reception support icon 81. When the motion detection device 10 detects the user U, the information processing device 30 displays the person icon 82 representing the user U on the operation reception support icon 81. The person icon 82 displays the person icon 82 at a position in the operation reception support icon 81 corresponding to the position of the user in the detection range 51.

Therefore, the user who is looking at the display 310 can know that he / she has been detected. Further, it can be grasped that when the user moves, the person icon 82 also moves.

The user knows from guidelines and the like that the person icon 82 must be moved to the operation position instruction display 83 in order to operate the digital signage. The user moves within the detection range 51 so that the person icon 82 whose position changes with his / her movement enters the operation position instruction display 83. When the person icon 82 is present on the operation position instruction display 83, the user is present at the operation position 52.

In this way, when the user moves so that the person icon 82 moves to the operation position instruction display 83, the user can naturally move to the operation position 52 where the motion detection device 10 detects the user's motion. Therefore, the information processing device 30 can guide the user to the operation position 52 by detecting the position of the user, converting this position into a position on the operation reception support icon 81, and displaying the position.

Therefore, in the information processing system of the present embodiment, the user can operate the digital signage wherever the digital signage administrator or the like moves without giving a physical index (marking such as tape or paint) to the operation position 52. It is possible to let the user know whether or not. However, when there is a physical index, the user can more easily grasp the operation position 52.

<Terminology>
The position information regarding the user's position is information that identifies the user's position, information that represents the user's position, and the like. For example, the position is relative to the motion detection device 10, but if the positions of the motion detection device 10 and the display device are fixed, it may be referred to as a relative position to the display device. The position information may be an absolute position including coordinates and the like.

The display position on the display device means a position on a display device such as a display. Specifically, the display position is indicated by a cursor or the like.

The content displayed on the display device refers to all the information that is displayed and can be visually judged. If audio is played along with the content, the audio is also included in the content. In this embodiment, the term "content" will be used.

The user position display information is display information indicating the user or the user's position on the display device. In this embodiment, the term human icon 82 will be used.

The guidance position display information is display information for guiding the user to a predetermined position on the display device. Alternatively, it is display information indicating a user's preferred position. In the present embodiment, the term operating position instruction display 83 will be described.

The operation information is information representing the operation of the user. Alternatively, the information is converted into a form in which the user's operation can be interpreted by the information processing device. Alternatively, it is information for inputting the user's operation to the information processing device without contact. Specifically, it includes a position, a posture, a movement of a limb, and the like, and will be described by the term sensor information in the present embodiment.

The condition when the condition of the predetermined speed is met means the condition for detecting the user's intention regarding the control of the cursor position. This intention refers to the intention to control the position accurately. The cases where the conditions are met include, for example, that the speed is less than the threshold value, that the speed is within a predetermined range, and the like. In the following, the speed will be described as an example of being less than the threshold value.

<System configuration example>
FIG. 2 is an example of a system configuration diagram of the information processing system 100. As shown in FIG. 2A, the information processing system 100 includes a display 310, an motion detection device 10, and an information processing device 30. The motion detection device 10 is connected to the information processing device 30 with a cable such as a USB cable or IEEE1394, and the information processing device 30 and the display 310 are DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (registered trademark. High). -It is connected via a cable that can communicate according to standards such as Definition Multimedia Interface) and VGA (Video Graphics Array). The motion detection device 10 and the information processing device 30 may communicate wirelessly with a wireless LAN, Bluetooth (registered trademark), ZigBee (registered trademark), or the like. Similarly, the information processing device 30 and the display 310 may communicate wirelessly with a wireless LAN, a wireless standard of HDMI, or the like.

As the information processing device 30, a general-purpose device such as a PC (Personal Computer), a tablet terminal, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant) can be used. The information processing device 30 operates an OS (Operating System) and has a function of detecting the position of a pointing device such as a mouse or a touch panel and displaying the cursor 3 at this position. The OS also detects events (left-click, right-click, double-click, mouse over, mouse down, mouse up, etc.) performed by the pointing device. In addition, the information processing device 30 operates various application software on the OS. Since the OS notifies the application software of the position and event of the pointing device, the application software operates based on the event performed at the position of the pointing device.

In the present embodiment, the motion detection device 10 and the gesture operation reception program described later operate as a pointing device. The gesture operation reception program is a program that acquires sensor information from the motion detection device 10 and converts it into the position of the cursor 3 or detects an event, and is operated by the information processing device 30. By notifying the application software 120 of the position of the cursor 3 and the event by the gesture operation reception program 110, the application software 120 does not need to be aware of the pointing device (no change due to the pointing device being the motion detection device 10). ) Can work.

The display 310 is a liquid crystal type display device, a display device based on organic EL as a light emitting principle, a display device using plasma light emission, or the like. The application software 120 creates a screen called a CUI (Character User Interface) or a GUI (Graphical User Interface) and displays it on the display 310.

As the display 310, any device provided with the display 310 may be used. For example, a display provided on an electronic blackboard, a notebook PC, or the like can be used as the display 310.

The motion detection device 10 is a sensor that detects the motion of a person. For example, when a space is imaged with a stereo camera, parallax is obtained for each pixel of the image data, and distance information for each pixel is obtained. In addition, by recognizing a pattern of image data, it becomes possible to detect a person or recognize a person's movement. Specifically, three-dimensional coordinates of a person's palm, wrist, shoulder, etc., posture, and the like can be obtained. Kinect (registered trademark; hereinafter, the appendix of the registered trademark is omitted) is known as a sensor suitable for detecting such a human motion. The details of Kinect will be described later. The motion detection device 10 does not have to be Kinect, and may be a sensor having the same function as Kinect, a stereo camera, a camera that detects a distance by a TOF (Time Of Flight), or the like.

The installation position of the motion detection device 10 in FIG. 2A is an example, and may be installed above, on the left side, or on the right side of the display 310. Further, the installation position of the information processing apparatus 30 is only an example, and may be installed behind the display 310, for example. Further, it may be connected to the motion detection device 10 via a network.

As shown in FIG. 2B, the information processing system 100 may have a projector 5 instead of the display 310. In this case, the information processing system 100 includes a motion detection device 10, a projector 5, and an information processing device 30. The projector 5 is a display device that projects an image onto a projection surface 6 such as a screen by an LCD (transmissive liquid crystal) method, an LCOS (reflection type liquid crystal) method, a DLP (Digital Light Processing) method, or the like.

The projector 5 and the information processing device 30 are connected in the same manner as in the connection mode between the display 310 and the information processing device 30. It is preferable that the projector 5 can project from a so-called polar single focus. This makes it easier to secure a space for the user U to operate. The motion detection device 10 may be on the user U side or the screen side of the projector 5. When it is on the screen side, it may be fixed to the lower side, the upper side, the left side or the right side of the screen. If it is on the user U side, it is installed on a prepared table. Further, the motion detection device 10 is installed at the right end of the display 310 as an example, and may be installed at the left end. The reason why it is installed at the right end or the left end is to avoid waste heat of the projector 5. If the waste heat can be ignored, it may be installed in the center of the display 310.

The display 310 and the projector 5 may be used together. In this case, for example, the projector 5 can guide the user to the operation position 52 by projecting an arrow or the like toward the floor. In addition, an arrow can be displayed to indicate the direction of travel (for example, the elevator is over there, the operation position 52 is over there, etc.).

<About hardware configuration>
<< Hardware configuration of information processing device 30 >>
FIG. 3 is an example of a hardware configuration diagram of the information processing device 30. The information processing device 30 includes a CPU 301, a ROM 302, a RAM 303, and an auxiliary storage device 304. Further, the private use server includes an input unit 305, a display control unit 306, a network I / F 307, and an external device I / F 308. Each part of the information processing apparatus 30 is connected to each other via the bus B.

The CPU 301 executes various programs, an OS, etc. stored in the auxiliary storage device 304. The ROM 302 is a non-volatile memory 16. The ROM 302 stores programs, data, and the like necessary for the CPU 301 to execute various programs stored in the auxiliary storage device 304.

The RAM 303 is a main storage device such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). Various programs stored in the auxiliary storage device 304 when executed by the CPU 301 are expanded in the RAM 303, and the RAM 303 becomes a work area of the CPU 301.

The auxiliary storage device 304 stores various programs executed by the CPU 301 and various databases used when the various programs are executed by the CPU 301. The auxiliary storage device 304 is, for example, a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The input unit 305 is an interface for the operator to input various instructions to the information processing device 30. For example, a keyboard, a mouse, a touch panel, a voice input device, and the like may not be connected in the present embodiment.

At the request of the CPU 301, the display control unit 306 displays various information contained in the information processing device 30 on the display 310 in the form of a cursor 3, a menu, a window, characters, an image, or the like. The display control unit 306 is, for example, a graphic chip or a display 310 I / F.

The network I / F 307 is a communication device that communicates with other devices via a network. The network I / F 307 is, for example, an Ethernet (registered trademark) card, but is not limited thereto.

The external device I / F 308 is an interface for connecting a USB cable, various storage media 320 such as a USB memory, and the like. In this embodiment, the motion detection device 10 is connected.

The information processing device 30 preferably supports so-called cloud computing, and the information processing device 30 does not need to be housed in one housing or provided as a group of devices. In this case, the configuration of the information processing device 30 is configured by dynamically connecting and disconnecting hardware resources according to the load.

<< Hardware configuration of motion detection device 10 >>
FIG. 4 is an example of a hardware configuration diagram of the motion detection device 10. The motion detection device 10 includes a CPU 11, a color camera 12, an infrared camera 13, an infrared projector 14, a main storage device 15, a non-volatile memory 16, a microphone 17, an acceleration sensor 18, and an LED 19. The CPU controls the entire motion detection device 10 by expanding the firmware stored in the non-volatile memory 16 to the main storage device 15 and executing the firmware. The main storage device 15 is a high-speed volatile memory such as a DRAM, and the non-volatile memory 16 is a flash memory or the like.

As an example, the color camera 12 is an imaging device that captures a color image at a resolution of 1920 × 1080. That is, the density information of R, G, and B is obtained for each pixel. The infrared camera 13 and the infrared projector 14 function as one depth sensor (distance sensor). The infrared projector 14 projects pulse-modulated infrared rays, and the infrared camera 13 captures the infrared rays projected. The depth sensor measures the time it takes for infrared rays to be projected and then reflected and returned, and obtains distance information for each of the projected infrared rays. The resolution of the distance information is 512 × 424, but this is just an example.

Further, the principle that the depth sensor measures the distance is only an example, and the projected infrared pattern may be imaged by the infrared camera 13 and the distance information may be acquired from the distortion of the pattern.

The microphone 17 is a device that detects surrounding sounds, and the CPU 11 estimates the direction of the sound source by comparing the phases of the sounds detected by the plurality of microphones 17. The acceleration sensor 18 is a sensor that detects the inclination of the motion detection device 10 with respect to the horizontal. The LED 19 displays the state of the motion detection device 10 by color, blinking, or the like.

<Information acquired by the motion detection device 10>
Subsequently, the information acquired by the motion detection device 10 will be described. The information processing device 30 can use any information acquired by the motion detection device 10. However, it is not necessary to use all the information described below. The information acquired by the motion detection device 10 described below is referred to as "sensor information".

1. 1. Color image This is image data including R, G, and B density information for each pixel. The infrared camera 13 of the motion detection device 10 captures a color image at a predetermined frame rate (for example, 30 fps).

2. 2. Distance image This is image data obtained with distance information at a resolution of 512 × 424 as described above. The depth sensor of the motion detection device 10 captures a distance image at a predetermined frame rate (for example, 30 fps). Therefore, the distance to the target including the user U can be obtained at a predetermined resolution.

3. 3. Infrared image This is a grayscale image captured by the infrared camera 13. The pulse projected from the infrared projector 14 can be imaged. In addition, a person (user U) who emits infrared rays can be detected.

4. The person shape data motion detection device 10 can create the person shape data by extracting only the person area from the distance image. For example, a human imaging region is specified from an infrared image, or a region whose distance changes according to a frame is specified as a human imaging region. The shape data of a person is obtained by extracting this imaging region from a distance image. Therefore, the information processing device 30 knows the pixel region in which the user U is located. Further, when a plurality of users U are detected, the motion detection device 10 detects the user U in association with the pixel area. The information processing device 30 can also cut out only a person area from a color image.

5. Skeleton information Coordinate information of joints. Sometimes called bone information or skeleton information. FIG. 5 shows a joint from which three-dimensional coordinates can be obtained. Note that FIG. 5 is a front view of the user's joint. The reference numerals and joint names in FIG. 5 will be described. Further, in FIG. 5, since the joint on the left half is the same as that on the right side, the reference numerals are omitted.
a ... head joint, b ... neck joint, c ... shoulder center joint, d ... right shoulder joint, e ... right elbow joint, f ... right wrist joint, g ... right wrist joint, h ... right wrist joint, i ... right thumb joint , J ... spine joint, k ... lumbar central joint, l ... right lumbar joint, m ... right knee joint, n ... right heel joint, o ... right ankle joint The motion detection device 10 is a three-dimensional coordinate of each joint in space. Not only (X, Y, Z), but also joint coordinates (x, y) on a color image (on display 310) are provided. This is because the pixel region in which the user U is located is clear. Further, since the three-dimensional coordinates of the "right hand joint", "right thumb joint", "left hand joint", and "left thumb joint" are known, the motion detection device 10 can determine whether the user U has the palm open or closed.

6. Facial expression This is the facial expression of the user U based on the recognition of the face. The motion detector 10 is usually laughing, surprised, with left / right eyes closed or open, mouth open or closed, looking away from the screen, wearing glasses , Etc. can be recognized.

7. Face orientation The orientation of the face. The motion detection device 10 can detect the yaw angle, the pitch angle, and the roll angle.

8. Heart rate The estimated heart rate of user U. The skin is estimated from color images and infrared images, and it is estimated from subtle changes in the skin due to blood flow.

<About the function of the information processing device 30>
First, the relationship between the application software 120 and the gesture operation reception program 110 will be described with reference to FIG. Both the application software 120 and the gesture operation reception program 110 operate on the OS 130. The application software 120 may be any one that can be operated by location information or an event. For example, there are application software 120 for displaying advertisements, application software 120 for presentations, and the like. However, it is not limited to these, and may include, for example, a browser, a word processor, a game, and the like.

The device driver 140 of the motion detection device 10 is installed in the information processing device 30. The device driver 140 is software that processes a signal detected by the motion detection device 10 into sensor information. The device driver 140 may be called an SDK (Software Development Kit). Any application software 120 can acquire the sensor information created by the device driver 140 via the API (Application Interface) of the OS 130. In the present embodiment, the gesture operation reception program 110 uses this API to acquire sensor information used for processing.

The gesture operation reception program 110 acquires sensor information from the motion detection device 10 and performs various processes, controls the gesture operation reception program 110 itself, and outputs the position and event of the cursor 3 for operation to the application software 120. Send. The application software 120 has a function of detecting the position and event of the cursor 3 input by operating a pointing device such as a mouse or a touch panel. Generally, the OS 130 detects the position and event of the pointing device and provides it to the application software 120. Since a general-purpose pointing device is not used in this embodiment (not connected to the information processing device 30), the gesture operation reception program 110 provides the position and event of the cursor 3 to the application software 120. The gesture operation reception program 110 can provide the position and event of the cursor 3 to the plurality of application software 120.

The provision method may follow the specifications of the OS 130 and the application software 120. For example, interprocess communication between the application software 120 provided by the OS 130 and the gesture operation reception program 110 may be used, or the gesture operation reception program 110 notifies the OS 130 of the position and event of the cursor 3 and notifies the application software. It may be the method that 120 acquires. With such a mechanism, the application software 120 does not need to be changed to acquire the position and event of the cursor 3 from the gesture operation reception program 110, and the gesture operation reception program 110 is used together with the general-purpose application software 120. Can be done.

Although the gesture operation reception program 110 is one of the application software, it may be implemented as a device driver.

<< Functions of Gesture Operation Reception Program 110 >>
FIG. 7 is an example of a functional block diagram in which the functions of the gesture operation reception program 110 are shown in a block shape. The gesture operation reception program 110 includes an information acquisition unit 31, a person detection unit 32, an operation start detection unit 33, an operation area creation unit 34, a coordinate conversion unit 35, a UI control unit 36, a speed detection unit 37, and an event determination unit 38. It has an information recording unit 43 and an output unit 39. Each of these parts is a function or means realized by operating any of the components shown in FIG. 3 by an instruction from the CPU 301 according to the program 304p expanded from the auxiliary storage device 304 to the RAM 303. .. The program 304p includes an OS 130, a device driver 140, application software 120, and a gesture operation reception program 110.

Further, the information processing device 30 has a sensor information DB 44 (database), and the information recording unit 43 records the sensor information in the sensor information DB 44. The sensor information DB 44 is built in the auxiliary storage device 304 and the RAM 303 shown in FIG.

The information acquisition unit 31 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and acquires sensor information from the device driver 140. Since the motion detection device 10 periodically detects the motion, the information acquisition unit 31 periodically and repeatedly acquires the sensor information. It is not necessary to acquire all the sensor information, but only the sensor information used for processing needs to be acquired. The following information is used in this embodiment.
-Three-dimensional coordinates of the right shoulder joint d, the right wrist joint f, the right wrist joint g, the right wrist joint h, and the right thumb joint i are used. Since the user can operate with either the right hand or the left hand, the three-dimensional coordinates of the left shoulder joint, the left wrist joint, the left wrist joint, the left wrist joint, and the left thumb joint are also used.
-Three-dimensional coordinates for each user U (mainly used to display the person icon 82)
-Face orientation (used to display the eyes of the person icon 82)
The person detection unit 32 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and the motion detection device 10 detects a person based on the sensor information acquired by the information acquisition unit 31. To detect. It may be possible to detect that the motion detection device 10 has detected a person or that a person has been detected by the number of people, or that when the motion detection device 10 detects a person, the person shape data or the skeleton information can be obtained. It may be detected that a person has been detected. The human detection range 51 is determined by the specifications of the motion detection device 10. For example, the horizontal angle is 70 degrees and the distance is in the range of 0.5 to 4.5 m. A person may be detected only in a range narrower than the specifications of the motion detection device 10, and the range for detecting a person may be changed if the specifications of the motion detection device 10 are changed. Further, the number of people that can be detected at the same time is 6, but this can also be changed if the specifications of the motion detection device 10 change.

The operation start detection unit 33 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and the user U intends to operate the application software 120 based on the sensor information acquired by the information acquisition unit 31. Is detected. In the present embodiment, the user U detects the intention to operate by "existing at the operation position 52 + raising the hand". Details will be described later. Further, the operation start detection unit 33 detects that the user U has indicated the intention to end the operation of the application software 120 based on the sensor information. In the present embodiment, the user U detects the intention to end the operation by "leaving from the operation position 52" or "lowering the hand".

The operation area creation unit 34 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and when the operation start detection unit 33 detects the intention of the operation, the user U performs the operation. The operation area 54 is created. The operation area 54 is a range (area) on the three-dimensional space in which the user U can operate the information processing device 30. When the right wrist joint g, the right wrist joint f, and the like of the user U are in the operation area 54, the three-dimensional coordinates of the right wrist joint g and the right wrist joint f are reflected at the position of the cursor 3, and a gesture is detected.

The coordinate conversion unit 35 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and the right wrist joint g of the user U in the operation area 54 based on the sensor information acquired by the information acquisition unit 31. The three-dimensional coordinates of the right wrist joint f are converted to the position of the cursor 3.

The speed detection unit 37 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and the right wrist joint g of the user U in the operation area 54 based on the sensor information acquired by the information acquisition unit 31. And the speed of the right wrist joint f is detected. Either the speed of the right wrist joint g or the right wrist joint f may be detected, but since the user U operates the elbow as the center, the joint closer to the hand can move the cursor 3 faster. On the contrary, the joint far from the hand is easier to control the minute movement amount.

The event determination unit 38 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and determines whether or not the user U has generated an event based on the sensor information acquired by the information acquisition unit 31. to decide. Events are left-click, right-click, double-click, mouse over, mouse down, mouse up, etc. as described above, but general gestures corresponding to these events are not defined. Therefore, the event determination unit 38 associates the movement of the user U with the event as a gesture in advance, and when the user U performs a predetermined gesture, it determines that the associated event has occurred. For example, when the user U makes a gesture of holding a hand in the operation area 54, the event determination unit 38 determines that it is a left click.

The information recording unit 43 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and stores the sensor information in the sensor information DB 44 in chronological order for each user. Since the motion detection device 10 detects the position of the user and the facial expression and face orientation of the user as described above, it records whether or not the user is looking at the display 310 to the extent that it can be estimated where the user is looking at the display 310. it can. In addition, the information recording unit 43 acquires a screenshot of the content displayed on the display 310 from the application software 120. By making the timing of recording the sensor information and the timing of taking the screenshot almost the same, it becomes easier for the content provider and the like to identify the content that the user is interested in.

The output unit 39 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and acquires the position of the cursor 3 converted by the coordinate conversion unit 35 and the event determined by the event determination unit 38. And output to the application software 120. The output unit 39 also has a function of acquiring the status of the application software 120. For example, when the application software 120 displays a moving image on the full screen, the information acquisition unit 31 does not acquire the sensor information because the user does not need to operate it. In addition, the output unit 39 notifies the application software 120 when the user is currently operating. As a result, the application software 120 does not shift to the full-screen moving image display during the user's operation, so that it is possible to prevent the user from suddenly becoming unable to operate.

The UI control unit 36 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like. The UI control unit 36 further includes a cursor display unit 41 and an icon display unit 42. The cursor display unit 41 displays the cursor 3 by using the function of the OS 130, or displays the guide mesh 4 instead of the normal cursor 3. Further, the icon display unit 42 displays the operation reception support icon 81 on the display 310. The icon display unit 42 changes the display mode of the operation reception support icon 81 according to the state of the user U. The operation reception support icon 81 functions as a user interface that assists the user U to easily operate the application software 120 by gestures. For example, the motion detection device 10 guides the user to the operation position 52 so that all the skeleton information of the user can be acquired. In addition, it is clearly indicated who the user has the operation right (even if a plurality of person icons 82 are displayed, the person icon 82 raising the hand has the operation right). The operation right is the right to operate the application software 120, and is granted to only one person at the same time.

<Human detection range 51 and operation position 52>
The human detection range 51 and the operation position 52 will be described with reference to FIG. FIG. 8 is an example of a diagram illustrating a person detection range 51 and an operation position 52 with respect to the motion detection device 10. As described above, the motion detection device 10 detects a person within a predetermined distance (for example, 0.5 to 4.5 m) within a range of a predetermined horizontal angle (for example, 70 degrees). As a matter of course, the operation position 52 is within the human detection range 51.

The operation position 52 is, for example, near the center of the human detection range 51, but may be anywhere inside the human detection range 51. The reason why the operation position 52 is intentionally determined is to prevent any user U who passes through the human detection range 51 from operating the application software 120 by raising his / her hand. Further, if the operation position 52 is too wide, the number of users U who can acquire the operation right increases, and it becomes difficult to determine whether or not the user U can operate the operation position 52. Further, since the operation position 52 is determined, the user U can grasp that the application software 120 can be operated by moving to the operation position 52. It is preferable that the actual operation position 52 is clearly indicated in a color different from that of other places, or that characters such as "Please operate here" are displayed.

As will be described later, the user moves so that the person icon 82 enters the operation position instruction display 83 of the operation reception support icon 81. When the person icon 82 enters the operation position instruction display 83, the user is present at the operation position 52. Therefore, the user is guided to the operation position 52 by the person icon 82, the operation reception support icon 81, and the operation position instruction display 83.

However, for example, it is assumed that the user deviates from the operation position 52 during the operation. For example, there is a case where the display 310 is approached to look closely.

FIG. 9 is an example of a diagram showing the detection range 51 in three dimensions. In FIG. 9A, the detection range 51 includes all of the user's skeleton information. Therefore, the motion detection device 10 can acquire all the skeleton information and send it to the gesture operation reception program 110. In other words, the user can operate the digital signage by the operation.

On the other hand, in FIG. 9B, a part of the user's body protrudes from the detection range 51. The motion detection device 10 sends only the acquired skeleton information to the gesture operation reception program 110. For example, each function of the gesture operation reception program 110 (operation start detection unit 33, operation area creation unit 34, coordinate conversion unit 35, UI control unit 36, speed detection unit 37, event determination unit 38) provides hand skeleton information. Although it is used, skeleton information cannot be acquired if it is not included in the sensor information. Therefore, it can be detected that the desired skeleton information has not been acquired.

In such a case, the UI control unit 36 can move the user to the operation position 52 by displaying the message 540 on the display 310 as shown in FIG. 9C. In FIG. 9C, the message “Please away from the screen” 540 is displayed. Therefore, when the position of the user is displaced to the extent that important (necessary for control) skeletal information is missing, the UI control unit 36 can display a guide and guide the user into the detection range 51.

Although FIG. 9 shows an example in which the user is too close to the display 310, when at least a part of the user's body including the joints protrudes from the detection range 51, the gesture operation reception program 110 is within the detection range 51. Can guide the user.

<Detection of operation intention>
The detection of the intention of operation and the operation area 54 will be described with reference to FIG. FIG. 10 is an example of a diagram for explaining the detection of the intention of operation and the operation area 54. FIG. 10A shows the initial operation performed by the user U at the operation position 52. When the user U reaches the operation position 52, he first raises his hand. User U already knows if he is instructed to perform this operation. The reason for detecting the intention to start the operation is that when the operation is accepted from any hand movement of the user U in the detection range 51, the operation not intended by the user is sent to the application software 120.

The operation start detection unit 33 detects the intention to start the operation when the right wrist joint g is higher than the height of the head joint a. The hand raised by the user U may be right or left. The operation start detection unit 33 preferentially detects the right hand, ignores even if the left hand is raised after raising the right hand, and ignores even if the right hand is raised after raising the left hand.

Further, the reason why the operation start detection unit 33 uses the fact that the right wrist joint g is higher than the height of the head joint a as the detection criterion of the intention to start the operation is that the user U generally does not reach out more than the head. Because. For example, when talking on a cell phone or touching glasses, the hands remain lower than the head. Therefore, it becomes easy to suppress false detection that there is an intention to operate the application software 120 due to the operation of a person who does not have the intention to operate the application software 120.

<Operation area 54>
The operation area creation unit 34 creates the operation area 54 using the lengths of the right shoulder joint d and the head joint a (hereinafter referred to as A). As shown in FIG. 10B, a circle 53 having a center on the right shoulder is created with the length A as the diameter. Then, as shown in FIG. 10C, the circumscribed rectangle of the circle 53 is created. This circumscribing rectangle is the operation area 54. Although the length A is about 30 cm, it is just an example and may be a length that can be reached by the user U.

By creating the operation area 54 from the circle 53 whose center is above the shoulder in this way, it becomes easier to image the user U's hand from the motion detection device 10. Although it depends on the height at which the motion detection device 10 is installed, when the motion detection device 10 is installed at the position of the waist of the user U, the motion detection device 10 moves to the shoulder when the hand reaches the same height as the shoulder. Cannot be imaged. Therefore, the error between the lengths of the right shoulder joint d and the head joint a becomes large (the length A becomes short or large), and the shape of the operation area 54 becomes unstable.

FIG. 11 is an example of a diagram for explaining the operation area 54 in detail. Let P1 (X1, Y1, Z1) be the three-dimensional coordinates of the head joint a, and P2 (X2, Y2, Z2) be the three-dimensional coordinates of the right shoulder joint d. The horizontal direction is the X-axis, the height direction is the Y-axis, and the vertical direction of the paper surface is the Z-axis. The length A is calculated as follows.
A = Y1-Y2
The coordinates of the center O of the circle 53 are (X2, Y2 + A / 2). Therefore, the operation area creation unit 34 may draw a circle 53 having a radius A / 2 centered on the center O. Further, the coordinates of the four vertices C1 to C4 of the operation area 54 can be calculated as follows without obtaining the circle 53. Although the Z coordinate is omitted, for example, Z1, Z2 or their average is used.
C1 = (X2-A / 2, Y2 + A)
C2 = (X2 + A / 2, Y2 + A)
C3 = (X2 + A / 2, Y2)
C4 = (X2-A / 2, Y2)
By creating the operation area 54 in this way, the user U can easily control the position of the cursor 3. Since the user U keeps his hand raised (it loses the operation right when lowered), it inevitably operates with the elbow as a reference. Since the position of the shoulder is almost fixed, the operation area 54 created above the shoulder can be operated around the elbow. Therefore, the user U can operate the cursor 3 by moving the right wrist joint f and the right wrist joint g around the elbow without searching for where the operation area 54 is.

<Non-operation operation without operation right>
When a user talks on a mobile phone or touches glasses, the user often raises his or her hand above the head. Therefore, it is more preferable that the operation start detection unit 33 detects the non-operation operation of the user as follows.

FIG. 12 is an example of a diagram illustrating a user's non-operation operation. The non-operation operation is an operation similar to an operation (for example, raising a hand) indicating the user's intention to operate the digital signage. For example, the user talks on a mobile phone, touches glasses, puts his hand on his forehead, scratches his head, and so on.

As shown in FIG. 12A, the operation start detection unit 33 defines a cube 55 centered on the head joint a. This definition may be referred to as setting, and specifically means specifying the coordinates of the cube 55. The size of the cube 55 is slightly larger than that of the head and face, and is large enough for the user's hand to touch the face. It may be variable or fixed for each user. Further, it may be a rectangular parallelepiped, a sphere or an ellipsoid instead of a cube.

When the user performs a non-operational operation, the user's hand often falls into this cube. Therefore, the operation start detection unit 33 can determine whether or not the operation is a non-operation operation depending on whether or not the hand enters the cube 55.

As shown in FIG. 12B, when the right wrist joint g does not enter the cube 55, the operation start detection unit 33 determines that the operation is not a non-operation operation. In this case, since the hand of the user U is raised, the operation start detection unit 33 grants the operation right.

As shown in FIG. 12C, when the right wrist joint g enters the cube 55, the operation start detection unit 33 determines that the operation is not an operation. In this case, even if the operation start detection unit 33 raises the hand, the operation right is not granted because the hand is inside the cube 55.

Therefore, it is possible to suppress erroneous detection that the user U who has no intention of operating is intended to operate.

FIG. 13 is an example of a side view of the cube 55 of FIG. Since it is a cube 55, the cube 55 has a depth in the distance direction (Z direction) from the motion detection device 10. Since the hand of the user U1 is not in the cube 55, it is determined that it is not a non-operational operation, and the hand of the user U2 is determined to be a non-operational operation because it is in the cube 55.

In this way, since the judgment is made by the cube 55, when the user raises his / her hand so as not to enter the depth of the cube 55, the operation start detection unit 33 grants the operation right to the user, and the coordinate conversion unit 35 gives the right wrist joint g or the like. Coordinates can be detected. On the other hand, when the user raises his / her hand so as to enter the depth of the cube 55, the operation start detection unit 33 does not grant the user the operation right even if the user raises his / her hand. Therefore, rather than simply determining whether or not the hand is near the face on the XY plane, the cube 55 makes it easier to distinguish between the non-operational operation and the operation indicating the intention to operate.

<Coordinate transformation>
Subsequently, the conversion from the coordinates of the wrist joint or the wrist joint to the position of the cursor 3 in the operation area 54 will be described with reference to FIG. FIG. 14 is an example of a diagram showing the coordinates of the wrist joint or the wrist joint and the position of the cursor 3 in the operation area 54. Note that FIG. 14 describes a method of determining the position of the cursor 3 when the speed of the hand is equal to or higher than the threshold value.

The coordinate conversion unit 35 associates the three-dimensional coordinates of the wrist joint or wrist joint in the operation area 54 with the position of the cursor 3 on the display 310 on a one-to-one basis. First, the coordinate conversion unit 35 determines the apex C1 of the upper left corner of the operation area 54 as the origin (0,0). Then, the three-dimensional coordinates of the wrist joint or the wrist joint in the operation area 54 are converted into the three-dimensional coordinates based on the origin (0,0). Since the operation area 54 is a two-dimensional plane of the XY plane, the Z coordinate of the wrist joint or the wrist joint is not used. Therefore, the Z coordinate may be regarded as always constant and may be omitted from the calculation. Let R (X, Y) be the three-dimensional coordinates of the wrist joint or wrist joint with C1 as the origin.

Further, the vertical resolution of the display 310 is H, and the horizontal resolution is W. The vertical and horizontal lengths of the operation area 54 are A. Therefore, the position S (x, y) of the cursor 3 on the display 310 can be calculated by a simple proportional calculation.
x = W × A / X
y = H × A / Y
Of course, the wrist joint or wrist joint of the user U may protrude outside the operation area 54, but the coordinate conversion unit 35 ignores the wrist joint or wrist joint outside the operation area 54.

The coordinate conversion unit 35 performs coordinate conversion each time the information acquisition unit 31 acquires sensor information. On the other hand, the operation area creation unit 34 updates the operation area 54 for each frame of the image data. As a result, an appropriate operation area 54 can be created even if the user faces the front or the shoulder is directed to the motion detection device 10 and the motion changes.

The operation area 54 is deleted when the operation start detection unit 33 detects the intention to end the operation (when the user U lowers his / her hand).

FIG. 15 is a side view of the three-dimensional coordinates of the wrist joint or wrist joint of the user U. That is, the lateral direction of FIG. 15 corresponds to the Z axis. User U1 in FIG. 15 is moving his arm near his body, and user U2 is moving his arm in front of his body. That is, the Z coordinates of the wrist joints or wrist joints of the users U1 and U2 are different. However, the Y coordinates of the wrist joints or wrist joints of the users U1 and U2 are almost the same (although it is not clear from FIG. 15, the X coordinates are also almost the same).

Even if the Z coordinates are different in this way, the coordinate conversion unit 35 does not use the Z coordinates when converting the coordinates of the wrist joint or the wrist joint to the position of the cursor 3, so that the position of the cursor 3 operated by the users U1 and U2 The (trajectory) is the same.

<< Modification example of operation area 54 >>
Generally, the aspect ratio of the display 310 is longer in the horizontal direction than in the one-to-one ratio. On the other hand, the vertical and horizontal lengths of the operation area 54 are both length A. Therefore, when the user U moves his / her hand in the horizontal direction, the cursor 3 moves greatly in the horizontal direction, so that it may be difficult for the user U to control the position of the cursor 3 in the horizontal direction. Therefore, it is effective to change the vertical and horizontal lengths of the operation area 54 according to the aspect ratio of the display 310. The aspect ratio of the display 310 is input to the information processing device 30.

FIG. 16 is an example of a diagram illustrating a modified example of the operation area 54. FIG. 16A shows an example of the operation area 54 when the aspect ratio of the display 310 is horizontally long. The operation area creation unit 34 creates the operation area 54 by adding the expansion areas 54a to the left and right of the square operation area 54 described above. The length α of the expansion region 54a is, for example, 10 cm. This is because if the length A is 30 cm, the user U can reach the range of about 50 cm in total. Therefore, 10 cm is only an example, and it may be expanded within reach in consideration of the length A. For example, it may be variable according to the height of the user U.

FIG. 16B shows an example of the operation area 54 when the aspect ratio of the display 310 is vertically long. The α may be the same as in FIG. 16 (a). By changing the vertical and horizontal lengths of the operation area 54 according to the aspect ratio of the display 310 in this way, the operability of the user U is improved.

<Speed detection>
The speed detection will be described with reference to FIG. FIG. 17 is an example of a diagram illustrating a method of detecting the speed of a wrist joint or a wrist joint. The speed detection unit 37 calculates the acceleration using the three-dimensional coordinates of the wrist joint or the wrist joint in the past several cycles. For example, the wrist joint or velocity V ₁ of the wrist of the time t1 can be obtained from the three-dimensional coordinates of the wrist joint or wrist of the time t0 and t1. The cycle in which the sensor information is acquired is defined as Δt. As described above, the velocity V1 at time t1 is calculated using only the X and Y components.
Vx = (X _t1- X _to ) / Δt, Vy = (Y _t1- Y _to ) / Δt,
V ₁ = √ (Vx ² + Vy ² )
Similarly, the wrist joint or wrist joint velocity V2 at time t2 can be obtained from the three-dimensional coordinates of the wrist joint or wrist joint at times t1 and t2. The acceleration α at time t2 is a value obtained by dividing the difference between the velocities V1 and V2 by the period Δt, and can be calculated as follows.
α = (V _2- V ₁ ) / Δt
Therefore, if there are three-dimensional coordinates of the past three cycles, the speed detection unit 37 can calculate the acceleration of the wrist joint or the wrist joint for each cycle.

In this embodiment, speed and acceleration are used as indexes of hand speed. The thresholds to be compared with these speeds shall be determined experimentally. For example, the threshold value is a speed slightly faster than stopping (for example, 1 to 10 [cm / sec]). The reason for clicking the operation button 103 is that the speed of the wrist joint or the wrist joint is often slow.

When the speed is used as the speed of the wrist joint or the wrist joint, the speed detection unit 37 can calculate the speed by using the three-dimensional coordinates of the past two cycles. Further, the acceleration of the acceleration may be further calculated and used as an index of speed. In this case, it can be calculated by using the three-dimensional coordinates of the past four cycles.

<Event detection>
An event detection method will be described with reference to FIG. FIG. 18A shows the joints and the shapes of the hands included in the hand portion, and FIG. 18B shows the joints and the shapes of the hands included in the hand portion in the state where the user U holds the hand. As an example, the event determination unit 38 detects as an event that the user U holds a hand and clicks a gesture that opens quickly. In other words, when you hold it (goo), the left mouse button is down, and when you open (par), the left button is up. By doing this, the user can click and drag while holding. Further, when the holding time is long, the event determination unit 38 detects an event of right button down.

As described above, the motion detection device 10 has three-dimensional coordinates of the right wrist joint g, the right wrist joint f, the right wrist joint h, and the right thumb joint i (and the left wrist joint, the left wrist joint, the left wrist joint, and the left thumb joint). Is being detected. As shown in the figure, when the user U holds the right hand, the three-dimensional coordinates of the right hand joint h and the right thumb joint i move significantly in a specific direction. When the X and Y coordinates are taken as shown in the figure, the Y coordinate of the right wrist joint h becomes smaller. In addition, the X coordinate of the right thumb joint i also becomes smaller.

The event determination unit 38 determines that the left button down event has occurred when the Y coordinate of the right hand joint h becomes small and the X coordinate of the right thumb joint i also becomes small. When the Y coordinate of the right hand joint h becomes large and the X coordinate of the right thumb joint i also becomes large, it is determined that the left button dup event has occurred. It may be determined that the event of left button down has occurred when the Y coordinate of the right hand joint h becomes smaller or the X coordinate of the right thumb joint i also becomes smaller.

Further, since the motion detection device 10 detects a so-called rock-paper-scissors (rock-paper-scissors, choki, par) gesture, it may be used to detect a grip gesture. For example, when a gesture of goo or choki is detected, it is determined that an event of left button down has occurred.

Further, the event determination unit 38 may detect a gesture of grasping using the Z coordinate instead of using only the two-dimensional coordinate among the three-dimensional coordinates. When the user U clicks the application software 120, he / she moves his / her hand forward (or backward), for example. The event determination unit 38 detects a change in the three-dimensional coordinates of the right wrist joint g or the like in the Z-axis direction at a speed equal to or higher than a predetermined speed, and determines that an event called a click has occurred.

However, when the user U tries to move his / her hand in the Z direction, the X and Y coordinates also move easily, so that the position of the cursor 3 may move from the position to be clicked. Therefore, as in the present embodiment, the gesture of holding a hand whose X-coordinate and Y-coordinate are difficult to change is effective. Therefore, gestures other than holding and opening the hand, such as moving a finger, may be associated with the click event.

<Coordinate transformation when the speed is less than the threshold>
The coordinate transformation when the speed is less than the threshold value will be described with reference to FIG. FIG. 19 is an example of a diagram illustrating coordinate transformation when the speed is less than the threshold value. When the speed detection unit 37 determines that the speed of the wrist joint or the wrist joint is less than the threshold value, the coordinate conversion unit 35 and the UI control unit 36 operate as follows.
(i) The cursor display unit 41 displays the guide mesh 4 at the position of the cursor 3.
(ii) Determine whether the three-dimensional coordinates have changed above the threshold (whether the wrist joint has moved above the threshold).
(iii) When the three-dimensional coordinates change by the threshold value or more, it is determined which component is larger in the X direction or the Y direction.
(iv) In the X direction or the Y direction, whichever has the larger component, the guide mesh 4 is moved along the guide mesh 4 by the number of squares according to the amount of movement.

(i) will be described. When the cursor display unit 41 acquires the information that the speed detection unit 37 has determined that the speed is less than the threshold value, the cursor display unit 41 starts displaying the guide mesh 4. As shown in FIG. 19A, the cursor display unit 41 associates the position of the cursor 3 acquired from the coordinate conversion unit 35 with a predetermined intersection of the guide mesh 4, and arranges the cursor 3 at the intersection to guide the guide mesh 4. Is displayed. A predetermined intersection is predetermined at any of substantially the center of the guide mesh 4. Therefore, the cursor 3 is displayed at substantially the center of the guide mesh 4. Further, the cursor 3 displayed together with the guide mesh 4 has a different shape from the cursor 3 when the speed is equal to or higher than the threshold value. In FIG. 19A, the vertical and horizontal double-ended arrows intersect vertically.

The cursor 3 moves on the intersection of the guide mesh 4. The guide mesh 4 is erased when the speed exceeds the threshold value. The guide mesh 4 may be moved or fixed while the speed is below the threshold.

Each square of the guide mesh 4 may be square, but the vertical and horizontal lengths may be determined by the same ratio as the aspect ratio of the display 310.

Next, the reason for making the determination as in (ii) is to prevent the position of the cursor 3 from changing due to a slight hand movement that the user U does not intend. Further, when the user U makes a gesture of holding a click while the cursor 3 is on the operation button 103 of the application software 120, the hand often makes a movement other than the gesture. In this case, when the gesture is made, the cursor 3 moves out of the operation button 103, and the user U may not be able to click the operation button 103. Therefore, by making a judgment as in (ii), it is possible to prevent the position of the cursor 3 from being changed by a slight hand movement that the user U does not intend.

In FIG. 19B, it is assumed that the three-dimensional coordinates have changed by the threshold value or more. That is, it is assumed that the coordinates of the wrist joint or the wrist joint have moved from the coordinates (X1, Y1) to (X2, Y2). The cursor display unit 41 calculates the movement amounts ΔX and ΔY in the X direction and the Y direction, respectively.
ΔX = X2-X1
ΔY = Y2-Y1
The cursor display unit 41 determines whether at least one of ΔX and ΔY is equal to or greater than the threshold value. The threshold value is determined in consideration of the general amount of movement of the wrist joint or wrist joint that causes the user U to stagger even when trying to stand still, and the error of the motion detection device 10. For example, it may be set to several cm, but the person in charge or the like can determine it experimentally as appropriate.

(iii) will be described. When at least one of the movement amounts ΔX and ΔY is equal to or greater than the threshold value, the cursor display unit 41 determines which of ΔX and ΔY is larger. In FIG. 19B, ΔX> ΔY.

(iv) will be described. The cursor display unit 41 determines the position of the cursor 3 using only X2 of the wrist joint or wrist joint coordinates (X2, Y2). For the position of the cursor 3 in the guide mesh in the y direction, the value immediately before is used as it is. Since the x-coordinate of the position of the cursor 3 converted by the coordinate conversion unit 35 is not always on the intersection of the mesh guides, the cursor display unit 41 corrects the x-coordinate on the intersection of the mesh guides. For example, the correction is made as follows.
-Correct to the intersection of the mesh guide where the x-coordinate of the cursor 3 position is the closest-Correct to the intersection of the mesh guide next to the x-coordinate of the cursor 3-position-The mesh guide before the x-coordinate of the cursor 3 position Correcting to the intersection By correcting in this way, the cursor display unit 41 can display the cursor 3 displayed together with the mesh guide on the intersection. In FIG. 19C, the cursor 3 is displayed by moving to an intersection that is moved to the right by 2 squares from the position of the cursor 3 immediately before.

<Switching wrist joint or wrist joint>
The user U may want to move the cursor 3 large, or may want to move the cursor small, such as near the operation button 103. Further, since the user U moves his / her hand around the elbow, the joint on the outside of the hand moves faster. Therefore, the coordinate conversion unit 35 switches the joint (part of the user's body) to be converted to the position of the cursor 3 depending on whether or not the speed detection unit 37 detects a speed equal to or higher than the threshold value.
A. When the speed is equal to or greater than the threshold value, the three-dimensional coordinates of the right wrist joint g are converted to the position of the cursor 3.
B. When the speed is less than the threshold value, the three-dimensional coordinates of the right wrist joint f are converted to the position of the cursor 3.

As a result, when the user U wants to move the cursor 3 greatly, the user U can easily move the cursor 3 quickly, and can easily move the cursor 3 to the intended position by using the wrist near the operation button 103 or the like. In the case of A, the three-dimensional coordinates of the right hand joint h and the right thumb joint i may be used. Further away from the elbow, the user can operate the cursor 3 faster.

<About the operation reception support icon>
The operation reception support icon 81 displayed by the icon display unit 42 will be described with reference to FIGS. 20 to 22. FIG. 20 is an example of a diagram showing an operation reception support icon 81 when the user U is detected by the motion detection device 10. The icon display unit 42 first causes the display 310 to display the operation reception support icon 81 in a large size. The operation reception support icon 81 has a shape substantially similar to the detection range 51 of the motion detection device 10.

The operation reception support icon 81 may be always displayed or may be displayed only when a person is detected, but is displayed in a large size when a person is detected. As a result, the user U can recognize that the motion detection device 10 has detected the detection. It also has the effect of directing the user's eyes to the display 310. The icon display unit 42 may not only display the operation reception support icon 81 in a large size, but may also display it by blinking it or changing its color. In addition, the operation reception support icon 81 may be displayed on a projector 5 different from the display 310 for phorogram projection (stereoscopic viewing).

In FIG. 20, the user U has entered the detection range 51 from the right rear side of the detection range 51 of the motion detection device 10. Since the information acquisition unit 31 has acquired the three-dimensional coordinates of the user U, the person icon 82 is displayed at a position (referred to as an icon upper position) corresponding to the position of the user U on the operation reception support icon 81. Since the person icon 82 is displayed at the position above the icon, the user U can grasp his / her position in the detection range 51. Further, since the information acquisition unit 31 has acquired the face orientation of the user U, the icon display unit 42 displays the eye position estimated from the face orientation on the person icon 82.

The location corresponding to the operation position 52 (hereinafter referred to as the operation position instruction display 83) is highlighted on the operation reception support icon 81, and the user U so that the person icon 82 enters the operation position instruction display 83. Moving.

FIG. 21 is an example of a diagram illustrating a method of calculating the position of the person icon 82 on the operation reception support icon 81. FIG. 21A shows an operation reception support icon 81, and FIG. 21B shows a top view of the detection range 51. The motion detection device 10 detects the coordinates of each joint of the user. For example, the user's X1 coordinates and Z1 coordinates can be specified with the center of the motion detection device 10 as the origin O. That is, the relative position (first relative position) of the user with respect to the motion detection device 10 is specified. The distance D from the center O and the direction θ with respect to the front are also known.

On the other hand, the operation reception support icon 81 has a similar (or similar) shape to the detection range 51. The origin O is the position of the operation reception support icon 81 (center of the upper side in the figure) corresponding to the position where the motion detection device 10 is installed. The icon display unit 42 converts the above relative position into a relative position (second relative position) of the operation reception support icon 81 with respect to the origin O, and displays the person icon 82 at the converted relative position.

Let A [m] be the distance that the detection range 51 detects the user. On the other hand, the length a [pixel] of the side corresponding to the distance A of the operation reception support icon 81 is known. Therefore, the coordinates (x1, z1) of the person icon 82 in the operation reception support icon 81 can be calculated using the a / A ratio.
x1 = X1 x a / A
z1 = Z1 × a / A
The icon display unit 42 updates the position of the person icon 82 every time the information acquisition unit 31 periodically acquires sensor information.

FIG. 22 is an example of a diagram showing an operation reception support icon 81 when the user U raises his / her hand at the operation position 52. Note that FIG. 22 shows an example of a screen when the speed of the wrist joint or the wrist joint is equal to or higher than the threshold value. The icon display unit 42 displays the operation reception support icon 81 in a small size when a predetermined time elapses after the user U enters the operation position 52 or when a predetermined time elapses after the user U raises his / her hand at the operation position 52. To do. This makes it easier for the user U to see the screen of the application software 120 displayed on the display 310.

Further, when the user U raises his / her hand, the operation start detection unit 33 detects that the user U has raised his / her hand, so that the icon display unit 42 raises the hand of the person icon 82. In other words, the person icon 82 with the right hand raised is displayed. As a result, the operation right is given to the user U who raised his hand. The gesture operation reception program 110 does not accept the operation even if another user U raises his / her hand. In FIG. 22, another user U has entered the detection range 51 from the rear left, but this user U cannot operate the application software 120. That is, the user U (only) who raises his hand earliest at the operation position 52 obtains the operation right.

The user U creates the operation area 54 at the operation position 52 as described above, moves the hand to display the cursor 3, or makes a gesture to generate an event.

FIG. 23A is an example of a diagram showing a guide mesh when the user U raises his / her hand at the operation position 52 but the speed of the wrist joint or the wrist joint is less than the threshold value. In this case, the cursor display unit 41 displays the cursor 3 at the intersection of the guide mesh 4. Further, since the cursor 3 moves only vertically or horizontally, the user U can control the minute movement of the cursor 3 with reference to the square of the guide mesh 4.

The guide mesh 4 does not have to be displayed. Even without the guide mesh 4, the cursor 3 still moves only vertically or horizontally, and the user U can control the minute movement of the cursor 3. However, since the user U can grasp that the cursor 3 can be moved only in units of squares by displaying the guide mesh 4, it is expected that the user U feels easy to operate.

Further, the horizontal line of the guide mesh 4 does not have to be horizontal, and the vertical line does not have to be vertical. For example, vertical and horizontal straight lines may intersect.

Further, as shown in FIG. 23B, the cursor 3 may move in the radial direction on the circumference of the concentric circles.

<Operation procedure>
FIG. 24 is an example of a flowchart showing a procedure until the gesture operation reception program 110 grants the operation right to the user U. The process of FIG. 24 is repeatedly executed, for example, while the information processing device 30 is executing the gesture operation reception program 110.

First, the person detection unit 32 determines whether or not the motion detection device 10 has detected a person (S10). If no person is detected (No in S10), the process of FIG. 24 ends.

When a person is detected (Yes in S10), the icon display unit 42 displays the operation reception support icon 81 in large size on the display 310 (S20). After that, the icon display unit 42 displays the operation reception support icon 81 in a small size after a certain period of time even if a person is detected.

Next, the icon display unit 42 displays the person icon 82 at the position above the icon of the operation reception support icon 81 based on the position of the user U (S30). Since the information acquisition unit 31 periodically acquires the sensor information, the icon display unit 42 updates the position of the person icon 82.

The operation start detection unit 33 determines whether or not the position of the user U has entered the operation position 52 (S40). That is, it is determined whether or not the X, Y, Z coordinates of the user U are included in the operation position 52.

Until the user U enters the operation position 52 (No in S40), it is determined whether or not a person has not been detected (S50). Then, when the person is not detected (Yes in S50), the process of FIG. 24 ends.

While the person is detected (No in S50), it is determined whether or not the user U has repeatedly entered the operation position 52.

Next, when a person enters the operation position 52 (Yes in S40), the operation start detection unit 33 determines whether or not the user U has raised his / her hand (S60). For example, it is determined whether or not the hand is located at a certain (or higher) height than the waist of the user.

The determinations after step S40 are repeatedly executed until the user U raises his hand (No in S60).

When the user U raises his / her hand (Yes in S60), the operation start detection unit 33 determines whether or not the hand is in the predetermined peripheral area (S70). The predetermined peripheral region means, for example, the inside of the cube 55. The determination using the cube 55 is an example of determination as to whether or not the cube is within a predetermined area. For example, confirm that the right wrist joint h and the right wrist joint g are not around the head joint a. Thereby, for example, it is possible to prevent the user U who is on the phone from operating the application software 120. The determination in step S70 prevents the user who has no intention of operating the operation right from being given the operation right, and is not essential. Further, in the determination of step S70, it is determined that the area (peripheral area near the body) of the rectangular parallelepiped (rectangular parallelepiped including a little before the head to the waist) created from the user's head to the waist is not touched. It is preferable to judge. This creates an area that does not depend on the user's posture. Specifically, even in the case where the user is carrying a load such as a binder, not only the area around the head but also the area around the waist can be included in the non-operation area.

If the determination in step S70 is Yes, the operation start detection unit 33 detects the intention to start the operation and grants the operation right to the user U who raised his hand (S80). Therefore, it is possible to prevent the application software 120 from being operated even if another user U raises his / her hand at the operation position 52.

Next, the icon display unit 42 displays the person icon 82 with the hand raised at the position above the icon (S90). As a result, the user U can determine that he / she has raised his / her hand and can start the operation.

When the intention to start the operation is detected, the operation area creation unit 34 creates the operation area 54 (S100). After that, the user U can operate the position of the cursor 3 by the wrist joint and the wrist joint, and can operate the application software 120 by the gesture. The gesture operation reception program 110 starts accepting the operation of the application software 120 by the user U.

FIG. 25 is an example of a flowchart showing a procedure in which the gesture operation reception program 110 accepts the operation of the user U and operates the application software 120. The process of FIG. 25 starts when the operation area 54 is created.

First, the information acquisition unit 31 acquires sensor information (S110).

The speed detection unit 37 determines whether or not the speed of the wrist joint is equal to or greater than the threshold value based on the sensor information (S120).

When the speed of the wrist joint is equal to or greater than the threshold value (Yes in S120), the coordinate conversion unit 35 converts the three-dimensional coordinates of the wrist joint to the position of the cursor 3 (S130).

The output unit 39 outputs the position of the cursor 3 to the application software 120 (S140).

In addition, the event determination unit 38 determines whether or not the user U has made a gesture to generate an event (S150). This is because the user U can make a gesture even if the guide mesh 4 is not displayed.

When an event is detected, the output unit 39 outputs the position of the cursor 3 and the event to the application software 120 (S160). The position of the cursor 3 and the event do not have to be output at the same time.

Next, when the speed of the wrist joint is not equal to or greater than the threshold value (No in S120), the cursor display unit 41 determines whether or not the guide mesh 4 is being displayed (S180). This is because it is not necessary to display the guide mesh 4 again while displaying the guide mesh 4.

When the guide mesh 4 is not being displayed (No in S180), the cursor display unit 41 displays the guide mesh 4 in which the cursor 3 is displayed in the center (S190). The cursor display unit 41 acquires the position of the cursor 3 from the coordinate conversion unit 35. After this, the process proceeds to step S140.

When the guide mesh 4 is being displayed (Yes in S180), the cursor display unit 41 determines whether or not the three-dimensional coordinates of the wrist joint have moved by the threshold value or more (S200). That is, by focusing on the wrist joint instead of the wrist joint, the user U can easily control the minute movement of the cursor 3.

If the determination in step S200 is Yes, the cursor display unit 41 updates the position of the cursor 3 on the guide mesh 4 (S210). That is, the cursor display unit 41 converts only the larger of the movement amounts in the X direction and the Y direction in which the wrist joint has moved to the position of the cursor 3. The cursor display unit 41 corrects the position of the cursor 3 to the intersection of the guide mesh 4. After that, the process proceeds to step S140, and the cursor 3 is displayed on the guide mesh 4.

In addition, the operation start detection unit 33 appropriately determines whether or not the user U is in the operation position 52 and whether or not the hand is raised (S170). This is to confirm the intention to operate.

If the determination in step S170 is Yes, the process returns to step S110 and the operation of the application software 120 is continued.

If the determination in step S170 is No, the output unit 39 requests the application software 120 to hide the guide mesh and the cursor (S220). This completes the process of FIG. 25. In this case, the operation area 54 is deleted and the user U loses the operation right.

<Recording and utilization of sensor information>
Subsequently, the recording and utilization of the sensor information will be described with reference to FIGS. 26 to 29. Conventional digital signage cannot keep a history of whether or not it has been viewed by a user. Since the information processing device 30 of the present embodiment acquires the face orientation of the user from the motion detection device 10, the content provider or the like can analyze how many users have browsed later. Further, since the information recording unit 43 records screenshots of the contents displayed on the display 310, it is possible to analyze what kind of screenshot the user is interested in. Since the data size of the screenshot is large, it is only necessary to record the screenshot when the user's face is facing the display 310.

FIG. 26 shows an example of sensor information recorded by the information recording unit 43 for analysis of face orientation and the like. In FIG. 26, sensor information recorded for one user is described in XML format. This information will be described below.
The <TrackingID> tag is user identification information assigned to a recognized user (including a user who does not have operation authority).
The <Glasses> tag indicates whether you are wearing glasses.
The <Happy> tag indicates whether you are smiling (looks happy).
The <LeftEyeClosed> tag indicates whether the left eye is closed.
The <RightEyeClosed> tag indicates whether the right eye is closed
The <LookingAway> tag indicates whether you are facing this.
The <MouthMoved> tag indicates whether the mouth is moving.
The <MouthOpen> tag indicates whether the mouth is open.
The <Yaw> tag indicates the lateral angle (yaw angle) of the face, and a positive value is obtained when the neck is bent to the left side toward the motion detection device 10, and a negative value is obtained when the neck is bent to the right.
The <Pitch> tag indicates the vertical angle (pitch angle) of the face, and a positive value is obtained when facing upward toward the motion detection device 10, and a negative value is obtained when facing downward.
The <Roll> tag indicates the angle (roll angle) in which the face is tilted, and is zero when the eyes and mouth are vertical.
The <PositionInfo> tag indicates the position of the user, and the <X> tag indicates the lateral distance (plus value on the right side, minus value on the left side) from the center of the motion detection device 10. The <Y> tag indicates the height, and the <Z> tag indicates the distance from the motion detection device 10.
The <LeftButtonDown> tag of the <MouseTrackingInfo> tag indicates whether the left mouse button is pressed, and the <RightButtonDown> tag indicates whether the right mouse button is pressed. The <MousePositionInfo> tag indicates the X and Y coordinates of the cursor.

Note that <MouseTrackingInfo> and <MousePositionInfo> are recorded only for users who have operation authority. The information recording unit 43 records such information for each user.

FIG. 27 is an example of a diagram showing a screenshot 201 acquired by the information recording unit 43 when the sensor information of FIG. 26 is acquired. Therefore, content providers, etc. use the <Yaw>, <Pitch>, and <Roll> tags to measure where the user is looking at the content (where they are interested), and the dwell time of the line of sight. You can do things like that. Also, with the <MouseTrackingInfo> tag, it is possible to determine from screenshot 201 what the user is doing and what he is interested in if he is doing it.

By the way, since the motion detection device 10 uses the distance information to detect the user, the user may not be detected depending on the positions of a plurality of users. FIG. 28 is an example of a diagram illustrating an undetected user. In FIG. 28, it is shown that there are three users U1 to U3 in the detection range 51 of the motion detection device 10. However, since the user U1 and the user U3 overlap each other in the distance direction when viewed from the motion detection device 10, the motion detection device 10 cannot detect the user U3. That is, the skeleton information cannot be acquired.

However, since the motion detection device 10 has captured the color image 202 as described above, the user U3 is shown in the color image 202. In addition, the distance information of the area in which the user U3 is captured is also acquired. The information recording unit 43 determines that the person is a person by, for example, recognizing a face or a silhouette from the color image 202. As a result, the number of users in the detection range 51 can be correctly acquired.

Since the area in which the user U3 appears can be extracted as an area in which the coordinates in the Z direction are within a certain range (for example, within the processing range behind the user U1), the area for performing face recognition or the like is larger than the user U1. The detection time can be shortened by limiting to the area having the Z coordinate behind.

In this way, the number of users that can be acquired at one time is limited only by the skeleton information (currently up to 6 people), but by combining with the color image 202 and distance information, the number of users can be increased even when people are crowded. Can be detected. Further, the color image 202 before the user U3 is blocked by the U1 and the color image after the blockage are compared, and the distance information before the user U3 is blocked by the U1 and the distance information after the blockage are compared. You can also. When the number of users detected by the person detection unit 32 suddenly decreases, the information recording unit 43 estimates that the user is hidden, and can search by face recognition or the like from an area where the distance information is the same as before the hiding.

FIG. 29 is an example of a diagram illustrating an example of analysis of sensor information stored in the sensor information DB 44 by the information recording unit 43. The information processing device 30 totals the number of users for each time zone (10 minutes in the figure) from the sensor information stored in the sensor information DB 44. In FIG. 29, the height of the bar graph 205 shows the number of users for each time zone. Further, in each bar graph 205, the number of users in the time zone is divided into the number 204 of the users who have the operation right and the number 203 of the other users. Since the same <Tracking ID> is assigned to the same user who is continuously detected, there is almost no possibility that the information processing apparatus 30 counts in duplicate.

Further, in FIG. 29, the screenshot 201 of each time zone is displayed above the bar graph 205 like a timeline. Alternatively, a representative screenshot 201 of each time zone may be displayed. The typical screenshot 201 is the first or last screenshot of the time zone, the screenshot with the largest change in the image, and the like.

Therefore, the content provider or the like can determine which content is a collection of many users. For example, in the case of instructors having different contents in FIG. 29 (teachers of schools and preparatory schools, supporters, etc.), it can be determined which instructor is popular.

<Summary>
As described above, when the speed of operation is less than the threshold value, the information processing apparatus 30 of the present embodiment improves the operability of the position of the cursor 3 by limiting the moving direction of the cursor 3 to a predetermined direction. be able to. Further, when the speed of movement is less than the threshold value, the cursor does not move unless the right wrist joint f is moved by the threshold value or more, so that it is easy to prevent the cursor from moving when the user makes a click gesture. In addition, the operability of the user can be improved because the joints of different users can be used for operation depending on whether the speed of operation is equal to or less than the threshold value. Further, since the vertical and horizontal lengths of the operation area 54 are appropriately changed according to the aspect ratio of the display 310, the operability of the cursor 3 by the user can be improved. Further, even when a plurality of users are detected, the operation right can be given to the user who first shows the intention of the operation. Further, by displaying the operation reception support icon 81, the user can easily move to the operation position 52, and it is easy to grasp that he / she is operating.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to Examples, but the present invention is not limited to these Examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, the information processing device 30 may operate in a server-client system. In this case, the information processing device 30 transmits the sensor information to the server, and receives the cursor position and the event from the server. Then, it is sent to the application software 120. A display system having such an information processing device 30 and a server allows a user to operate digital signage.

Further, although it has been explained that the gesture operation reception program mainly displays the cursor 3, the cursor 3 may be displayed by appropriately using the function of the OS 130. Further, the application software 120 may display the cursor 3.

Further, instead of having a one-to-one correspondence between the coordinates in the operation area 54 and the position of the cursor, the user may operate the position of the cursor by the amount of movement of the hand. For example, when moving with the cursor, hold your hand and move it at a certain speed or higher. The coordinate conversion unit converts the movement amount of the hand into the movement amount of the cursor and updates the position of the last cursor. Therefore, in this case, the operation area 54 may not be provided.

Further, although the left-click gesture has been described in this embodiment, it is also possible to input the right-click gesture by detecting the right hand and the left hand separately by the information processing device 30. It is also possible to move the cursor based on the three-dimensional coordinates of joints such as the legs and the direction of the face. In addition to holding hands, a wide variety of human gestures can be associated with events.

Further, although it has been explained that the action of raising the hand is the action of obtaining the operation right, the operation right may be granted by the following actions.
-When a voice or sound (clapping a hand) is detected by the microphone 17-When a voice or sound (clapping a hand) detected by the operation start detection unit 33 is determined to have a specific rhythm-Operation start detection unit 33 When it analyzes the image data and determines that it has a wireless device such as a smartphone ・ When the operation start detection unit 33 detects a signal transmitted from a wireless device such as a smartphone owned by the user ・ When the user has a foot When the operation start detection unit 33 detects that the hand shape has been raised ・ When the operation start detection unit 33 detects that the shape of the hand has become a specific shape that matches a predetermined condition (goo, choki, par, etc.) )
-When the operation start detection unit 33 detects that the user is standing (the operation right can be given when a person sitting at the venue stands at a seminar or the like).
-When the operation start detection unit 33 detects that the facial expression has become a specific facial expression that matches a predetermined condition (for example, laughing).
-When the operation start detection unit 33 detects that the user's line of sight (face orientation) is heading toward a specific area Further, the configuration example shown in FIG. 7 is for facilitating the understanding of the processing by the information processing device 30. In addition, it is divided according to the main functions. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the information processing apparatus 30 can be divided into more processing units according to the processing content. Further, one processing unit can be divided so as to include more processing.

The icon display unit 42 is an example of the position display means, the information acquisition unit 31 is an example of the operation information acquisition means, and the coordinate conversion unit 35 and the event determination unit 38 are examples of the operation reception means.

3 Cursor 4 Guide mesh 10 Motion detection device 30 Information processing device 31 Information acquisition unit 32 Person detection unit 33 Operation start detection unit 34 Operation area creation unit 35 Coordinate conversion unit 36 UI control unit 37 Detection unit 38 Event judgment unit 39 Output unit 41 Cursor display 42 Icon display 43 Information recording 54 Operation area 81 Operation reception support icon 100 Information processing system

JP-A-2015-176451

Claims (9)

An information processing device that acquires position information related to a user's position from a device that detects the user's position around the display device and operation information related to the user's operation.
It has a position display means for converting the position information into a display position in the display device and displaying the user position display information indicating the user's position at the display position.
The position display means displays on the display device guidance position display information for guiding the user to a position where the user operates the contents displayed on the display device according to the operation of the user.
The position display means displays on the display device a detection range in which the device can detect a user.
An operation information acquisition means for acquiring the operation information and
It has an operation receiving means for receiving an operation of application software operating in the information processing device based on the operation information.
When the operation information indicating that the user has raised his / her hand is received from the device, the position display means displays the user position display information in the form of a person raising his / her hand, and the operation receiving means raises his / her hand. An information processing device that gives operation rights only to users who have used it and does not accept operations even if other users raise their hands. Before SL inductive position display information processing apparatus according to claim 1 representing the position of the user the apparatus can detect the operation information. Before SL inductive position display information processing apparatus according to claim 2 in which the user indicating a position that can manipulate the application software. If the device detects a user
The information processing device according to claim 1, wherein the position display means displays the user position display information within the detection range. The position information represents a first relative position with respect to the device.
The position display means converts the first relative position into a second relative position with respect to a predetermined position in the detection range, and displays the user position display information at the second relative position in the detection range. The information processing apparatus according to claim 1 or 4. When the motion information acquisition means acquires the motion information that the speed of the wrist joint or the wrist joint of the user raising the hand is less than the threshold value.
The information processing device according to any one of claims 1 to 5, wherein the position display means displays a guide mesh in which a cursor moves an intersection by the operation of the user. A display system having one or more information processing devices that guide a user based on the position information about the user's position acquired from the device that detects the user's position around the display device and the operation information about the user's operation.
It has a position display means for converting the position information into a display position in the display device and displaying the user position display information indicating the user's position at the display position.
The position display means displays on the display device guidance position display information for guiding the user to a position where the user operates the contents displayed on the display device according to the operation of the user.
The position display means displays on the display device a detection range in which the device can detect a user.
An operation information acquisition means for acquiring the operation information and
It has an operation receiving means for receiving an operation of application software operating in the information processing device based on the operation information.
When the operation information indicating that the user has raised his / her hand is received from the device, the position display means displays the user position display information in the form of a person raising his / her hand, and the operation receiving means raises his / her hand. A display system that gives operation rights only to users who have used it, and does not accept operations even if other users raise their hands. A device that detects the user's position around the display device and detects the position information related to the user's position and the operation information related to the user's operation .
An operation information acquisition means for acquiring position information regarding a user's position and the operation information from the device, and
A position display means that converts the position information into a display position in the display device and displays the user position display information indicating the user's position at the display position.
Operation acceptance means for accepting an operation of application software that operates in the information processing device based on the operation information, and then caused to function the information processing apparatus,
It said position display means displays the derived position display information for the user to guide the user in a position to operate in accordance with contents displayed on the display device to the operation of the user,
When the operation information indicating that the user has raised his / her hand is received from the device, the position display means displays the user position display information in the form of a person raising his / her hand, and the operation receiving means raises his / her hand. A program that gives the operation right only to the user who has the information and does not accept the operation even if another user raises his / her hand.
The display device that displays the detection range in which the device can detect the user,
Display system having a. When installed in the information processing device, the information processing device is installed.
An operation information acquisition means for acquiring the position information regarding the user's position and the operation information from the device for detecting the user's position around the display device and the operation information regarding the user's operation, and
A position display means that converts the position information into a display position in the display device and displays the user position display information indicating the user's position at the display position.
A program that functions as an operation receiving means that accepts operations of application software that operates in the information processing device based on the operation information.
The position display means displays on the display device guidance position display information for guiding the user to a position where the user operates the contents displayed on the display device according to the operation of the user.
The position display means displays on the display device a detection range in which the device can detect a user.
When the operation information indicating that the user has raised his / her hand is received from the device, the position display means displays the user position display information in the form of a person raising his / her hand, and the operation receiving means raises his / her hand. A program that gives operation rights only to users who have used it, and does not accept operations even if other users raise their hands.

Images