JP6699406B2 - Information processing device, program, position information creation method, information processing system - Google Patents

Description

  The present invention relates to an information processing device, a program, a position information creating method, and an information processing system.

  There is known a system that displays information on a display installed in a place where people gather or pass. An information processing device is connected to the display via a network or the like, and by allowing the information processing device to display a still image or a moving image on the display, the information processing device transmits useful information such as advertisement, weather, and news. it can. Such a system or mechanism is called digital signage. The display is installed outdoors, in stores, in public spaces, in transportation facilities, in corridors, etc., and can provide detailed and new information to visitors, visitors, users of facilities (hereinafter simply referred to as users), and the like.

  By the way, among the current digital signage, a large-screen digital signage has a main function of unilaterally displaying information. However, from the user's point of view, since the information changes from moment to moment, it may not be possible to browse slowly even if the user is interested in certain displayed information. Further, from the viewpoint of the information provider, since the reaction from the user cannot be obtained simply by passing the information, it is not possible to grasp what information the user is interested in. In addition, it is difficult for the user to display more detailed information using the information of digital signage as a window, or for the provider to provide detailed information of information that the user is interested in. That is, it is difficult to operate interactively (bidirectionally) with the conventional digital signage.

  In order to realize interactive operation, an input device for the user to input an operation should be prepared near the display, but it is difficult to secure an installation space, or even if there is an installation space, the input device may be damaged. There is a risk of theft. Therefore, an operation mode in which the digital signage detects the user's gesture and accepts the operation is considered (for example, see Patent Document 1). Patent Document 1 discloses a pointing control device that moves a cursor displayed on a display surface based on the detected movement of a user and adjusts the movement of the cursor based on a distance from the user.

  However, there is a problem that it is not always easy for the user to operate the cursor at an arbitrary position by the movement of the hand or arm. This is because the movements of the hands and arms are affected by the unconscious shaking of the human body in addition to the conscious movements of the hands and arms for operating the cursor. Further, the information that detects the position of the user's hand or arm in the space includes an error. Therefore, for example, it is not easy to move the cursor to the position intended by the user or press the operation button at the position intended by the user.

  The present invention has been made in view of the above problems, and an object thereof is to provide an information processing apparatus in which position information can be easily input.

The present invention is an information processing device for creating position information of position indication display information indicating a position indicated by a user displayed on a display device based on motion information related to a user's motion detected by a motion detection device, If the speed detecting means for detecting the speed of the user's operation based on the operation information and the speed detected by the speed detecting means do not match the condition of the predetermined speed, the movement of the position indication information is carried out. Position information creating means for creating the position information by limiting the direction to a predetermined direction, and the position information creating means differs depending on whether the speed detected by the speed detecting means is equal to or more than a threshold value or less than the threshold value. The position information is created using the body part of the user .

  It is possible to provide an information processing device that allows easy input of position information.

It is an example of a diagram for explaining the control of the position of the display and the cursor that the information processing system has. It is an example of a system configuration diagram of an information processing system. It is an example of a hardware configuration diagram of an information processing apparatus. It is an example of a hardware configuration diagram of an operation detection device. It is a figure which shows the joint in which three-dimensional coordinates are obtained. It is an example of a diagram for explaining the relationship between the application software and the gesture operation reception program. It is an example of a functional block diagram in which the functions of a gesture operation reception program are shown in blocks. It is an example of a diagram for explaining a detection range and an operation position of a person with respect to the motion detection device. It is an example of the figure which shows a detection range three-dimensionally. It is an example of a diagram illustrating an operation area. It is an example of a diagram for explaining the operation area in detail. It is an example of a diagram for explaining a non-operation operation of the user. It is an example of the figure which looked at the cube of FIG. 12 from the side. It is an example of a diagram showing coordinates of a wrist joint or a wrist joint and a position of a cursor in an operation area. It is the figure which looked at the three-dimensional coordinates of the wrist joint of a user U or a wrist joint from the side. It is an example of the figure explaining the modification of the operation area. It is an example of a diagram for explaining a method of detecting the speed of the wrist joint or the wrist joint. It is an example of a diagram illustrating a method of detecting an event. It is an example of a diagram for explaining the coordinate conversion when the speed is less than the threshold value. It is an example of a diagram showing an operation reception support icon when a user is detected by the motion detection device. It is an example of a diagram showing an operation reception support icon when the user raises his/her hand at the operation position. It is an example of a diagram showing a guide mesh when the user raises his/her hand at the operation position but the speed of the wrist joint or the wrist joint is less than a threshold value. It is an example of a flowchart showing a procedure until the gesture operation reception program gives the operation right to the user. It is an example of a flow chart showing a procedure in which a gesture operation reception program receives a user's operation and operates application software. It is a figure which shows an example of the sensor information which an information recording part records for analysis, such as a face direction. It is an example of the figure which shows the screenshot which the information recording part acquired when the sensor information of FIG. 25 was acquired. It is an example of the figure explaining the user who is not detected. It is an example of a diagram for explaining an analysis example of the sensor information stored in the sensor information DB by the information recording unit.

  Hereinafter, an information processing system 100 for implementing the present invention and a position information creating method performed by the information processing system 100 will be described with reference to the drawings.

<Outline of information processing system>
The schematic features of the information processing system 100 will be described with reference to FIG. FIG. 1 is an example of a diagram illustrating control of the positions of the display 310 and the cursor 3 included in the information processing system 100.

  As shown in FIG. 1A, a list of contents 101 and a moving image 102 are displayed on a display 310, and various operation buttons 103 (reproduction, stop, etc. of the moving image 102) for operation input are displayed. . When the motion detection device 10 detects the user U, the information processing device 30 displays the operation reception support icon 81. A person icon 82 representing the user U is displayed on the operation reception support icon 81, and the user U can operate while understanding that his/her standing position and motion are detected.

  The user U performs a predetermined action for operating the cursor 3 in front of the action detection device 10. The motion detection device 10 detects the three-dimensional coordinates of the joint of the user U and transmits it to the information processing device 30. The information processing device 30 converts three-dimensional coordinates of, for example, a wrist joint or a wrist joint into a position of the cursor 3 on the display 310 and displays the cursor 3 at this position.

  The information processing apparatus 30 of the present embodiment switches the method for determining the position of the cursor 3 depending on the “speed” of the operation of the user U. First, when the operation speed is equal to or higher than the threshold value, the information processing device 30 converts the three-dimensional coordinates of the hand or wrist into the position of the cursor 3. That is, the three-dimensional coordinates and the position of the cursor 3 have a one-to-one correspondence. Such a method of determining the position of the cursor 3 is hereinafter referred to as a "normal method of determining the position of the cursor 3."

  Next, a case where the operation speed is less than the threshold will be described with reference to FIG. In this case, the information processing device 30 classifies the movement direction of the user's hand or wrist into one of up, down, left and right, and moves the cursor 3 in a grid pattern. As shown in FIG. 1B, when the operation speed is less than the threshold value, the information processing device 30 displays the guide mesh 4. The current position of the cursor 3 is highlighted on the guide mesh 4. Even if the user U moves the wrist joint or the wrist joint diagonally, the information processing apparatus 30 determines that the information processing apparatus 30 has moved vertically or horizontally, and moves the cursor 3 only in the vertical and horizontal directions.

  In this way, the user controls the position of the cursor 3 and makes a click gesture (described later). For example, the operation button 103 can be left-clicked or the content list 101 to be scrolled can be selected.

  The method of determining the position of the cursor 3 whose operation speed is less than the threshold limits the moving direction of the cursor 3. However, even if the user U tries to move slightly so that the movement amount of the cursor 3 becomes minute, the user's own body may shake, so that the wrist joint or wrist joint can be accurately moved in the direction intended by the user U. is not. For example, there is a case in which the intention to move in the diagonal direction is a movement in the vertical direction or the horizontal direction, and the intention to move in the vertical direction or the horizontal direction is a movement in the diagonal direction. Moreover, although there is an error in the three-dimensional coordinates detected by the motion detection device 10, when the user U slightly tries to move, this error becomes difficult to ignore. Therefore, even if the three-dimensional coordinates are directly converted to the position of the cursor 3, it is difficult to move the cursor 3 to the position intended by the user U.

  On the other hand, when the movement direction of the cursor 3 is limited to the vertical and horizontal directions, the cursor 3 is likely to move in the direction intended by the user U. For example, when the user wants to move to the right, even if the body of the user U shakes or the wrist joint or wrist joint does not move in the intended direction, the wrist joint or wrist moves upward, downward, or leftward. Joints rarely move. Further, even if there is a detection error of the motion detection device 10, it affects only the vertical and horizontal directions. Therefore, there is an advantage that the user U can easily move the cursor 3 in the intended direction.

  As described above, the information processing system 100 according to the present embodiment estimates that the user U moves the cursor 3 slightly when the operation speed of the user U is less than the threshold value, and sets the moving direction of the cursor 3 to the predetermined direction. By limiting the direction, it is possible to improve the operability when inputting the position of the cursor 3.

<About terms>
The motion information is information indicating the motion of the user. Alternatively, it is information converted into a form in which the user's action can be interpreted by the information processing device. Alternatively, it is information for non-contact input of the user's action into the information processing device. Specifically, it is a position, a posture, a movement of a limb, and the like, and in the present embodiment, the term sensor information will be described.

  The position designation display information is a display object that displays the position designated by the user on the display. The display object is a concept including an image, a number, a character, a symbol, and a figure, and these two or more kinds of display objects may be combined and displayed. Generally, the position is designated by some pointing device such as an input device such as a mouse (device) or a trackball. Specifically, it is called a cursor, a mouse, a mouse cursor, a mouse pointer, a move key, or the like. In this embodiment, the term cursor is used as an example.

  The gesture means a predetermined operation among the operations performed by the user. That is, the specific action is called a gesture. As an example, the gesture is used as an event for operating the information processing device.

  The speed of movement is defined by the amount of movement per unit time, the amount of movement of a predetermined part of the user per unit time, and the like. In this embodiment, the speed of the hand will be described as an example. Further, the hand includes a wrist and a portion beyond the wrist, but may include an elbow or a portion between the elbow and the wrist.

  The condition that matches the predetermined speed condition is a condition for detecting the user's intention regarding the control of the position of the cursor. This intention means an intention to control the position with high accuracy. The case where the conditions are met includes, for example, that the speed is less than a threshold value, that the speed is within a predetermined range, and the like. In the following, the case where the speed is less than the threshold will be described as an example.

<System configuration example>
FIG. 2 is an example of a system configuration diagram of the information processing system 100. As shown in FIG. 2A, the information processing system 100 includes a display 310, a motion detection device 10, and an information processing device 30. The motion detection device 10 is connected to the information processing device 30 via a cable such as a USB cable or IEEE1394, and the information processing device 30 and the display 310 are DisplayPort (registered trademark), DVI (Digital Visual Interface), and HDMI (registered trademark. High. -Connected via a cable capable of communication according to standards such as Definition Multimedia Interface (VFD) and VGA (Video Graphics Array). The motion detection device 10 and the information processing device 30 may communicate with each other wirelessly such as a wireless LAN, Bluetooth (registered trademark), ZigBee (registered trademark). Similarly, the information processing device 30 and the display 310 may communicate with each other wirelessly such as a wireless LAN and a wireless standard of HDMI.

  As the information processing device 30, a general-purpose device such as a PC (Personal Computer), a tablet terminal, a smartphone, a mobile phone, and a PDA (Personal Digital Assistant) can be used. An OS (Operating System) is operating in the information processing device 30 and has a function of detecting the position of a pointing device such as a mouse or a touch panel and displaying the cursor 3 at this position. Further, the OS detects an event (left click, right click, double click, mouse over, mouse down, mouse up, etc.) performed by the pointing device. Further, the information processing apparatus 30 runs various application software on the OS. Since the OS notifies the application software of the position of the pointing device and the event, the application software operates based on the event performed at the position of the pointing device.

  In the present embodiment, the motion detection device 10 and the gesture operation reception program described later operate as a pointing device. The gesture operation acceptance program is a program that acquires sensor information from the motion detection device 10 and converts it into the position of the cursor 3 or detects an event, and is operating in the information processing device 30. The gesture operation reception program 110 notifies the application software 120 of the position of the cursor 3 and the event, so that the application software 120 does not have to be aware of the pointing device (no change is required because the pointing device is the motion detection device 10). ) Can work.

  The display 310 is a liquid crystal display device, a display device using organic EL as a light emitting principle, or a display device using plasma emission. The application software 120 creates a screen called CUI (Character User Interface) or GUI (Graphical User Interface) and displays it on the display 310.

  Note that as the display 310, any device including the display 310 may be used. For example, a display provided on an electronic blackboard or a notebook PC can be used as the display 310.

  The motion detection device 10 is a sensor that detects a motion of a person. For example, when a space is imaged by a stereo camera, parallax is obtained for each pixel of image data and distance information for each pixel is obtained. Further, it becomes possible to detect a person or recognize a person's motion by recognizing the pattern of the image data. Specifically, the three-dimensional coordinates and posture of a person's palm, wrist, shoulder, etc. can be obtained. Kinect (registered trademark; hereinafter, abbreviated as a registered trademark) is known as a sensor suitable for detecting such a human motion. Details of Kinect will be described later. The motion detection device 10 does not have to be Kinect, but may be a sensor having the same function as Kinect, a stereo camera, a camera that detects a distance by TOF (Time Of Flight), or the like.

  The installation position of the motion detection device 10 of FIG. 2A is an example, and may be installed above, on the left side, or on the right side of the display 310. Further, the installation position in the information processing device 30 is also only an example, and may be installed, for example, behind the display 310. Further, it may be connected to the motion detection device 10 via a network.

  As shown in FIG. 2B, the information processing system 100 may include the projector 5 instead of the display 310. In this case, the information processing system 100 includes the motion detection device 10, the projector 5, and the information processing device 30. The projector 5 is a display device that projects an image on a projection surface 6 such as a screen by an LCD (transmissive liquid crystal) system, an LCOS (reflective liquid crystal) system, a DLP (Digital Light Processing) system, or the like.

  The projector 5 and the information processing device 30 are connected in the same manner as the connection mode between the display 310 and the information processing device 30. The projector 5 is preferably capable of projecting from a so-called extremely monofocal point. This makes it easier to secure a space in which the user U operates. The motion detection device 10 may be on the user U side or the screen side of the projector 5. When it is on the screen side, it may be fixed to the lower side, upper side, left side or right side of the screen. If it is on the side of the user U, it is installed on a prepared stand. Further, the motion detection device 10 is installed at the right end of the display 310, which is an example, and may be installed at the left end. It is installed at the right end or the left end in order to avoid waste heat of the projector 5. If the waste heat can be ignored, it may be installed in the center of the display 310.

  The display 310 and the projector 5 may be used together. In this case, the projector 5 can guide the user to the operation position 52 by projecting an arrow or the like toward the floor. In addition, an arrow can be displayed to indicate the traveling direction (for example, the elevator is over there, the operation position 52 is over there, etc.).

<About hardware configuration>
<<Hardware Configuration of Information Processing Device 30>>
FIG. 3 is an example of a hardware configuration diagram of the information processing device 30. The information processing device 30 includes a CPU 301, a ROM 302, a RAM 303, and an auxiliary storage device 304. Furthermore, the private use server includes an input unit 305, a display control unit 306, a network I/F 307, and an external device I/F 308. Note that the respective units of the information processing device 30 are connected to each other via a bus B.

  The CPU 301 executes various programs, OS, etc. stored in the auxiliary storage device 304. The ROM 302 is the non-volatile memory 16. The ROM 302 stores various programs stored in the auxiliary storage device 304, such as programs and data necessary for the CPU 301 to execute them.

  The RAM 303 is a main storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). Various programs stored in the auxiliary storage device 304 when executed by the CPU 301 are expanded in the RAM 303, and the RAM 303 serves as a work area of the CPU 301.

  The auxiliary storage device 304 stores various programs executed by the CPU 301 and various databases used when the various programs are executed by the CPU 301. The auxiliary storage device 304 is a non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

  The input unit 305 is an interface for an operator to input various instructions to the information processing device 30. For example, a keyboard, a mouse, a touch panel, a voice input device, or the like, which may not be connected in this embodiment.

  In response to a request from the CPU 301, the display control unit 306 displays various information included in the information processing device 30 on the display 310 in the form of the cursor 3, menus, windows, characters, images, or the like. The display control unit 306 is, for example, a graphic chip or a display 310 I/F.

  The network I/F 307 is a communication device that communicates with other devices via the network. The network I/F 307 is, for example, an Ethernet (registered trademark) card, but is not limited to this.

  The external device I/F 308 is an interface for connecting a USB cable or various storage media 320 such as a USB memory. In this embodiment, the motion detection device 10 is connected.

  The information processing device 30 is preferably compatible with so-called cloud computing, and the information processing device 30 does not have to be housed in a single housing or provided as a unit. In this case, the configuration of the information processing device 30 is configured by dynamically connecting/disconnecting hardware resources according to the load.

<<Hardware Configuration of Motion Detection Device 10>>
FIG. 4 is an example of a hardware configuration diagram of the motion detection device 10. The motion detection device 10 includes a CPU 11, a color camera 12, an infrared camera 13, an infrared projector 14, a main storage device 15, a nonvolatile memory 16, a microphone 17, an acceleration sensor 18, and an LED 19. The CPU controls the entire operation detection device 10 by expanding the firmware stored in the non-volatile memory 16 into the main storage device 15 and executing it. The main storage device 15 is a high-speed volatile memory such as a DRAM, and the non-volatile memory 16 is a flash memory or the like.

  The color camera 12 is, for example, an imaging device that captures a color image with a resolution of 1920×1080. That is, the R, G, B density information is obtained for each pixel. The infrared camera 13 and the infrared projector 14 function as one depth sensor (distance sensor). The infrared projector 14 projects pulse-modulated infrared light, and images the infrared light projected by the infrared camera 13. The depth sensor measures the time it takes for the infrared rays to be projected, then reflected and returned, and obtains distance information for each projected infrared ray. The resolution of the distance information is 512×424, but this is merely an example.

  Further, the principle that the depth sensor measures the distance is merely an example, and the infrared pattern projected may be imaged by the infrared camera 13 and the distance information may be acquired from the distortion of the pattern.

  The microphone 17 is a device that detects ambient sound, and the CPU 11 estimates the direction of the sound source by comparing the phases of the sounds detected by the plurality of microphones 17. The acceleration sensor 18 is a sensor that detects the inclination of the motion detection device 10 with respect to the horizontal. The LED 19 displays the state of the operation detecting device 10 by color or blinking.

<Information acquired by the motion detection device 10>
Next, the information acquired by the motion detection device 10 will be described. The information acquired by the motion detection device 10 can be used by the information processing device 30. However, it is not necessary to use all the information described below. Information acquired by the motion detection device 10 described below is referred to as “sensor information”.

1. Color image This is image data that includes R, G, and B density information for each pixel. The infrared camera 13 of the motion detection device 10 captures a color image at a predetermined frame rate (for example, 30 fps).

2. Distance image This is image data for which distance information has been obtained at a resolution of 512×424 as described above. The depth sensor of the motion detection device 10 captures a distance image at a predetermined frame rate (for example, 30 fps). Therefore, the distance to the target including the user U can be obtained with a predetermined resolution.

3. Infrared image This is a grayscale image captured by the infrared camera 13. The pulse projected from the infrared projector 14 can be imaged. Further, a person (user U) who emits infrared rays can be detected.

4. The person shape data motion detection device 10 can create the person shape data by extracting only the person area from the distance image. For example, a person's imaging area is specified from the infrared image, or an area whose distance changes according to the frame is specified as a person's imaging area. A person's shape data is obtained by extracting this imaging region from the distance image. Therefore, the information processing device 30 knows the pixel region in which the user U is located. Further, when a plurality of users U are detected, the motion detection apparatus 10 detects the user U and the pixel area in association with each other. The information processing device 30 can also cut out only the person region from the color image.

5. Skeleton information Joint coordinate information. It is sometimes called bone information or skeleton information. FIG. 5 shows a joint from which three-dimensional coordinates can be obtained. In addition, in FIG. 5, the joint of the user is viewed from the front. The reference numerals and joint names in FIG. 5 will be described. Also, in FIG. 5, the joints of the left half are the same as those on the right side, so the reference numerals are omitted.
a...head joint, b...neck joint, c...shoulder central joint, d...right shoulder joint, e...right elbow joint, f...right wrist joint, g...right wrist joint, h...right wrist joint, i...right thumb joint , J... spine joint, k... waist central joint, l... right hip joint, m... right knee joint, n... right heel joint, o... right foot joint In addition, the motion detection device 10 has three-dimensional coordinates in space of each joint. It provides not only (X,Y,Z) but also the coordinates (x,y) of the joint on the color image (on the display 310). This is because the pixel area in which the user U is located is clear. Further, since the three-dimensional coordinates of the “right hand joint”, the “right thumb joint”, the “left hand joint”, and the “left thumb joint” are known, the motion detection device 10 can determine whether the user U has the palm open or closed.

6. Facial Expression The facial expression of the user U based on face recognition. The motion detection device 10 is usually laughing, surprised, left or right eyes closed or open, mouth open or closed, eyes off the screen, or wearing glasses. Can recognize, etc.

7. Face orientation This is the face orientation of the person. The motion detection device 10 can detect a yaw angle, a pitch angle, and a roll angle.

8. Heart rate This is the estimated heart rate of the user U. The skin is estimated from the color image and the infrared image, and is estimated from the subtle changes in the skin portion due to the blood flow.

<Regarding Functions of Information Processing Device 30>
First, the relationship between the application software 120 and the gesture operation acceptance program 110 will be described with reference to FIG. Both the application software 120 and the gesture operation acceptance program 110 operate on the OS 130. The application software 120 may be one that can be operated by position information or an event. For example, there are application software 120 for displaying advertisements, application software 120 for presentations, and the like. However, it is not limited to these, and may include, for example, a browser, a word processor, a game, and the like.

  The device driver 140 of the motion detection device 10 is installed in the information processing device 30. The device driver 140 is software that processes a signal detected by the motion detection device 10 into sensor information. The device driver 140 may be called SDK (Software Development Kit). Arbitrary application software 120 can acquire the sensor information created by the device driver 140 via an API (Application Interface) of the OS 130. In this embodiment, the gesture operation acceptance program 110 uses this API to acquire sensor information used for processing.

  The gesture operation reception program 110 acquires sensor information from the motion detection device 10 and performs various processes to control the gesture operation reception program 110 itself, and at the same time, to the application software 120 to display the position of the cursor 3 and an event for operation. Send. The application software 120 has a function of detecting the position of the cursor 3 and an event input by operating a pointing device such as a mouse or a touch panel. Generally, the OS 130 detects the position and event of the pointing device and provides them to the application software 120. In this embodiment, since a general-purpose pointing device is not used (not connected to the information processing device 30), the gesture operation reception program 110 provides the position of the cursor 3 and the event to the application software 120. The gesture operation reception program 110 can provide the position of the cursor 3 and the event to a plurality of application software 120.

  The providing method may follow the specifications of the OS 130 and the application software 120. For example, the inter-process communication between the application software 120 provided by the OS 130 and the gesture operation reception program 110 may be used, or the gesture operation reception program 110 notifies the OS 130 of the position of the cursor 3 and an event, and notifies the application software of this. The method acquired by 120 may be used. With such a mechanism, there is no need for the application software 120 to change the position of the cursor 3 and the event from the gesture operation reception program 110, and the gesture operation reception program 110 should be used together with the general-purpose application software 120. You can

  Although the gesture operation acceptance program 110 is one of application software, it may be implemented as a device driver.

<<Functions of the gesture operation reception program 110>>
FIG. 7 is an example of a functional block diagram in which the functions of the gesture operation reception program 110 are shown in blocks. The gesture operation acceptance program 110 includes an information acquisition unit 31, a person detection unit 32, an operation start detection unit 33, an operation area creation unit 34, a coordinate conversion unit 35, a UI control unit 36, a speed detection unit 37, an event determination unit 38, It has an information recording unit 43 and an output unit 39. Each of these units is a function or means that is realized by any one of the components shown in FIG. 3 being operated by an instruction from the CPU 301 according to a program 304p expanded from the auxiliary storage device 304 to the RAM 303. .. The program 304p includes an OS 130, a device driver 140, application software 120, and a gesture operation acceptance program 110.

  Further, the information processing device 30 has a sensor information DB 44 (database), and the information recording unit 43 records the sensor information in the sensor information DB 44. The sensor information DB 44 is constructed in the auxiliary storage device 304 and the RAM 303 shown in FIG.

The information acquisition unit 31 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and acquires sensor information from the device driver 140. Since the motion detection device 10 periodically detects motion, the information acquisition unit 31 periodically and repeatedly acquires sensor information. In addition, it is not necessary to acquire all the sensor information, and only the sensor information used for the processing may be acquired. In this embodiment, the following information is used.
The three-dimensional coordinates of the right shoulder joint d, the right wrist joint f, the right hand joint g, the right hand joint h, and the right thumb joint i are used. Since the user can operate with either the right hand or the left hand, the three-dimensional coordinates of the left shoulder joint, the left wrist joint, the left hand joint, the left hand joint, and the left hand thumb joint are also used.
Three-dimensional coordinates for each user U (mainly used to display the person icon 82)
Face orientation (used to display the eyes of the person icon 82)
The human detection unit 32 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and the action detection device 10 detects that a person is detected based on the sensor information acquired by the information acquisition unit 31. To detect. The motion detection apparatus 10 may detect that a person is detected or the number of people is detected by the number of people, or by utilizing that the shape data and skeleton information are obtained when the motion detection apparatus 10 detects a person. May detect that a person has been detected. The human detection range 51 is determined by the specifications of the motion detection device 10. For example, the angle in the horizontal direction is 70 degrees, and the distance is in the range of 0.5 to 4.5 m. A person may be detected only in a range narrower than the specification of the motion detection device 10, or the range of detecting a person may be changed if the specification of the motion detection device 10 changes. Further, although the number of people who can be detected at the same time is 6, this can be changed if the specifications of the motion detection apparatus 10 change.

  The operation start detection unit 33 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110, and the like, and the user U intends to operate the application software 120 based on the sensor information acquired by the information acquisition unit 31. Is detected. In the present embodiment, the user U detects the intention to operate by “being at the operation position 52 and +raising the hand”. Details will be described later. In addition, the operation start detection unit 33 detects that the user U indicates the intention to end the operation of the application software 120 based on the sensor information. In the present embodiment, the user U detects the intention to end the operation by “moving away from the operation position 52” or “lowering the hand”.

  The operation area creation unit 34 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like. When the operation start detection unit 33 detects the intention of the operation, the user U performs the operation. The operation area 54 is created. The operation area 54 is a range (area) in the three-dimensional space in which the user U can operate the information processing device 30. When the right hand joint g, the right wrist joint f, and the like of the user U are in the operation area 54, the three-dimensional coordinates of the right hand joint g and the right wrist joint f are reflected in the position of the cursor 3, and a gesture is detected.

  The coordinate conversion unit 35 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and based on the sensor information acquired by the information acquisition unit 31, the right hand joint g of the user U in the operation area 54 or The three-dimensional coordinates of the right wrist joint f are converted into the position of the cursor 3.

  The speed detection unit 37 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and the right hand joint g of the user U in the operation area 54 based on the sensor information acquired by the information acquisition unit 31. Or the speed of the right wrist joint f is detected. Either speed of the right hand joint g and the right wrist joint f may be detected, but since the user U operates with the elbow as the center, the cursor 3 can be moved earlier in the joint closer to the hand. On the contrary, the joint farther from the hand is easier to control the minute movement amount.

  The event determination unit 38 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 or the like, and based on the sensor information acquired by the information acquisition unit 31, whether the user U has generated an event or not. to decide. Events are left click, right click, double click, mouse over, mouse down, mouse up, etc. as described above, but general gestures corresponding to these events are not defined. For this reason, the event determination unit 38 associates the movement of the user U as a gesture with the event in advance, and when the user U performs the predetermined gesture, it determines that the associated event has occurred. For example, when the user U makes a gesture of holding a hand in the operation area 54, the event determination unit 38 determines that the event is a left click.

  The information recording unit 43 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation accepting program 110 or the like, and stores the sensor information in the sensor information DB 44 in time series for each user. As described above, the motion detection device 10 detects the position of the user, the facial expression and the face direction of the user, and thus records whether or not the user is looking at the display 310 to the extent that the user can estimate where the display 310 is looking. it can. The information recording unit 43 also acquires a screen shot of the content displayed on the display 310 from the application software 120. By setting the timing at which the sensor information is recorded and the timing at which the screen shot is acquired to be substantially the same, the content provider or the like can easily identify the content that the user is interested in.

  The output unit 39 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation accepting program 110 or the like, and acquires the position of the cursor 3 converted by the coordinate conversion unit 35 and the event determined by the event determination unit 38. And outputs it to the application software 120. The output unit 39 also has a function of acquiring the status of the application software 120. For example, when the application software 120 displays a moving image on the entire screen, the user does not need to operate the information acquisition unit 31, so the information acquisition unit 31 does not acquire the sensor information. Also, the output unit 39 notifies the application software 120 of the fact that the user is currently operating. Accordingly, the application software 120 does not shift to full-screen moving image display during the user's operation, and thus it is possible to prevent the user from suddenly becoming unable to operate.

  The UI control unit 36 is realized by the CPU 301 shown in FIG. 3 executing the gesture operation reception program 110 and the like. The UI control unit 36 further includes a cursor display unit 41 and an icon display unit 42. The cursor display unit 41 displays the cursor 3 using the function of the OS 130, or displays the guide mesh 4 instead of the normal cursor 3. Further, the icon display unit 42 displays the operation reception support icon 81 on the display 310. The icon display unit 42 changes the display mode of the operation reception support icon 81 according to the state of the user U. The operation reception support icon 81 functions as a user interface that assists the user U in easily operating the application software 120 with a gesture. For example, the motion detection apparatus 10 guides the user to the operation position 52 so that all the skeleton information of the user can be acquired. In addition, it clearly indicates who has the operation right (the person icon 82 raising the hand has the operation right even if a plurality of people icons 82 are displayed). The operation right is a right to operate the application software 120, and is granted to only one person at the same time.

<Person detection range 51 and operation position 52>
The human detection range 51 and the operation position 52 will be described with reference to FIG. FIG. 8 is an example of a diagram illustrating a human detection range 51 and an operation position 52 for the motion detection device 10. As described above, the motion detection device 10 detects a person within a predetermined distance (for example, 0.5 to 4.5 m) within a predetermined horizontal angle range (for example, 70 degrees). As a matter of course, the operation position 52 is within the human detection range 51.

  The operation position 52 is, for example, near the center of the human detection range 51, but may be anywhere as long as it is inside the human detection range 51. The operation position 52 is intentionally determined in order to prevent any user U who passes the human detection range 51 from raising the hand to operate the application software 120. Also, if the operation position 52 is too wide, the number of users U who can acquire the operation right increases, and it becomes difficult to determine whether or not the user can operate. Further, since the operation position 52 is determined, the user U can understand that the application software 120 can be operated by moving to the operation position 52. It is preferable that the actual operation position 52 is clearly displayed in a color different from that of other places, or that characters such as “Please operate here” are displayed.

  As described later, the user moves so that the person icon 82 enters the operation position instruction display 83 of the operation reception support icon 81. When the person icon 82 enters the operation position instruction display 83, the user is in the operation position 52. Therefore, the user is guided to the operation position 52 by the person icon 82, the operation reception support icon 81, and the operation position instruction display 83.

  However, for example, it may be assumed that the user deviates from the operation position 52 during the operation. For example, there is a case where the user approaches the display 310 in order to see it closely.

  FIG. 9 is an example of a diagram showing the detection range 51 in three dimensions. In FIG. 9A, the detection range 51 includes all the skeleton information of the user. Therefore, the motion detection device 10 can acquire all the skeleton information and send it to the gesture operation reception program 110. In other words, the user can operate the digital signage by the operation.

  On the other hand, in FIG. 9B, a part of the user's body protrudes from the detection range 51. The motion detection device 10 sends only the acquired skeleton information to the gesture operation reception program 110. For example, each function of the gesture operation reception program 110 (operation start detection unit 33, operation area creation unit 34, coordinate conversion unit 35, UI control unit 36, speed detection unit 37, event determination unit 38) stores the skeleton information of the hand. It is used, but if it is not included in the sensor information, the skeleton information cannot be acquired. Therefore, it can be detected that the desired skeleton information has not been acquired.

  In such a case, the UI control unit 36 can display the message 540 on the display 310 as shown in FIG. 9C to move the user to the operation position 52 and receive the message. In FIG. 9C, a message 540 "Please away from the screen" is displayed. Therefore, the UI control unit 36 can display a guide and guide the user into the detection range 51 when the position of the user is displaced to such an extent that important skeletal information (necessary for control) is missing.

  Note that FIG. 9 illustrates an example in which the user is too close to the display 310. However, if at least a part of the user's body including the joint is outside the detection range 51, the gesture operation reception program 110 is set within the detection range 51. Can guide the user.

<Detection of intention of operation>
The detection of the intention of the operation and the operation area 54 will be described with reference to FIG. FIG. 10 is an example of a diagram for explaining the detection of the intention of the operation and the operation area 54. FIG. 10A shows an initial operation performed by the user U at the operation position 52. When the user U reaches the operation position 52, he first raises his hand. The user U has already grasped whether or not it is instructed to perform this operation. The intention to start the operation is detected because an operation not intended by the user is sent to the application software 120 when the operation is accepted from an arbitrary hand movement of the user U in the detection range 51.

  The operation start detection unit 33 detects the intention to start the operation when the right-hand joint g is higher than the height of the head joint a. Note that the hand of the user U may be right or left. The operation start detection unit 33 preferentially detects the right hand and ignores raising the left hand after raising the right hand, and ignores raising the left hand after raising the left hand.

  In addition, the reason why the operation start detection unit 33 detects that the right hand joint g is higher than the height of the head joint a as the detection reference of the intention to start the operation is that the user U generally does not reach beyond the head. This is because. For example, when talking on a cell phone or touching glasses, the hand remains lower than the head. For this reason, it becomes easy to suppress erroneous detection that the user has the intention to operate the application software 120 by the action of a person who has no intention to operate the application software 120.

<Operation area 54>
The operation area creation unit 34 creates the operation area 54 using the lengths of the right shoulder joint d and the head joint a (hereinafter referred to as A). As shown in FIG. 10B, a circle 53 whose center is on the right shoulder is created with the length A as a diameter. Then, as shown in FIG. 10C, a circumscribed rectangle of the circle 53 is created. This circumscribed rectangle is the operation area 54. The length A is about 30 cm, but it is merely an example and may be a length that the user U can reach.

  In this way, by creating the operation area 54 from the circle 53 centered above the shoulder, it becomes easier for the motion detection device 10 to capture the image of the hand of the user U. Although it depends on the height at which the motion detecting device 10 is installed, when the motion detecting device 10 is installed at the position of the waist of the user U, the motion detecting device 10 moves to the shoulder when the hand is as high as the shoulder. Cannot be imaged. Therefore, the length error between the right shoulder joint d and the head joint a becomes large (the length A becomes short or large), and the shape of the operation area 54 becomes unstable.

FIG. 11 is an example of a diagram for explaining the operation area 54 in detail. The three-dimensional coordinate of the head joint a is P1 (X1, Y1, Z1), and the three-dimensional coordinate of the right shoulder joint d is P2 (X2, Y2, Z2). Note that the horizontal direction is the X axis, the height direction is the Y axis, and the vertical direction of the paper surface is the Z axis. The length A is calculated as follows.
A=Y1-Y2
The coordinates of the center O of the circle 53 are (X2, Y2+A/2). Therefore, the operation area creation unit 34 may draw the circle 53 with the radius A/2 centered on the center O. Further, the coordinates of the four vertices C1 to C4 of the operation area 54 can be calculated as follows without obtaining the circle 53. Although the Z coordinate is omitted, for example, Z1, Z2 or the average thereof is used.
C1=(X2-A/2, Y2+A)
C2=(X2+A/2, Y2+A)
C3=(X2+A/2, Y2)
C4=(X2-A/2, Y2)
By creating the operation area 54 in this way, the user U can easily control the position of the cursor 3. Since the user U keeps raising his/her hand (when the user U lowers it, he/she loses the operation right), he/she inevitably operates based on the elbow. Since the position of the shoulder is almost fixed, the operation area 54 created above the shoulder can be operated around the elbow. Therefore, the user U can operate the cursor 3 by moving the right wrist joint f or the right wrist joint g around the elbow without searching for where the operation area 54 is.

<Non-operational action to which operation right is not granted>
When a user talks on a mobile phone or touches eyeglasses, the user often raises his or her hands above the head. Therefore, more preferably, it is effective that the operation start detection unit 33 detects the non-operation operation of the user as follows.

  FIG. 12 is an example of a diagram for explaining the non-operation operation of the user. The non-operation operation is an operation similar to an operation (for example, raising a hand) indicating that the user intends to operate the digital signage. For example, a user makes a call with a mobile phone, touches glasses, puts a hand on the forehead, and scratches the head.

  As shown in FIG. 12A, the operation start detection unit 33 defines a cube 55 centered on the head joint a. This definition may be referred to as setting, and specifically means to specify the coordinates of the cube 55. The size of the cube 55 is slightly larger than the size of the head or face, and is large enough to fit when the user's hand touches the face. It may be variable or fixed for each user. Further, it may be a rectangular parallelepiped, a sphere, or an ellipsoid instead of a cube.

  When the user performs a non-operating action, the user's hand often goes into this cube. Therefore, the operation start detection unit 33 can determine whether or not the operation is a non-operation operation depending on whether the hand enters the cube 55.

  As shown in FIG. 12B, when the right hand joint g does not enter the cube 55, the operation start detection unit 33 determines that it is not a non-operation operation. In this case, since the user U's hand is raised, the operation start detection unit 33 gives the operation right.

  As shown in FIG. 12C, when the right hand joint g enters the cube 55, the operation start detection unit 33 determines that it is a non-operation operation. In this case, the operation start detection unit 33 does not give the operation right even if the hand is raised because the hand is inside the cube 55.

  Therefore, it is possible to suppress erroneous detection that the user U who has no intention to operate has an intention to operate.

  FIG. 13 is an example of a view of the cube 55 of FIG. 12 viewed from the side. Since it is the cube 55, the cube 55 has a depth in the distance direction (Z direction) from the motion detection device 10. Since the hand of the user U1 is not in the cube 55, it is determined that it is not a non-operation motion, and since the hand of the user U2 is in the cube 55, it is determined that it is a non-operation motion.

  In this way, since the determination is made by the cube 55, when the user raises his/her hand so as not to enter the depth of the cube 55, the operation start detection unit 33 gives the operation right to the user, and the coordinate conversion unit 35 makes the right hand joint g etc. The coordinates of can be detected. On the other hand, when the user raises his/her hand to enter the depth of the cube 55, the operation start detection unit 33 does not grant the user the operation right even if the user raises his/her hand. Therefore, rather than simply determining whether or not there is a hand near the face on the XY plane, the determination by the cube 55 makes it easier to distinguish and detect the non-operation action and the action indicating the intention to operate.

<Coordinate conversion>
Next, conversion of the wrist joint or the coordinates of the wrist joint in the operation area 54 to the position of the cursor 3 will be described with reference to FIG. FIG. 14 is an example of a diagram showing the coordinates of the wrist joint or the wrist joint and the position of the cursor 3 in the operation area 54. Note that FIG. 14 illustrates a method of determining the position of the cursor 3 when the speed of the hand is equal to or higher than the threshold value.

  The coordinate conversion unit 35 associates the three-dimensional coordinates of the wrist joint or the wrist joint in the operation area 54 with the position of the cursor 3 on the display 310 in a one-to-one correspondence. First, the coordinate conversion unit 35 determines the vertex C1 of the upper left corner of the operation area 54 as the origin (0,0). Then, the three-dimensional coordinates of the wrist joint or the wrist joint in the operation area 54 are converted into three-dimensional coordinates with the origin (0,0) as a reference. Since the operation area 54 is a two-dimensional plane of the XY plane, the wrist joint or the Z coordinate of the wrist joint is not used. Therefore, the Z coordinate may be regarded as constant and may be omitted from the calculation. Let R(X,Y) be the three-dimensional coordinates of the wrist joint or wrist joint with C1 as the origin.

Further, the vertical resolution of the display 310 is H and the horizontal resolution is W. The vertical and horizontal lengths of the operation area 54 are A. Therefore, the position S(x, y) of the cursor 3 on the display 310 can be calculated by a simple proportional calculation.
x=W×A/X
y=H×A/Y
Of course, the wrist joint or the wrist joint of the user U may protrude outside the operation area 54, but the coordinate conversion unit 35 ignores the wrist joint or the wrist joint outside the operation area 54.

  The coordinate conversion unit 35 performs coordinate conversion each time the information acquisition unit 31 acquires sensor information. On the other hand, the operation area creation unit 34 updates the operation area 54 for each frame of image data. This makes it possible to create an appropriate operation area 54 even if the user changes his/her action such that he/she faces the front or his/her shoulder faces the action detecting device 10.

  The operation area 54 is erased when the operation start detection unit 33 detects the intention of ending the operation (when the user U lowers his hand).

  FIG. 15 is a side view of the three-dimensional coordinates of the wrist joint or the wrist joint of the user U. That is, the horizontal direction of FIG. 15 corresponds to the Z axis. The user U1 in FIG. 15 moves his arm near his body, and the user U2 moves his arm forward of his body. That is, the Z coordinates of the wrist joint or the wrist joint of the users U1 and U2 are different. However, the Y-coordinates of the wrist joints or the wrist joints of the users U1 and U2 are almost the same (although not clear from FIG. 15, the X-coordinates are almost the same).

  Even if the Z-coordinates are different as described above, the coordinate conversion unit 35 does not use the Z-coordinates when converting the coordinates of the wrist joint or the wrist joint into the position of the cursor 3, so that the positions of the cursor 3 operated by the users U1 and U2 can be changed. The (trajectory) will be the same.

<<Modification of operation area 54>>
In general, the aspect ratio of the display 310 is not 1:1 but is longer in the horizontal direction. On the other hand, both the vertical and horizontal lengths of the operation area 54 are the length A. Therefore, when the user U moves his or her hand in the horizontal direction, the cursor 3 largely moves in the horizontal direction, which may make it difficult for the user U to control the position of the cursor 3 in the horizontal direction. Therefore, it is effective to change the vertical and horizontal lengths of the operation area 54 according to the aspect ratio of the display 310. The aspect ratio of the display 310 is input to the information processing device 30.

  FIG. 16 is an example of a diagram illustrating a modified example of the operation area 54. FIG. 16A shows an example of the operation area 54 when the aspect ratio of the display 310 is horizontally long. The operation area creating unit 34 creates the operation area 54 by adding the extension areas 54a to the left and right of the square operation area 54 described above. The length α of the expansion region 54a is, for example, 10 cm. This is because if the length A is 30 cm, the user U can reach within a range of about 50 cm as a whole. Therefore, 10 cm is only an example, and the length A may be expanded within a reachable range. For example, it may be variable according to the height of the user U.

  FIG. 16B shows an example of the operation area 54 when the aspect ratio of the display 310 is vertically long. α may be the same as that in FIG. In this way, the vertical and horizontal lengths of the operation area 54 are changed according to the aspect ratio of the display 310, so that the operability of the user U is improved.

<Detection of speed>
The speed detection will be described with reference to FIG. FIG. 17 is an example of a diagram illustrating a method for detecting the speed of the wrist joint or the wrist joint. The speed detection unit 37 calculates the acceleration by using the past three-dimensional coordinates of the wrist joint or the wrist joint. For example, the velocity V ₁ of the wrist joint or the wrist joint at time t1 can be obtained from the three-dimensional coordinates of the wrist joint or the hand joint at times t0 and t1. The period at which the sensor information is acquired is Δt. As described above, the velocity V1 at time t1 is calculated using only the X and Y components.
Vx=(X _t1 −X _to )/Δt, Vy=(Y _t1 −Y _to )/Δt,
V ₁ =√(Vx ² +Vy ² )
Similarly, the wrist joint or wrist joint velocity V2 at time t2 can be obtained from the three-dimensional coordinates of the wrist joint or wrist joint at times t1 and t2. Then, the acceleration α at the time t2 is a value obtained by dividing the difference between the speeds V1 and V2 by the period Δt, and thus can be calculated as follows.
α=(V ₂ −V ₁ )/Δt
Therefore, if there are three-dimensional coordinates in the past three cycles, the speed detection unit 37 can calculate the wrist joint or the acceleration of the wrist joint for each cycle.

  In this embodiment, speed and acceleration are used as an index of hand speed. The thresholds to be compared with these rates shall be determined experimentally. For example, the speed that is slightly faster than the stop is set as the threshold value (for example, 1 to 10 [cm/sec]). The reason for clicking the operation button 103 is that the speed of the wrist joint or the wrist joint is often slow.

  When using velocity as the velocity of the wrist joint or the wrist joint, the velocity detection unit 37 can calculate the velocity by using the three-dimensional coordinates of the past two cycles. Further, the acceleration of the acceleration may be further calculated and used as the speed index. In this case, it can be calculated by using the three-dimensional coordinates of the past four cycles.

<Event detection>
A method of detecting an event will be described with reference to FIG. FIG. 18A shows the joints and the shape of the hand included in the hand portion, and FIG. 18B shows the joints and the shape of the hand included in the hand portion when the user U holds the hand. The event determination unit 38 detects, as an example, an event in which the user U holds a hand and clicks a gesture that opens quickly. In other words, at the time of grasping (goo), it becomes an event of left mouse button down, open (par) and left button up. This allows the user to click and drag while holding the hand. Further, when the holding time is long, the event determining unit 38 detects the event of right button down.

  As described above, the motion detection device 10 uses the three-dimensional coordinates of the right joint g, the right wrist joint f, the right wrist joint h, the right thumb joint i (and the left wrist joint, the left wrist joint, the left wrist joint, and the left thumb joint). Is being detected. As shown in the figure, when the user U grips the right hand, the three-dimensional coordinates of the right-hand joint h and the right-hand thumb joint i largely move in a specific direction. As shown in the figure, if the X and Y coordinates are taken, the Y coordinate of the right hand joint h becomes smaller. Further, the X coordinate of the right thumb joint i also becomes smaller.

  When the Y coordinate of the right hand joint h decreases and the X coordinate of the right thumb joint i also decreases, the event determination unit 38 determines that the event of the left button down has occurred. When the Y coordinate of the right hand joint h increases and the X coordinate of the right thumb joint i also increases, it is determined that a left button up event has occurred. Note that it may be determined that the event of left button down occurs when the Y coordinate of the right hand joint h decreases or the X coordinate of the right thumb joint i also decreases.

  In addition, since the motion detection device 10 detects a so-called rock-paper-scissors (guo, choki, par) gesture, it may be used to detect a gesture of gripping. For example, when the gesture of goo or choki is detected, it is determined that the event of left button down has occurred.

  Further, the event determination unit 38 may detect a gesture of gripping by using the Z coordinate instead of using only the two-dimensional coordinate of the three-dimensional coordinate. When the user U clicks the application software 120, for example, the user U moves his or her hand forward (or backward). The event determination unit 38 detects a change in the three-dimensional coordinates in the Z-axis direction at a speed equal to or higher than a predetermined speed, such as the right-hand joint g, and determines that an event of click has occurred.

  However, when the user U tries to move his or her hand in the Z direction, the X and Y coordinates also tend to move, and the position of the cursor 3 may move from the position where the user wants to click. Therefore, as in the present embodiment, a gesture of holding a hand in which the X coordinate and the Y coordinate are difficult to change is effective. Therefore, gestures other than holding and opening the hand, such as mainly moving a finger, may be associated with the click event.

<Coordinate conversion when speed is less than threshold>
The coordinate conversion when the speed is less than the threshold will be described with reference to FIG. FIG. 19 is an example of a diagram illustrating coordinate conversion when the speed is less than the threshold value. When the speed detecting unit 37 determines that the speed of the wrist joint or the wrist joint is less than the threshold value, the coordinate conversion unit 35 and the UI control unit 36 operate as follows.
(i) The cursor display unit 41 displays the guide mesh 4 at the position of the cursor 3.
(ii) It is determined whether the three-dimensional coordinates have changed by a threshold value or more (whether the wrist joint has operated with a size of the threshold value or more).
(iii) If the three-dimensional coordinates have changed by a threshold value or more, it is determined which component is larger in the X direction or the Y direction.
(iv) The component having the larger component in the X direction or the Y direction is moved along the guide mesh 4 by only the grid corresponding to the movement amount.

  (i) will be described. When the cursor display unit 41 acquires information indicating that the speed detection unit 37 determines that the speed is less than the threshold value, the cursor display unit 41 starts displaying the guide mesh 4. As shown in FIG. 19A, the cursor display unit 41 associates the position of the cursor 3 acquired from the coordinate conversion unit 35 with a predetermined intersection of the guide mesh 4 and arranges the cursor 3 at the intersection to place the guide mesh 4 in the guide mesh 4. Is displayed. The predetermined intersection is set in advance at any one of the approximate centers of the guide mesh 4. Therefore, the cursor 3 is displayed in the approximate center of the guide mesh 4. Further, the cursor 3 displayed together with the guide mesh 4 has a different shape from the cursor 3 when the speed is equal to or higher than the threshold value. In FIG. 19A, the vertical and horizontal double-ended arrows vertically intersect.

  The cursor 3 moves on the intersection of the guide mesh 4. The guide mesh 4 is erased when the speed reaches or exceeds the threshold value. While the speed is less than the threshold value, the guide mesh 4 may move or may be fixed.

  Although each square of the guide mesh 4 may be square, the vertical and horizontal lengths may be determined with the same ratio as the aspect ratio of the display 310.

  Next, the determination (ii) is made to prevent the position of the cursor 3 from changing due to a slight hand movement that the user U does not intend. In addition, when the user U makes a gesture of grasping to click while the cursor 3 is on the operation button 103 of the application software 120, there are many cases where the hand makes a motion other than the gesture. In this case, when the gesture is performed, the cursor 3 has moved to the outside of the operation button 103, and the user U may not be able to click the operation button 103. Therefore, by making the determination as in (ii), it is possible to prevent the position of the cursor 3 from changing due to a slight hand movement not intended by the user U.

In FIG. 19B, it is assumed that the three-dimensional coordinates have changed by a threshold value or more. That is, it is assumed that the wrist joint or the coordinates of the wrist joint have moved from the coordinates (X1, Y1) to (X2, Y2). The cursor display unit 41 calculates the movement amounts ΔX and ΔY in the X direction and the Y direction, respectively.
ΔX=X2-X1
ΔY=Y2-Y1
The cursor display unit 41 determines whether at least one of ΔX and ΔY is equal to or more than a threshold value. The threshold value is determined in consideration of the amount of movement of a general wrist joint or a wrist joint that causes the user U to wander when trying to stand still, and an error of the motion detection device 10. For example, it may be set to several cm, but can be appropriately determined experimentally by a person in charge.

  (iii) will be described. When at least one of the movement amounts ΔX and ΔY is equal to or more than the threshold value, the cursor display unit 41 determines which of ΔX and ΔY is larger. In FIG. 19B, ΔX>ΔY.

(iv) will be described. The cursor display unit 41 determines the position of the cursor 3 with respect to only X2 of the wrist joint or the coordinates (X2, Y2) of the wrist joint. For the position in the y direction on the guide mesh at the position of the cursor 3, the previous value is used as it is. Since the x coordinate of the position of the cursor 3 converted by the coordinate conversion unit 35 is not always on the intersection of the mesh guides, the cursor display unit 41 corrects the x coordinate on the intersection of the mesh guides. For example, the correction is made as follows.
-Correct to the intersection of the mesh guide where the x-coordinate of the position of the cursor 3 is closest-Correct to the intersection of the mesh guide next to the x-coordinate of the position of the cursor 3-For the mesh guide before the x-coordinate of the position of the cursor 3 Correction to the intersection By performing the correction in this way, the cursor display unit 41 can display the cursor 3 displayed together with the mesh guide on the intersection. In FIG. 19(c), the cursor 3 is moved and displayed at the intersection point that has been moved two spaces to the right from the position of the previous cursor 3.

<Wrist joint or hand joint switching>
The user U may want to move the cursor 3 largely, or may want to move it small, such as near the operation button 103. Further, since the user U moves his or her hand around the elbow, the joint outside the hand moves faster. Therefore, the coordinate conversion unit 35 switches the joint (part of the user's body) to be converted to the position of the cursor 3 depending on whether or not the speed detection unit 37 detects a speed equal to or higher than the threshold value.
A. When the speed is equal to or higher than the threshold value, the three-dimensional coordinates of the right hand joint g are converted into the position of the cursor 3.
B. When the speed is less than the threshold value, the three-dimensional coordinates of the right wrist joint f are converted into the position of the cursor 3.

  Accordingly, when the user wants to move the cursor 3 largely, the user U can easily move the cursor 3 quickly, and can easily move the cursor 3 to an intended position by using the wrist near the operation button 103 or the like. In the case of A, the three-dimensional coordinates of the right hand joint h and the right thumb joint i may be used. Further away from the elbow, the user can operate the cursor 3 faster.

<About the operation reception support icon>
The operation reception support icon 81 displayed by the icon display unit 42 will be described with reference to FIGS. 20: is an example of the figure which shows the operation reception assistance icon 81 when the user U is detected by the motion detection apparatus 10. As shown in FIG. The icon display unit 42 first displays the operation reception support icon 81 in a large size on the display 310. The operation reception support icon 81 has a shape substantially similar to the detection range 51 of the motion detection device 10.

  The operation reception support icon 81 may be always displayed, or may be displayed only when a person is detected, but is displayed large when it is detected. As a result, the user U can recognize that the motion detection device 10 has detected it. Moreover, there is an effect that the user's eyes are turned to the display 310. It should be noted that the icon display unit 42 may not only display the operation reception support icon 81 in a large size, but may also display it by blinking it or changing its color. In addition to this, the operation reception support icon 81 may be displayed on the projector 5 different from the display 310 to perform follogram projection (stereoscopic viewing).

  In FIG. 20, the user U enters the detection range 51 from the right rear of the detection range 51 of the motion detection device 10. Since the information acquisition unit 31 has acquired the three-dimensional coordinates of the user U, the information acquisition unit 31 displays the person icon 82 at the position corresponding to the position of the user U (referred to as the icon position) in the operation reception support icon 81. Since the person icon 82 is displayed at the position on the icon, the user U can grasp his/her position in the detection range 51. Since the information acquisition unit 31 has acquired the orientation of the face of the user U, the icon display unit 42 displays the eye position estimated from the orientation of the face on the person icon 82.

  On the operation reception support icon 81, a place corresponding to the operation position 52 (hereinafter, referred to as an operation position instruction display 83) is emphasized and displayed, and the user U makes the person icon 82 enter the operation position instruction display 83. Moving.

  FIG. 21 is an example of a diagram showing the operation reception support icon 81 when the user U raises his/her hand at the operation position 52. Note that FIG. 21 shows an example of a screen when the speed of the wrist joint or the wrist joint is equal to or higher than the threshold value. The icon display unit 42 displays the operation reception support icon 81 in a small size when a predetermined time has elapsed after the user U entered the operation position 52 or when a predetermined time has elapsed after the user U raised his/her hand at the operation position 52. To do. This makes it easier for the user U to see the screen of the application software 120 displayed on the display 310.

  When the user U raises his/her hand, the operation start detection unit 33 detects that the user U has raised his/her hand, so that the icon display unit 42 raises the hand of the person icon 82. In other words, the person icon 82 with the right hand raised is displayed. As a result, the operation right is given to the user U who has raised his hand. The gesture operation acceptance program 110 does not accept an operation even if another user U raises his hand. In FIG. 21, another user U enters the detection range 51 from the left rear, but this user U cannot operate the application software 120. That is, the user U (only) who raises his/her hand at the operation position 52 earliest obtains the operation right.

  The user U creates the operation area 54 at the operation position 52 as described above, moves the hand to display the cursor 3, or makes a gesture to generate an event.

  FIG. 22A is an example of a diagram showing a guide mesh when the user U raises his/her hand at the operation position 52 but the speed of the wrist joint or the wrist joint is less than the threshold value. In this case, the cursor display unit 41 displays the cursor 3 at the intersection of the guide mesh 4. Moreover, since the cursor 3 moves only vertically or horizontally, the user U can control the minute movement of the cursor 3 with reference to the squares of the guide mesh 4.

  The guide mesh 4 may not be displayed. Even if the guide mesh 4 is not provided, the cursor 3 moves only vertically or horizontally, and the user U can control the minute movement of the cursor 3. However, by displaying the guide mesh 4, the user U can understand that the cursor 3 can be moved only in units of squares, so that it is expected that the user U feels that the operability is easy.

  The horizontal lines of the guide mesh 4 may not be horizontal, and the vertical lines may not be vertical. For example, vertical and horizontal straight lines may intersect.

  Further, as shown in FIG. 22B, the cursor 3 may move in the radial direction on the circumference of the concentric circle.

<Operating procedure>
FIG. 23 is an example of a flowchart showing a procedure until the gesture operation reception program 110 gives an operation right to the user U. The processing of FIG. 23 is repeatedly executed while the information processing apparatus 30 is executing the gesture operation reception program 110, for example.

  First, the person detection unit 32 determines whether or not the motion detection apparatus 10 has detected a person (S10). When no person is detected (No in S10), the process of FIG. 23 ends.

  When a person is detected (Yes in S10), the icon display unit 42 displays the operation reception assistance icon 81 in a large size on the display 310 (S20). After that, the icon display unit 42 displays the operation reception support icon 81 in a small size after a certain time even if a person is detected.

  Next, the icon display unit 42 displays the person icon 82 at the position above the icon of the operation reception support icon 81 based on the position of the user U (S30). Since the information acquisition unit 31 periodically acquires the sensor information, the icon display unit 42 updates the position of the person icon 82.

  The operation start detection unit 33 determines whether the position of the user U has entered the operation position 52 (S40). That is, it is determined whether or not the X, Y, Z coordinates of the user U are included in the operation position 52.

  Until the user U enters the operation position 52 (No in S40), it is determined whether a person is not detected (S50). Then, when no person is detected (Yes in S50), the process of FIG. 23 ends.

  While the person is being detected (No in S50), it is repeatedly determined whether or not the user U has entered the operation position 52.

  Next, when a person enters the operation position 52 (Yes in S40), the operation start detection unit 33 determines whether the user U has raised his hand (S60). For example, it is determined whether or not the hand is located at a certain height (or higher) than the waist of the user.

  Until the user U raises his/her hand (No in S60), the determinations in and after step S40 are repeatedly executed.

  When the user U raises his/her hand (Yes in S60), the operation start detection unit 33 determines whether or not the hand is in the predetermined peripheral area (S70). The predetermined peripheral region is, for example, the inside of the cube 55. Note that the determination using the cube 55 is an example of determination as to whether or not it is within a predetermined area. For example, it is confirmed that the right hand joint h and the right hand joint g are not around the head joint a. As a result, it is possible to prevent the user U on the phone from operating the application software 120, for example. Note that the determination in step S70 is to prevent the user who has no intention to operate from being given the operation right, and is not essential. In addition, in the determination in step S70, it is determined that the area (the peripheral area near the body) of the rectangular parallelepiped (the rectangular parallelepiped including the area from the head to the front of the waist) created from the user's head toward the waist is not occupied. It is preferable to judge. This creates a region that does not depend on the posture of the user. Specifically, even in the case where the user is carrying luggage such as a binder, not only the area around the head but also the area around the waist can be included in the non-operation area.

  When the determination in step S70 is Yes, the operation start detection unit 33 detects the intention to start the operation and gives the operation right to the user U who raised his hand (S80). Therefore, it is possible to prevent other users U from operating the application software 120 even if they raise their hands at the operation position 52.

  Next, the icon display unit 42 displays the person icon 82 with the hand raised at the position above the icon (S90). Thereby, the user U can determine that he/she has recognized that he has raised his hand, and can start the operation.

  When the intention to start the operation is detected, the operation area creation unit 34 creates the operation area 54 (S100). After that, the user U can operate the position of the cursor 3 with a wrist joint or a wrist joint, and can operate the application software 120 with a gesture. The gesture operation acceptance program 110 starts accepting the operation of the application software 120 by the user U.

  FIG. 24 is an example of a flowchart showing a procedure in which the gesture operation reception program 110 receives the operation of the user U and operates the application software 120. The process of FIG. 24 starts when the operation area 54 is created.

  First, the information acquisition unit 31 acquires sensor information (S110).

  The speed detection unit 37 determines whether the speed of the wrist joint is equal to or higher than a threshold value based on the sensor information (S120).

  When the speed of the wrist joint is equal to or more than the threshold value (Yes in S120), the coordinate conversion unit 35 converts the three-dimensional coordinates of the wrist joint into the position of the cursor 3 (S130).

  The output unit 39 outputs the position of the cursor 3 to the application software 120 (S140).

  In addition, the event determination unit 38 determines whether or not the user U has made a gesture that causes an event (S150). This is because the user U can make a gesture even if the guide mesh 4 is not displayed.

  When an event is detected, the output unit 39 outputs the position of the cursor 3 and the event to the application software 120 (S160). The position of the cursor 3 and the event may not be output at the same time.

  Next, when the speed of the wrist joint is not equal to or more than the threshold value (No in S120), the cursor display unit 41 determines whether the guide mesh 4 is being displayed (S180). This is because it is not necessary to display the guide mesh 4 again while displaying the guide mesh 4.

  If the guide mesh 4 is not being displayed (No in S180), the cursor display unit 41 displays the guide mesh 4 in which the cursor 3 is displayed in the center (S190). The cursor display unit 41 acquires the position of the cursor 3 from the coordinate conversion unit 35. Thereafter, the process proceeds to step S140.

  If the guide mesh 4 is being displayed (Yes in S180), the cursor display unit 41 determines whether the three-dimensional coordinates of the wrist joint have moved by a threshold value or more (S200). That is, by paying attention to the wrist joint instead of the wrist joint, the user U can easily control the minute movement of the cursor 3.

  If the determination in step S200 is Yes, the cursor display unit 41 updates the position of the cursor 3 on the guide mesh 4 (S210). That is, the cursor display unit 41 converts only the larger amount of movement of the wrist joint in the X and Y directions to the position of the cursor 3. The cursor display unit 41 corrects the position of the cursor 3 to the intersection of the guide mesh 4. After that, the process proceeds to step S140, and the cursor 3 is displayed on the guide mesh 4.

  In addition, the operation start detection unit 33 appropriately determines whether the user U is in the operation position 52 and raising his hand (S170). This is to confirm the intention to operate.

  If the determination in step S170 is Yes, the process returns to step S110 and the operation of the application software 120 is continued.

  When the determination in step S170 is No, the output unit 39 requests the application software 120 to hide the guide mesh and the cursor (S220). With the above, the processing of FIG. 24 is completed. In this case, the operation area 54 is deleted and the user U loses the operation right.

<Recording and utilizing sensor information>
Subsequently, recording and utilization of sensor information will be described with reference to FIGS. 25 to 28. Conventional digital signage cannot keep a history as to whether or not it is viewed by a user. Since the information processing apparatus 30 of the present embodiment acquires the face orientation of the user from the motion detection apparatus 10, the content provider or the like can analyze how many users have browsed later. Further, since the information recording unit 43 records the screen shot of the content displayed on the display 310, it is possible to analyze what kind of screen shot the user is interested in. Since the screen shot has a large data size, it may be recorded only when the user's face is facing the display 310.

FIG. 25 shows an example of the sensor information recorded by the information recording unit 43 for analyzing the face orientation and the like. In FIG. 25, sensor information recorded for one user is described in XML format. Hereinafter, these pieces of information will be described.
The <TrackingID> tag is identification information of a user assigned to a recognized user (including a user who does not have operation authority).
The <Glasses> tag indicates whether you are wearing glasses.
The <Happy> tag indicates whether or not you are smiling (or happy).
The <LeftEyeClosed> tag indicates whether the left eye is closed.
<RightEyeClosed> tag indicates whether the right eye is closed
The <LookingAway> tag indicates whether or not it is facing you.
The <MouthMoved> tag indicates whether the mouth is moving.
The <MouthOpen> tag indicates if the mouth is open.
The <Yaw> tag indicates an angle (yaw angle) in the lateral direction of the face, which has a positive value when the head is bent laterally toward the left toward the motion detection device 10 and has a negative value when the head is bent to the right.
The <Pitch> tag indicates the angle (pitch angle) in the vertical direction of the face, and has a positive value when facing upward toward the motion detection device 10 and a negative value when facing downward.
The <Roll> tag indicates the angle (roll angle) in which the face is tilted, and is zero when the eyes and mouth are vertical.
The <PositionInfo> tag indicates the position of the user, and the <X> tag indicates the distance in the lateral direction from the center of the motion detection device 10 (right side plus value, left side minus value). The <Y> tag indicates the height, and the <Z> tag indicates the distance from the motion detection device 10.
The <LeftButtonDown> tag of the <MouseTrackingInfo> tag indicates whether the left mouse button is pressed, and the <RightButtonDown> tag indicates whether the right mouse button is pressed. The <MousePositionInfo> tag indicates the X coordinate and Y coordinate of the cursor.

  Note that <MouseTrackingInfo> and <MousePositionInfo> are recorded only for users who have operation authority. The information recording unit 43 records such information for each user.

  FIG. 26 is an example of a diagram showing a screen shot 201 acquired by the information recording unit 43 when the sensor information of FIG. 25 is acquired. Therefore, the content provider, etc. can measure where the user is looking at the content (where they are interested) and the dwell time of the line of sight using the <Yaw>, <Pitch>, and <Roll> tags. You can do things. Further, with the <MouseTrackingInfo> tag, it is possible to determine from the screenshot 201 whether the user is performing any operation or what the user is interested in doing.

  By the way, since the motion detection device 10 uses the distance information to detect the user, the user may not be detected depending on the positions of the plurality of users. FIG. 27 is an example of a diagram illustrating a user who is not detected. FIG. 27 shows that there are three users U1 to U3 in the detection range 51 of the motion detection device 10. However, since the user U1 and the user U3 overlap in the distance direction when viewed from the motion detection device 10, the motion detection device 10 cannot detect the user U3. That is, the skeleton information cannot be acquired.

  However, since the motion detection device 10 captures the color image 202 as described above, the user U3 appears in the color image 202. Further, the distance information of the area where the user U3 is captured is also acquired. The information recording unit 43 determines that the person is a person by recognizing a face or a silhouette from the color image 202, for example. Thereby, the number of users in the detection range 51 can be correctly acquired.

  The area in which the user U3 appears can be extracted as an area (for example, within the processing range behind the user U1) in the Z-direction coordinate range. The detection time can be shortened by limiting the area to the rearward Z coordinate.

  In this way, although there is a limit to the number of users that can be acquired at one time only with skeleton information (currently, up to 6 people), by combining with color image 202 and distance information, the number of users can be increased even when people are crowded Can be detected. Also, comparing the color image 202 before the user U3 is blocked by U1 with the color image after being blocked, or comparing the distance information before the user U3 is blocked by U1 with the distance information after being blocked. You can also When the number of users detected by the person detection unit 32 suddenly decreases, the information recording unit 43 estimates that the user is hiding, and can search by face recognition or the like from a region where the distance information is the same as before the hiding.

  FIG. 28 is an example of a diagram illustrating an example of analysis of sensor information stored in the sensor information DB 44 by the information recording unit 43. The information processing device 30 totals the number of users for each time period (10 minutes in the figure) from the sensor information stored in the sensor information DB 44. In FIG. 28, the height of the bar graph 205 indicates the number of users for each time period. Further, in each bar graph 205, the number of users in the time zone is divided into the number of users 204 who have the operation right and the number of users 203 other than that, and the number of users is displayed. Since the same <TrackingID> is given to the same user that is continuously detected, there is almost no possibility that the information processing device 30 will count duplicately.

  Further, in FIG. 28, the screen shot 201 of each time zone is displayed above the bar graph 205 like a timeline. Alternatively, a representative screenshot 201 of each time period may be displayed. The representative screen shot 201 is a screen shot at the beginning or the end of the time period, or a screen shot with the largest change in the image.

  Therefore, a content provider or the like can determine which content is a collection of many users. For example, in the case of teachers (teachers in schools or prep schools, supporters, etc.) having different contents in FIG. 28, it is possible to determine which teacher is popular.

<Summary>
As described above, the information processing apparatus 30 of the present embodiment improves the operability of the position of the cursor 3 by limiting the moving direction of the cursor 3 to a predetermined direction when the operation speed is less than the threshold value. be able to. Further, when the operation speed is less than the threshold value, the cursor does not move unless the right wrist joint f moves by the threshold value or more, and thus it is easy to prevent the cursor from moving when the user makes a click gesture. In addition, since the operation can be performed using the joints of different users depending on whether the operation speed is equal to or higher than the threshold value or lower than the threshold value, the operability of the user can be improved. Moreover, since the vertical and horizontal lengths of the operation area 54 are changed as appropriate according to the aspect ratio of the display 310, the operability of the cursor 3 by the user can be improved. Further, even when a plurality of users are detected, the operation right can be given to the user who first indicates the intention of the operation. Further, by displaying the operation reception support icon 81, the user can easily move to the operation position 52, and can easily understand that he/she is operating.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments, but the present invention is not limited to these embodiments, and various modifications can be made without departing from the gist of the present invention. And substitutions can be added.

  For example, in the present embodiment, the operation such as the movement of the cursor or the click is mainly described as an example, but the information processing apparatus 30 can acquire various information from the operation of the user. For example, since the content is often a moving image and changes automatically, the gesture operation reception program 110 may superimpose (superimpose) a question such as “Do you want to fix the channel?” on the content. .. On the other hand, when the user takes a predetermined action, the event determination unit 38 detects it and the output unit 39 requests the application software 120 to pause. Note that it is preferable to display the question only when the user is detected.

  Further, a simple question (questionnaire) can be similarly displayed on the content the user is looking at, and the questionnaire result can be acquired by the user's action.

  For example, the information processing device 30 may operate in a server client system. In this case, the information processing device 30 transmits the sensor information to the server and receives the cursor position and the event from the server. Then, it is sent to the application software 120.

  Although it has been described that the gesture operation reception program mainly displays the cursor 3, the cursor 3 may be displayed by appropriately using the function of the OS 130. Further, the application software 120 may display the cursor 3.

  Further, instead of associating the coordinates in the operation area 54 with the cursor position on a one-to-one basis, the user may operate the cursor position with the amount of movement of the hand. For example, when moving with the cursor, hold the hand and move it at a certain speed or more. The coordinate conversion unit converts the amount of movement of the hand into the amount of movement of the cursor, and updates the position of the last cursor. Therefore, in this case, the operation area 54 may be omitted.

  Further, although the left-click gesture has been described in the present embodiment, it is also possible to input the right-click gesture by the information processing device 30 distinguishingly detecting the right hand and the left hand. Also, the cursor can be moved according to the three-dimensional coordinates of a joint such as a foot or the face orientation. In addition to holding hands, various human gestures can be associated with events.

Further, although the operation of raising the hand is the operation of obtaining the operation right, the operation right may be given by the following operation.
When the voice or sound (clapping hand) is detected by the microphone 17. When the voice or sound (clapping hand) detected by the operation start detection unit 33 is determined to have a specific rhythm. If the user analyzes the image data and determines that he/she has a wireless device such as a smartphone, or if the operation start detection unit 33 detects a signal transmitted from a wireless device such as a smartphone that the user has, When the operation start detection unit 33 detects that the hand has been raised-When the operation start detection unit 33 detects that the shape of the hand has become a specific shape that matches a predetermined condition (such as goo, choki, par, etc.) )
When the operation start detection unit 33 detects that the user is standing (the operation right can be given when a person sitting at the venue stands at a seminar etc.)
When the operation start detection unit 33 detects that the facial expression has a specific facial expression that matches a predetermined condition (for example, laughing)
-When the operation start detection unit 33 detects that the user's line of sight (face direction) is directed to a specific area Further, the configuration example of FIG. 7 and the like is for facilitating the understanding of the processing by the information processing apparatus 30. It is divided according to the main functions. The present invention is not limited by the division method or name of the processing unit. The processing of the information processing device 30 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

  The speed detection unit 37 is an example of a speed detection unit, the coordinate conversion unit 35 is an example of a position information creation unit, the cursor display unit 41 is an example of a position instruction display information display unit, and an event determination unit. 38 is an example of a gesture detecting unit, the output unit 39 is an example of an output unit, the operation start detecting unit 33 is an example of a start action detecting unit, the operation area creating unit 34 is an example of an area creating unit, The icon display unit 42 is an example of an icon display unit. The information recording unit 43 is an example of an image recording unit.

3 cursor 4 guide mesh 10 motion detection device 30 information processing device 31 information acquisition unit 32 person detection unit 33 operation start detection unit 34 operation area creation unit 35 coordinate conversion unit 36 UI control unit 37 detection unit 38 event determination unit 39 output unit 41 Cursor display part 42 Icon display part 43 Information recording part 54 Operation area 81 Operation acceptance support icon 100 Information processing system

JP, 2005-176451, A

Claims (19)

An information processing device for creating position information of position indication display information indicating a position indicated by a user displayed on a display device, based on motion information related to a user's motion detected by a motion detection device,
Speed detection means for detecting the speed of the user's motion based on the motion information,
Position information creating means for creating the position information by limiting the moving direction of the position indication display information to a predetermined direction when the speed detected by the speed detecting means does not match a predetermined speed condition; Has,
The information processing apparatus, wherein the position information creating unit creates the position information using a body part of a user that differs depending on whether the speed detected by the speed detecting unit is equal to or more than a threshold value or less than a threshold value .   The information processing apparatus according to claim 1, wherein the position information creating unit limits a moving direction of the position instruction display information in a horizontal direction or a vertical direction of the display device. A position indication display information display means for displaying the position indication display information based on the position information,
The information processing according to claim 1 or 2, wherein when the speed detected by the speed detection means is less than a threshold value, the position indication display information display means displays the position indication display information together with a mesh at an intersection of the mesh. apparatus.   When the speed detected by said speed detecting means is less than a threshold value, said position information creating means updates the position of said position indication display information on said mesh when the size of the user's movement is greater than or equal to a predetermined value. The information processing apparatus according to item 3. The body part of the user used when the speed detected by the speed detecting means is equal to or higher than the threshold is a wrist joint, and the body of the user used when the speed detected by the speed detecting means is less than the threshold. The information processing apparatus according to claim 1, wherein the part is a wrist joint. Gesture detection means for detecting a predetermined gesture based on the motion information,
The information processing apparatus according to claim 1, further comprising: an output unit that outputs the position information and the gesture detected by the gesture detection unit to application software.   7. The information processing apparatus according to claim 6, wherein the gesture detection unit detects a gesture in which the user moves a hand without moving a wrist based on the motion information. A start motion detection unit that detects a predetermined motion performed by the user to start inputting the position information based on the motion information;
When the motion detection device detects a plurality of users, the start motion detection means has an operation right for inputting the position information to the user who has performed the predetermined motion earliest among the plurality of users. give,
The information processing apparatus according to any one of claims 1 to 7, wherein the position information creating unit creates the position information from the action of the user who has been given the operation right by the start action detecting unit. The predetermined motion is a motion of raising a hand,
When the start motion detecting means detects the predetermined motion, the area creating means creates an operation area above the shoulder,
The information processing apparatus according to claim 8, wherein the position information creating unit converts the coordinates of the user's hand with respect to the origin of the operation area as the position information of the position indication display information on the display device.   The information processing apparatus according to claim 9, wherein the predetermined motion is a motion of raising a hand and a motion in which the hand is not included in a predetermined peripheral area of the user's body.   The information processing apparatus according to claim 9 or 10, wherein the area creating means creates the operation areas having different vertical and horizontal lengths according to an aspect ratio of the display device.   When the motion detection device detects a user, the detection range of the user of the motion detection device is displayed on the display device, and the user is included in the detection range based on the position of the user included in the motion information. The information processing apparatus according to any one of claims 1 to 11, further comprising: an icon display unit that displays a person icon indicating.   The information processing apparatus according to claim 12, wherein the icon display unit clearly indicates a position for a user to input the position information in the detection range.   The information processing apparatus according to claim 13, wherein when the user raises a hand at the position, the icon display unit displays the icon of the person holding the hand.   The information processing apparatus according to any one of claims 1 to 14, wherein the case where the speed does not meet a predetermined speed condition is a case where the speed is less than a threshold value.   The image recording means for recording the information regarding the face orientation of the user included in the motion information, the number of users, and the image displayed on the display device in association with each other. The information processing device described. An information processing device that creates position information of position indication display information indicating a position indicated by a user displayed on a display device based on motion information regarding a user's motion detected by a motion detection device
Speed detection means for detecting the speed of the user's motion based on the motion information,
If the speed detected by the speed detecting means does not match the condition of a predetermined speed, position information creating means for creating the position information by limiting the moving direction of the position indication display information to a predetermined direction , an order of the program to function as,
The position information creating means is a program for creating the position information using a body part of a user which differs depending on whether the speed detected by the speed detecting means is equal to or higher than a threshold or lower than the threshold . A position information creating method performed by an information processing device, which creates position information of position pointing display information indicating a position pointed by a user displayed on a display device based on motion information related to a user's motion detected by a motion detecting device. ,
Speed detecting means detects the speed of the user's motion based on the motion information;
When the speed information detecting means does not match the speed detected by the speed detecting means with a predetermined speed condition, the position information creating means creates the position information by limiting the moving direction of the position indication display information to a predetermined direction. And a step of
The position information creating means creates the position information using the body part of the user, which differs depending on whether the speed detected by the speed detecting means is equal to or more than a threshold value or less than the threshold value . A motion detection device that detects motion information related to a user's motion;
Speed detection means for detecting the speed of the user's motion based on the motion information,
When the speed detected by the speed detecting means does not match the predetermined speed condition, the moving direction of the position indication display information indicating the position indicated by the user displayed on the display device is restricted to a predetermined direction. Position information creating means for creating position information of the position indication display information,
An information processing system, wherein the position information creating unit includes an information processing device that creates the position information using a body part of a user that differs depending on whether the speed detected by the speed detecting unit is equal to or more than a threshold value or less than a threshold value. ..

Images