JP2016148900A - Electronic apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an electronic apparatus that can perform control on a device on the basis of a photographed image.SOLUTION: A portable electronic apparatus includes a display provided on a front face, and an infrared camera provided on a side face. The portable electronic apparatus determines a user's gesture on the basis of an image photographed with the infrared camera, and controls an application being executed on the basis of a result of the determination of the gesture. When the user's hand is not detected in the image photographed with the infrared camera, the portable electronic apparatus temporarily stops the application being executed and display a menu screen. When the user holds the hand over the infrared camera again, the application is resumed.SELECTED DRAWING: Figure 19

Description

  The present invention relates to a handheld electronic device capable of performing predetermined information processing.

  Conventionally, there is a portable game device including a display and a camera provided on the back of the display (for example, Patent Document 1). In such a conventional game device, a captured image can be acquired by a camera, and an image based on a result of analyzing the acquired image can be displayed on a display.

JP 2006-271663 A

  However, the above-described conventional apparatus displays a captured image from the camera on a display, and there is room for improvement in that various controls of the apparatus are performed using the captured image.

  Therefore, one of the objects of the present invention is to provide an electronic device capable of performing control related to an apparatus based on a captured image.

  In the present invention, the following configuration is adopted in order to solve the above-described problems.

  An electronic apparatus according to an embodiment is a handheld electronic apparatus, and includes a camera, an acquisition unit, and a control unit. The camera images the side direction of the electronic device. The acquisition unit acquires an input image captured by the camera. The control unit performs at least one of application start, pause, and restart based on the input image acquired by the acquisition unit.

  Based on the above, it is possible to control start, pause, and restart of an application based on an image captured by a camera that captures the lateral direction.

  In another configuration, the application may be an interactive application in which processing is performed according to an input by a user.

  Based on the above, it is possible to control start, pause, and restart of an interactive application.

  In another configuration, the image processing apparatus may further include application control means for controlling the progress of the application being executed based on the input image acquired by the acquisition means.

  According to the above, based on the input image captured by the camera, the start, pause, and resume of the application can be controlled, and the progress of the application can be controlled, and at least any of the start, pause, and resume of the application Since it is not necessary to switch the operation unit depending on the progress, it is possible to improve user convenience.

  In another configuration, a detection unit that detects an object based on the input image acquired by the acquisition unit may be further provided. The control unit performs the control of the application based on a detection result of the object by the detection unit.

  Based on the above, it is possible to detect an object based on the input image and control the application based on the detection result.

  In another configuration, the detection unit may detect a user's hand.

  Based on the above, it is possible to detect the user's hand based on the input image and control the application based on the hand detection result.

  In another configuration, the detection unit may detect a gesture made by a user's hand. The control means performs the control of the application based on the gesture detected by the detection means.

  Based on the above, it is possible to control the start, pause, and restart of an application by a gesture.

  In another configuration, the control unit may start the application when the detection unit detects the object.

  According to the above, when an object is detected, the application can be started.

  In another configuration, the control unit may pause an application being executed when the object is not detected by the detection unit.

  Based on the above, it is possible to temporarily stop an application being executed when an object is not detected.

  In another configuration, the control unit may temporarily stop the application when a predetermined period has elapsed after the detection unit no longer detects the object.

  According to the above, the application can be temporarily stopped when a predetermined period has elapsed since the object is no longer detected. Thereby, for example, the application can be controlled as intended by the user.

  In another configuration, the control unit may resume the paused application when the object is detected by the detection unit after the currently running application is paused.

  According to the above, even when the application is temporarily stopped without detecting the object, the application can be resumed when the object is detected.

  In another configuration, the control unit may start or restart a second application different from the application being executed when the object is not detected by the detection unit.

  According to the above, when an object is not detected, the second application different from the application being executed can be started or restarted.

  In another configuration, when the object is detected by the detection unit, the control unit may start or restart the application that was executed before the second application was started.

  According to the above, when an object is detected after the second application is started without detecting the object, it is possible to start or restart the application that was being executed before the second application was started.

  In another configuration, the detection unit detects whether or not the electronic device is gripped by a user based on the input image, and the control unit determines that the electronic device is gripped by the user. When the detection means detects, the application may be started.

  According to the above, the application can be started when the electronic device is gripped.

  In another configuration, application control means for controlling the progress of the application being executed may be further provided based on the detection result of the object by the detection means.

  Based on the above, it is possible to control the progress of the application based on the detection result of the object.

  In another configuration, a distance calculation unit that calculates a distance between the object and the camera based on the input image may be further provided. The control means performs the control of the application based on the distance calculated by the distance calculation means.

  Based on the above, it is possible to control the application based on the distance to the object.

  In another configuration, the apparatus may further include a speed calculation unit that calculates the speed of the object based on the input image. The control means performs the control of the application based on the speed calculated by the speed calculation means.

  Based on the above, the application can be controlled based on the speed of the object.

  Further, in another configuration, when the application is suspended by the control unit, a notification unit that notifies the user that the application is suspended may be further provided.

  According to the above, it is possible to notify the user when the application is paused.

  In another configuration, the information processing apparatus may further include a selection unit that selects any one of the plurality of applications according to a user input. The control means starts an application that has already been selected by the selection means.

  According to the above, it is possible to select and start any one of a plurality of applications.

  In another configuration, the detection unit includes a first determination unit that determines whether or not a first condition relating to the detection of the object is satisfied, and a second condition that is stricter than the first condition. And second determination means for determining whether or not the condition is satisfied. When the first condition is satisfied, the control unit performs a first process related to the start or end of the application, but does not perform a second process following the first process, and performs the second process. If the condition is satisfied, the first process and the second process of the application are performed.

  Based on the above, it is possible to perform the first process and the second process related to the start or end of the application based on the first condition related to object detection and the second condition that is stricter than the first condition. For example, when the application is started, if the first condition is satisfied and the second condition is not satisfied, the application is started, but the predetermined processing in the application is not started. When the second condition is satisfied, a predetermined process in the application is started. Further, for example, when the application is terminated, if the first condition is satisfied and the second condition is not satisfied, the application is temporarily stopped, but the application is not terminated. When the second condition is satisfied, the application is terminated.

  In another configuration, when the application is paused, the control unit may accept an instruction from a user as to whether or not to terminate the application, and terminate the application according to the instruction from the user. .

  According to the above, after the application is paused, the application can be terminated in accordance with an instruction from the user.

  In another configuration, the control unit may terminate the application when a predetermined time elapses when the application is temporarily stopped.

  According to the above, the application can be terminated as the predetermined time elapses after the application is paused.

  In another configuration, the camera may be an infrared camera.

  In another configuration, the camera may be provided on a side surface of the electronic device.

  In addition, an electronic apparatus according to an embodiment includes an acquisition unit, an application execution unit, and a control unit. The acquisition unit acquires an input image captured by the camera. The application execution means executes an interactive application that proceeds based on a user operation. The control unit temporarily stops the application being executed based on the input image acquired by the acquisition unit.

  Based on the above, it is possible to pause the running application based on the input image captured by the camera.

  An electronic device according to an embodiment includes a camera that captures an image of the side direction of the electronic device, an acquisition unit, and an application activation unit. The acquisition unit acquires an input image captured by the camera. The application activation unit activates the application based on the input image acquired by the acquisition unit.

  Based on the above, it is possible to start an application based on the input image captured by the camera.

  According to an example of the present invention, an application can be controlled based on an input image from a camera.

Front view of portable electronic device 1 of the present embodiment Right side view of portable electronic device 1 Rear view of portable electronic device 1 The block diagram which shows an example of an internal structure of the portable electronic device 1 It is a figure which shows an example of the use condition of the portable electronic device 1, and is a figure which shows a mode when a user hold | grips the portable electronic device 1 with both hands The figure which shows a mode that a user performs gesture input using the right hand in the right side surface direction of the portable electronic device 1. The figure which shows a mode that a user performs gesture input using the right hand in the right side surface direction of the portable electronic device 1. The figure which shows an example of the image imaged with the infrared camera 4 at the time of gesture input as shown to FIG. 6A being performed The figure which shows an example of the image imaged with the infrared camera 4 at the time of gesture input as shown to FIG. 6B being performed The figure which shows an example of the guide display of this embodiment The figure which shows an example of the infrared image imaged with the infrared camera 4 in the state shown in FIG. The figure which shows an example of the image after the outline drawing process and guide frame image drawing process when the image shown in FIG. 9 is acquired The figure which shows the state which the user moved the hand upward (Y-axis direction) from the state shown in FIG. The figure which shows the state which the user moved the hand to the back direction (Z-axis direction) of the screen from the state shown in FIG. The figure which shows the synthesized image 41 which synthesize | combined the outline image 21 and the guide frame image 22 in the state shown in FIG. The figure which shows the synthesized image 41 which synthesize | combined the outline image 21 and the guide frame image 22 in the state shown in FIG. Schematic showing an example of processing for performing a perspective display The figure which shows an example of the menu screen displayed on the display 2 of the portable electronic device 1 The figure which shows a mode that the application B selected by the menu screen started The figure which shows an example of the screen immediately after the game of the activated application B is started The figure which shows an example of the screen transition while a game is running The figure which shows an example of the gesture for displaying the menu screen in other embodiment. The flowchart which shows an example of the process for performing a guide display The flowchart which shows an example of the process for displaying a menu screen Flow chart showing an example of application control processing

  Hereinafter, a portable electronic device according to an embodiment of the present invention will be described. The portable electronic device is a hand-held information processing apparatus that can be operated by a user's hand. For example, the portable electronic apparatus may be a game device, a mobile phone (smartphone, etc.), a tablet terminal, a camera, or a wristwatch type terminal. Arbitrary arbitrary apparatuses, such as, may be sufficient. FIG. 1 is a front view of the portable electronic device 1 of the present embodiment. FIG. 2 is a right side view of the portable electronic device 1. FIG. 3 is a rear view of the portable electronic device 1.

  As shown in FIG. 1, the portable electronic device 1 includes a display 2, a touch panel 3, an infrared camera 4, a distance measuring sensor 5, input buttons 6 (6 </ b> A to 6 </ b> D), an irradiation unit 7, and a projector 8. These are housed in the housing 10. The housing 10 (portable electronic device 1) has a plate shape and is a size that can be held by a user with one hand or both hands.

  As the display 2, for example, a liquid crystal display device or an organic EL display device is used, and any other display device may be used. The screen of the display 2 is provided so as to be exposed on the front surface (T5 surface) of the housing 10. The touch panel 3 is provided on the screen of the display 2 and detects a position on the screen touched by the user. As the touch panel 3, one that can detect one point or one that can detect multiple points is used. For example, an arbitrary one such as a capacitance method or a resistance film method may be used.

  The input buttons 6A to 6D accept input (pressing) by the user. The input buttons 6 </ b> A to 6 </ b> D are provided at positions where the user's finger can reach when the user grips both ends of the mobile electronic device 1. Specifically, the input buttons 6A and 6C are in a position where the finger of the right hand can reach when the user holds the portable electronic device 1 with the right hand, the input button 6A is in a position where the thumb of the right hand can reach, and the input button 6C is It is provided at a position where the index finger or middle finger of the right hand can reach. The input buttons 6B and 6D are in a position where the left hand can reach when the user holds the portable electronic device 1 with the left hand, the input button 6B is in a position where the left thumb can reach, and the input button 6D is the left index finger. Or it is provided at a position where the middle finger can reach. As shown in FIG. 1, the input buttons 6A and 6B are provided on the front surface (T5 surface) of the housing 10, and the input buttons 6C and 6D are provided on the upper surface (T4 surface) of the housing 10. In addition to the input buttons 6A to 6D, an input unit that receives input by the user may be provided with a cross key for inputting a direction, an analog stick, or the like.

  The infrared camera 4 includes a lens and a sensor that is sensitive to light (infrared rays, specifically near infrared rays). The sensor of the infrared camera 4 is an image sensor in which a plurality of elements sensitive to infrared rays are arranged vertically and horizontally, and each element of the image sensor receives infrared rays and converts them into electric signals to output a two-dimensional infrared image. .

  The distance measuring sensor 5 reflects light (for example, infrared light) emitted from a light source provided in the distance measuring sensor 5 to an object, and receives the reflected light with a light receiving element, thereby distance from the object. Measure. As the distance measuring sensor 5, an arbitrary sensor such as a triangular distance measuring method, a TOF (Time Of Flight) method, or the like may be used. As a light source of the distance measuring sensor 5, an LED, a laser diode, or the like that emits infrared light in a specific direction is used.

  The irradiation unit 7 irradiates infrared rays at a predetermined time interval (for example, 1/60 second interval). The irradiation unit 7 emits infrared rays in synchronization with the timing when the infrared camera 4 captures an image. The irradiation unit 7 irradiates infrared rays to a predetermined range in the right side surface direction of the portable electronic device 1. The infrared light emitted by the irradiation unit 7 is reflected by an object, and the reflected infrared light is received by the infrared camera 4, whereby an infrared image is acquired. The irradiation unit 7 may be used for capturing an infrared image by the infrared camera 4 and for measuring a distance by the distance measuring sensor 5. That is, the infrared light from the irradiation unit 7 may be used to capture an image by the infrared camera 4 and the distance sensor 5 may measure the distance.

  The projector 8 has a light source that emits visible light, and projects characters, images, and the like onto a projection surface (a screen, a user's hand, etc.) using the light from the light source.

  The infrared camera 4, the distance measuring sensor 5, the irradiation unit 7, and the projector 8 are provided on the side surface (for example, the right side surface: T1 surface) of the housing 10. Specifically, the imaging direction (optical axis) of the infrared camera 4 is oriented in a direction perpendicular to the right side surface. The detection direction of the distance measuring sensor 5 and the direction in which the projector 8 emits light are also perpendicular to the right side surface. That is, when the user holds the portable electronic device 1 with the left hand, the infrared camera 4 images the space in the right side surface direction of the portable electronic device 1, and the ranging sensor 5 is in the right side surface direction of the portable electronic device 1. Measure the distance to an object in space. The projector 8 projects an image or the like by irradiating visible light in the same direction as the infrared camera 4 and the distance measuring sensor 5.

  An outer camera 9 is provided on the back surface (T6 surface) of the portable electronic device 1 (FIG. 3). The outer camera 9 can typically capture a direction perpendicular to the imaging direction of the infrared camera 4 and can capture a direction perpendicular to the back surface. The outer camera 9 includes a lens and an image sensor that is sensitive to visible light. The outer camera 9 captures a space in the back direction and acquires a color image (RGB image). In addition to the rear outer camera 9, a camera may be provided on the front, or the rear outer camera 9 is not provided, and a camera is provided on the front (the surface on which the screen of the display 2 is provided). Also good.

  FIG. 4 is a block diagram illustrating an example of the internal configuration of the mobile electronic device 1. As shown in FIG. 4, the portable electronic device 1 includes a vibrator 11, a microphone 12, a speaker 13, a control unit 14, a communication unit 15, a posture detection unit 16, a GPS receiver 17, and a geomagnetic sensor 18 in addition to the above-described units. Prepare. The mobile electronic device 1 includes a battery (not shown), and power is supplied from the battery. These parts are accommodated in the housing 10.

  The control unit 14 includes a display 2, a touch panel 3, an infrared camera 4, a distance measuring sensor 5, an input button 6, an irradiation unit 7, a projector 8, a vibrator 11, a microphone 12, a speaker 13, a communication unit 15, a posture detection unit 16, and a GPS. It is connected to each part such as the receiver 17 and the geomagnetic sensor 18 and controls each part.

  Specifically, the control unit 14 includes a CPU, a memory (RAM), and the like, and a predetermined program (for example, a nonvolatile memory or a hard disk) stored in the portable electronic device 1 (not shown) For example, predetermined processing is performed based on game processing, image processing, and an application program for performing various calculations. For example, the control unit 14 acquires an image from the infrared camera 4 and analyzes the image, calculates a distance to an object based on a signal from the distance measuring sensor 5, or from the touch panel 3 or the input button 6. The process according to the input signal is performed. Then, the control unit 14 generates an image based on the result of the predetermined process, and outputs the image to the display 2. Note that a program for performing predetermined processing may be downloaded from the outside via the communication unit 15.

  The vibrator 11 operates based on a command from the control unit 14 and vibrates the entire portable electronic device 1. The vibrator 11 is provided at a predetermined position where vibration is easily transmitted to the user's hand (for example, a position biased to either the central portion or the left and right in the housing 10).

  The microphone 12 and the speaker 13 are used for inputting / outputting sound. The communication unit 15 is used for communicating with other devices by a predetermined communication method (for example, wireless LAN or the like). In addition, the posture detection unit 16 is, for example, an acceleration sensor or an angular velocity sensor, and detects the posture of the mobile electronic device 1.

  The GPS receiver 17 receives a signal from a GPS (Global Positioning System) satellite, and the portable electronic device 1 can calculate its own position by receiving the signal. For example, when a predetermined operation (for example, gesture input or button input using an infrared camera 4 to be described later, operation of shaking the mobile electronic device 1) is performed at a specific position, the mobile electronic device 1 An object associated with a specific position may be displayed. For example, when a game is played in the mobile electronic device 1, when the mobile electronic device 1 is at a specific position, an object corresponding to the specific position may appear in the game.

  The geomagnetic sensor 18 is a sensor capable of detecting the direction and magnitude of magnetism. For example, the mobile electronic device 1 determines whether or not it is facing a specific direction based on the detection result of the geomagnetic sensor 18, and when a predetermined operation (such as the above gesture input) is performed in the specific direction. The object may be displayed. For example, when a game is played on the mobile electronic device 1, an object corresponding to a specific direction may appear in the game. Moreover, the portable electronic device 1 may combine GPS information using the GPS receiver 17 and azimuth information using a geomagnetic sensor. For example, when the mobile electronic device 1 is at a specific position and is facing a specific direction, an object corresponding to the specific position and the specific direction is displayed, or the object is made to appear in the game. Also good.

  FIG. 5 is a diagram illustrating an example of a usage state of the mobile electronic device 1 and is a diagram illustrating a state when the user holds the mobile electronic device 1 with both hands. As shown in FIG. 5, the user grips both left and right ends of the mobile electronic device 1. When the user grips both left and right ends, the user can press the input button 6B with the thumb of the left hand and can press the input button 6A with the thumb of the right hand. Further, when the user grips both left and right ends, the user can move the index finger or middle finger of the left hand and can press the input button 6D with the index finger or middle finger of the left hand, and move the index finger or middle finger of the right hand The input button 6C can be pressed with the index finger or middle finger of the right hand.

  Next, input to the portable electronic device 1 will be described. In the present embodiment, the user can perform gesture input on the mobile electronic device 1 using the right hand while holding the mobile electronic device 1 with the left hand.

  6A and 6B are diagrams illustrating a state where the user performs gesture input using the right hand in the right side surface direction of the mobile electronic device 1. FIG. 7A is a diagram illustrating an example of an image captured by the infrared camera 4 when the gesture input illustrated in FIG. 6A is performed. FIG. 7B is a diagram illustrating an example of an image captured by the infrared camera 4 when the gesture input illustrated in FIG. 6B is performed.

  As illustrated in FIGS. 6A and 6B, the user performs various gesture inputs using the right hand at a position in the right side surface direction of the mobile electronic device 1. The portable electronic device 1 captures an infrared image with an infrared camera 4 provided on the right side surface of the portable electronic device 1 and analyzes the captured infrared image to determine a user's gesture input.

  Specifically, when the image shown in FIG. 7A or 7B is acquired from the infrared camera 4, the portable electronic device 1 displays a specific object (specifically, a human hand) included in the acquired image. The type of gesture performed by the user is determined based on the detected shape and the like of the specific object. For example, the mobile electronic device 1 determines whether or not an object having a predetermined shape exists in the acquired image by pattern matching or the like. For example, when the image shown in FIG. 7A is acquired, the mobile electronic device 1 can recognize “goo” from the number of fingers, and when the image shown in FIG. "Par" can be recognized. And the portable electronic device 1 performs the process according to the kind of gesture so that it may mention later.

  Here, the determined gesture is a gesture / hand gesture using a part or the whole of the user's hand or face, and the portable electronic device 1 recognizes the stationary state of the hand as a gesture input. Alternatively, a series of actions using the hand may be recognized as a gesture input. In addition, the mobile electronic device 1 may recognize a gesture input performed while the user holds an object. In this case, the mobile electronic device 1 may recognize a stationary state or an operating state of only an object held by the user as a gesture input, or a gesture of an entire stationary state or an operating state including the user's hand and the object. It may be recognized as an input.

  When a plurality of objects are included in the infrared image captured by the infrared camera 4, the mobile electronic device 1 detects only a specific object among the plurality of objects, and performs processing based on the detected object. May be performed. For example, when a user's hand is imaged using the infrared camera 4, the hand and an object behind the hand may be included in the infrared image. In this case, the mobile electronic device 1 may recognize only the user's hand. For example, the mobile electronic device 1 extracts only the image of the object having the highest average brightness from the images of the plurality of objects included in the infrared image captured by the infrared camera 4 and analyzes the image of the object. The user's hand may be recognized. Further, only the user's hand may be recognized based on pattern matching or the like from images of a plurality of objects included in the infrared image captured by the infrared camera 4.

  In the present embodiment, the user performs various operations on the mobile electronic device 1 by performing a gesture using the right hand in the direction of the right side of the mobile electronic device 1, for example. First, the user needs to recognize whether or not his / her right hand is normally captured by the infrared camera 4 arranged on the right side surface of the mobile electronic device 1 and gesture input is possible. Therefore, in the present embodiment, guide display is performed for allowing the user to recognize a position where gesture input is possible (for guiding the user's hand to a position suitable for gesture input). For example, a guide display is performed before an application described later is executed.

(Guide display)
FIG. 8 is a diagram illustrating an example of a guide display according to the present embodiment. FIG. 8 shows an example of a guide display when the user's right hand is within the imaging range of the infrared camera 4 and the user's right hand is imaged from the front of the infrared camera by the infrared camera 4.

  As shown in FIG. 8, a contour image 21 and a guide frame image 22 are displayed on the right side of the screen of the display 2 as a guide display for allowing the user to recognize the correct gesture input position. The contour image 21 is an image indicating the contour of the image of the user's hand included in the infrared image captured by the infrared camera 4. The guide frame image 22 is an image for allowing the user to recognize the correct gesture input position, and is an image showing a range narrower than the imaging range captured by the infrared camera 4.

  As shown in FIG. 8, for example, the right direction (long side direction) of the portable electronic device 1 is defined as the X axis, the upward direction (short side direction) is defined as the Y axis, and the back direction of the screen of the display 2 is defined as the Z axis. When the straight line extending in the X-axis direction from the infrared camera 4 hits almost the center of the palm of the user's right hand and the user's right hand exists at an appropriate position in the X-axis direction, the infrared camera 4 captures the entire image of the user's right hand. Is done.

  FIG. 9 is a diagram illustrating an example of an infrared image captured by the infrared camera 4 in the state illustrated in FIG. 8. FIG. 10 is a diagram illustrating an example of an image after the contour drawing process and the guide frame image drawing process when the image illustrated in FIG. 9 is acquired.

  As shown in FIG. 9, the infrared image 40 captured by the infrared camera 4 includes an image of the right hand of the user. A contour drawing process and a guide frame image drawing process are performed on the captured infrared image 40, thereby generating a composite image 41 shown in FIG. Specifically, as illustrated in FIG. 10, when the infrared image 40 is captured by the infrared camera 4, the portable electronic device 1 detects the contour of the right hand included in the infrared image 40 by edge detection, for example, and other than the contour The contour image 21 is generated by erasing the pixel information inside the. In addition, the mobile electronic device 1 draws the guide frame image 22 so as to overlap the generated contour image 21. Specifically, the guide frame image 22 is an image showing a rectangular frame having a certain size, which is fixed at the center of the infrared image 40 captured by the infrared camera 4.

  A composite image 41 obtained by combining the contour image 21 and the guide frame image 22 is tilted in a predetermined direction and displayed on the display 2. Specifically, as illustrated in FIG. 8, the mobile electronic device 1 parses the composite image 41 and displays the parsed composite image (an image obtained by viewing the composite image 41 from an oblique direction) on the display 2. A specific example of the parsing process will be described later.

  Here, when the user moves the right hand from the state shown in FIG. 8, the image displayed on the display 2 changes. FIG. 11 is a diagram illustrating a state where the user has moved his / her hand upward (Y-axis direction) from the state illustrated in FIG. 8. FIG. 12 is a diagram illustrating a state in which the user moves his / her hand in the back direction (Z-axis direction) of the screen from the state illustrated in FIG. 8.

  As shown in FIG. 11, when the center of the user's hand is above the center of the imaging range of the infrared camera 4, an image in which a part of the contour image 21 protrudes above the guide frame image 22 is displayed on the display 2. Is displayed. As shown in FIG. 12, when the center of the user's hand is on the left side (Z-axis direction side in FIG. 12) with respect to the center of the imaging range of the infrared camera 4, the contour image is on the left side of the guide frame image 22. An image in which a part of 21 protrudes is displayed on the display 2. Here, as shown in FIGS. 11 and 12, the part of the contour image 21 that protrudes from the guide frame image 22 is displayed in a display mode different from the part that fits in the guide frame image 22. For example, a portion of the contour image 21 that protrudes from the guide frame image 22 has a different color or is displayed with a thick line.

  FIG. 13 is a view showing a composite image 41 obtained by combining the contour image 21 and the guide frame image 22 in the state shown in FIG. FIG. 14 is a view showing a composite image 41 obtained by combining the contour image 21 and the guide frame image 22 in the state shown in FIG.

  As shown in FIG. 13, when the user's hand is present above the center of the infrared image (the center of the imaging range of the infrared camera 4), a part of the contour of the user's hand protrudes above the guide frame image 22. . As shown in FIG. 14, when the user's hand is present on the left side of the center of the infrared image, a part of the contour of the user's hand protrudes to the left of the guide frame image 22. These protruding portions are displayed in a display mode (for example, red) different from other portions (for example, black) that do not protrude. After such a composite image 41 is generated, these images are parsed and displayed on the display 2 (the composite image 41 is parsed as shown in FIGS. 11 and 12).

  In this way, by displaying the parsed image (an image obtained by viewing the composite image 41 from an oblique direction), the user can easily recognize the position in the depth direction, and his / her hand is within the imaging range of the infrared camera 4. It can be easily recognized whether or not it is present. In other words, the parsed image is the same as when the user looks directly at his / her hand, so the user can easily recognize the depth direction, and the position of his / her hand in the current depth direction. Can be easily recognized.

  Further, since the guide frame image 22 fixed in the imaging range of the infrared camera 4 is displayed, the user can easily adjust the position of the hand so that the hand enters the guide frame image 22. In addition, since the portion protruding from the guide frame image 22 is displayed in a different form from the portion in the guide frame image 22, the user can easily recognize that the hand has moved out of the guide frame image 22. Easy to adjust hand position.

  Further, the guide frame image 22 is smaller than the infrared image captured by the infrared camera 4. For this reason, even if the user's hand protrudes from the guide frame image 22, if the user's hand (finger) is included in the infrared image captured by the infrared camera 4, the mobile electronic device 1 makes a gesture made by the user. Can be recognized. By displaying the guide frame image 22, it is possible to warn the user that the user's hand comes out of the imaging range of the infrared camera 4 before the gesture made by the user cannot be recognized.

  In the present embodiment, the right side surface direction of the portable electronic device 1 is imaged by the infrared camera 4, and the captured image is displayed on the right side of the screen of the display 2. That is, the side direction of the portable electronic device 1 is captured by the infrared camera 4, and an image is displayed on the same side as the side direction captured by the infrared camera 4 in the screen of the display 2. For this reason, the user can easily recognize the position of his / her hand intuitively.

  Thus, in this embodiment, guide display (contour image 21 and guide frame image 22) is performed. Thereby, a user can be made to recognize the correct gesture input position. That is, by performing the guide display as described above, it is possible to make the user recognize whether or not the hand is in an appropriate position.

  Here, an example of processing for performing the perspective display as exemplified in FIGS. 8, 11, and 12 will be described.

  FIG. 15 is a schematic diagram illustrating an example of processing for performing a perspective display. As shown in FIG. 15, the contour image 21 and the guide frame image 22 are arranged at predetermined positions in the virtual space. For example, each point forming the contour image 21 and each point forming the guide frame image 22 are plotted on the yz plane of the virtual space defined by the x axis, the y axis, and the z axis. These points are translated in the x-axis direction to form a contour image 21 'and a guide frame image 22'. The projected image is displayed on the display 2 by projective transformation of each point after movement (the contour image 21 ′ and the guide frame image 22 ′). Specifically, the virtual camera 50 is arranged on the yz plane, the imaging direction (cz-axis direction) of the virtual camera 50 is set parallel to the z-axis, and the upward direction (cy-axis direction) of the virtual camera 50 is the y-axis. Is set in parallel. An image obtained by capturing the contour image 21 ′ and the guide frame image 22 ′ with such a virtual camera 50 is displayed on the display 2.

  The process for the perspective display is not limited to the process shown in FIG. For example, the synthesized image 41 obtained by synthesizing the contour image 21 and the guide frame image 22 is rotated by a predetermined angle around the vertical axis (Y axis in FIG. 8), and the rotated image is displayed on the display 2. Alternatively, an image in which the synthesized image 41 is parsed may be displayed.

  Moreover, in the said embodiment, although the outline image 21 which shows the outline of the image of the hand contained in the infrared image imaged with the infrared camera 4 was displayed on the display 2, the infrared image imaged with the infrared camera 4 instead of the outline is displayed. May be displayed on the display 2 as it is. Further, the infrared image captured by the infrared camera 4 may not be displayed on the display 2 as it is, but may be abstractly displayed on the display 2 as a perspective display. The infrared image captured by the infrared camera 4 is not limited to displaying the outline of the hand as described above, but, for example, by displaying bones (line segments connecting the feature points) connecting the captured hand feature points. May be abstracted and displayed, or the infrared image may be abstracted by other methods.

  Moreover, in the said embodiment, by displaying the part which protruded from the guide frame image 22 in the specific aspect different from another part among the outlines of the image of a hand, it is notified to a user that the hand protruded from the guide frame. It was. In another embodiment, when the user's hand protrudes from the guide frame, the user may be warned by displaying the entire hand image (or the protruding portion) in a specific manner. Further, when the user's hand protrudes from the guide frame image 22, a warning is given to the user by displaying the guide frame image 22 in a specific manner different from the normal in addition to or instead of the hand image. Also good. Here, as a specific aspect, it is not restricted to what makes a color different from normal as mentioned above, or changes the thickness of a line. For example, it may be made different from normal by blinking display. Further, when the user's hand protrudes from the guide frame, other images, characters, sounds, vibrations (vibration with respect to the hand (left hand) holding the portable electronic device 1), light using the projector 8 (which may be an LED lamp) For example, the user may be warned.

  In the above embodiment, the infrared image captured by the infrared camera 4 is parsed and displayed on the display 2. However, in another embodiment, the captured image (or an image obtained by abstracting the captured image) is displayed. It is also possible to display without parsing. That is, as long as the user can display the relative position of the hand with respect to the mobile electronic device 1, the display may be performed in any manner. For example, a range image indicating an imaging range captured by the infrared camera 4 is displayed on the display, and an image indicating the position of the user's hand is displayed on the range image in an overlapping manner (or without overlapping). It may indicate at which position the user's hand is located. For example, a range image indicating the position in the depth direction is displayed so that the user can easily recognize the position in the depth direction of the hand, and an image indicating the position of the hand is displayed on or over the range image. May be. Specifically, for example, a predetermined position (for example, the center of gravity) of the user's hand in the captured image is calculated from the infrared image captured by the infrared image 4. And the range image which shows the range of the left-right direction (Z-axis direction of FIG. 9) of an infrared image is displayed, and the image which shows the position of a hand is displayed in the said calculated position. Thereby, the user can recognize where the user's hand is located in the depth direction (Z-axis direction).

  In addition, the guide display described above is performed before the execution of an application described later, but the guide display may or may not be performed depending on the execution status of the information processing performed by the mobile electronic device 1. Also good. For example, guide display may always be performed during execution of an application described later, or display / non-display of the guide display may be switched according to a user instruction. In addition, when the user gives an instruction to execute a program for displaying a guide, the guide display may be performed. Further, the guide display may be performed only when setting the mobile electronic device 1 or during a tutorial for teaching the user how to use the mobile electronic device 1.

  Moreover, in the said embodiment, when a user's hand protrudes from the guide frame image 22 as a prior warning until it becomes impossible to recognize a user's hand, the said protrusion part was displayed in the specific aspect. Specifically, even when the user's hand can be recognized from the infrared image captured by the infrared camera 4, if the user's hand protrudes from the guide frame, the guide frame image 22 as described above. The user is warned in advance by displaying the protruding portion in a specific display mode. In this case, when the user's hand cannot be recognized from the infrared image actually captured by the infrared camera 4, another warning may be given. For example, a second warning different from the prior warning using the guide frame image 22 described above may be performed using voice, characters, images, vibration, light, or the like. Further, in the case of performing a prior warning using the guide display during execution of an application as described later, when the user's hand cannot be recognized from the infrared image captured by the infrared camera 4, the second After performing the warning (or without performing the second warning), the application may be terminated, or the application may be paused as described later.

  In the above-described embodiment, even when the user's hand is detected, the user is warned when at least a part of the user's hand protrudes from the guide frame. In other embodiments, the conditions for warning are not limited to this. For example, in another embodiment, it is determined based on an image captured by the infrared camera 4 whether or not a predetermined condition that makes it impossible to recognize a user's gesture is satisfied, and when the predetermined condition is satisfied A warning may be given. For example, the predetermined condition may be that an image captured by the infrared camera 4 is too dark (or too bright), a noise level included in the image is equal to or higher than a predetermined level, and the like. For example, when the average brightness of the entire image is less than a predetermined value, the user's gesture cannot be determined if the brightness falls below that value. In such a case, a warning may be given.

  Further, the shape and size of the guide frame are merely examples, and guide frames having other arbitrary shapes and sizes may be set and displayed as a guide frame image. Moreover, not only a frame image but another arbitrary image may be displayed. That is, any image may be displayed as long as the user can recognize whether or not an object (for example, a user's hand) included in the imaging range of the infrared camera 4 is in an appropriate position. For example, when the hand is in an appropriate position, a predetermined image is displayed at the edge of the screen, and when the hand is not in the appropriate position (when the hand is within the imaging range and outside the appropriate range) A predetermined image may be displayed blinking.

(Application control)
Next, application control based on an infrared image captured by the infrared camera 4 in the portable electronic device 1 will be described. FIG. 16 is a diagram illustrating an example of a menu screen displayed on the display 2 of the mobile electronic device 1. FIG. 17 is a diagram illustrating a state where the application B selected on the menu screen is activated.

  As shown in FIG. 16, a list of applications that can be executed in the mobile electronic device 1 is displayed on the menu screen. The portable electronic device 1 transmits and receives, for example, a game application for playing a predetermined game, an application for playing music and video, an application for making a call, an application for capturing and displaying an image, an e-mail, a message, and the like Arbitrary applications such as an application for browsing, a browser application for browsing a Web page on the Internet, and an application for creating and editing a document may be executable. Moreover, the program for performing the setting of the portable electronic device 1 from a menu screen may be executable.

  The user selects an icon corresponding to a desired application from icons indicating a plurality of applications displayed on the menu screen. For example, the user may be able to select an application using an input button, or may be able to select an application by performing a touch operation on the touch panel 3 using a finger or a touch pen. In addition, the user may be able to select a desired application from a plurality of applications displayed on the menu screen by performing a gesture on the right side surface of the mobile electronic device 1.

  For example, when the user brings his hand close to the right side in a state where the app B is selected as shown in FIG. 16, the app B is activated as shown in FIG. Specifically, the mobile electronic device 1 detects an object (for example, a user's hand) by analyzing an infrared image captured by the infrared camera 4. When the object is detected, the mobile electronic device 1 activates the selected application B. Specifically, the CPU of the control unit 14 of the portable electronic device 1 reads the application B program from a storage device (for example, a non-volatile memory) inside the portable electronic device 1 into the RAM, and starts executing the program. . Alternatively, the program of the application B may be stored outside the mobile electronic device 1, and the mobile electronic device 1 may acquire the program via the communication unit 15 and read it into the RAM.

  Note that the condition for starting the selected application is not limited to that the portable electronic device 1 has detected an object, but that the portable electronic device 1 has detected the user's hand, That the right side surface of the portable electronic device 1 is gripped by the user, the portable electronic device 1 has detected a specific object, and the distance between the portable electronic device 1 and the specific object is a predetermined value. It may be that the object is moving at a speed higher than a predetermined speed.

  For example, by analyzing an infrared image captured by the infrared camera 4, the mobile electronic device 1 can detect that the user has gripped the mobile electronic device 1 or the distance between the mobile electronic device 1 and an object. Further, the mobile electronic device 1 may determine the relative distance between the mobile electronic device 1 and the specific object based on the size of the specific object included in the infrared image acquired from the infrared camera 4. Good. For example, the mobile electronic device 1 may determine that the mobile electronic device 1 is gripped by the user based on the luminance information of the infrared image captured by the infrared camera 4 or may calculate the distance between the mobile electronic device 1 and the object. Good. The closer the distance between the object and the portable electronic device 1, the higher the luminance of each pixel of the infrared image. For this reason, the portable electronic device 1 can determine that the user has gripped, for example, when the region of the captured infrared image having a luminance equal to or higher than a predetermined value is equal to or higher than a predetermined ratio.

  Further, the mobile electronic device 1 can calculate the relative distance from the object based on the luminance of the object included in the infrared image, and calculates the speed of the object based on the change in the relative distance. be able to. Based on the distance and speed calculated in this way, the mobile electronic device 1 may start (start) the application.

  The activated application B is started when a predetermined condition is satisfied. For example, when the application B is a game application, the game is started on condition that a predetermined instruction for starting the game (an instruction using a button, a touch panel, a gesture, or the like) is given by the user. Alternatively, the game may be started on condition that the mobile electronic device 1 detects the user's hand. When the app B is a game using a gesture, the game cannot be performed unless the user's hand is detected. Therefore, the game start condition may be that the user's hand is detected. Alternatively, the game may be started on the condition that a predetermined time has elapsed since the activation of the application B.

  FIG. 18 is a diagram illustrating an example of a screen immediately after the game of the activated application B is started. As shown in FIG. 18, when the game is started, a display indicating the remaining time until the end of the game and a display indicating the points acquired during the game are performed on the display 2. For example, the app B is a game that aims to acquire more points within a predetermined time, and may be a game that a user plays alone, or other portable electronic devices 1 (or other The game may be a game in which a large number of people cooperate to acquire points by bidirectionally communicating with a device, or a game in which many people compete for points. For example, the application B is a game in which points are added when a user performs a gesture according to an instruction displayed on the display 2. In such a game, when the game is started after the application B is activated, an image shown in FIG. 18 is displayed.

  FIG. 19 is a diagram illustrating an example of screen transition while the game is being executed. As shown in FIG. 19A, when 30 seconds have elapsed from the start of the game, the display indicating the remaining time is “30” and the score is “75”. In addition, an instruction for the user is displayed on the screen, and the user performs a gesture according to the instruction. As shown in (B), when 2 seconds have passed and 32 seconds have passed since the start of the game, the remaining time is displayed as “28”, and points are also added when the user performs a gesture according to the instruction. The In this way, in this game, for example, instructions are displayed one after another according to a predetermined music rhythm, and when a gesture according to the displayed instructions is performed, points are added.

  Here, as shown in FIG. 19C, when the user's hand is out of the imaging range of the infrared camera 4 and the user's hand cannot be detected by the infrared camera 4, the game of the app B is temporarily stopped. And return to the menu screen. For example, when a predetermined waiting time has elapsed since the user's hand is no longer detected, the game screen of app B is switched to the menu screen. At this time, the game is paused internally, and the progress of the game is temporarily stopped. However, the game being executed is not terminated, and data relating to the execution of the game remains in the RAM. For example, a value indicating the progress of the game (for example, remaining time or score) is stored in the RAM, and the value indicating the progress is in accordance with the passage of time (or user operation) while the game is being executed. To increase. When the progress of the game is stopped, the increase (or decrease) of the value indicating the progress is also stopped. On this menu screen, the user can also select another desired application.

  Then, as shown in FIG. 19D, when the user's hand enters the imaging range of the infrared camera 4 again with the application B selected, the mobile electronic device 1 detects the user's hand and Resume the game. Specifically, the game of the app B starts again from the state when it is paused. For example, if the game is paused when the remaining time is 28 seconds and the score is 80 points, the game is resumed from the state where the remaining time is 27 seconds and the score is 80 points.

  More specifically, the mobile electronic device 1 resumes the game of the application B immediately after detecting the user's hand (or after a period shorter than the waiting time has elapsed). The mobile electronic device 1 waits for a predetermined time when the application is temporarily stopped, and resumes the application immediately without waiting for the predetermined time when the application is resumed. It is possible to control the pause / resume of the application. That is, in an application that performs a gesture using a hand, the user may unintentionally move away from the imaging range of the infrared camera 4 for an instant. In such a case, if the currently running application is paused immediately, the application is temporarily stopped against the user's intention. However, when the application is paused, the waiting time is set, so that the application can be prevented from being unintentionally paused. On the other hand, when resuming the application from the menu screen, the user intentionally puts his hand in the imaging range of the infrared camera 4. For this reason, the portable electronic device 1 resumes an application immediately, without providing the said standby | waiting time, when resuming an application.

  Note that the mobile electronic device 1 may resume the application after a predetermined waiting time has elapsed after detecting the user's hand, as in the case of temporarily stopping the application. Also, the mobile electronic device 1 may pause / resume the application immediately without waiting for a predetermined time, whether the application is temporarily stopped or the application is resumed. In addition, when the application is temporarily stopped as illustrated in FIG. 19C, if the application is not resumed even after a predetermined time has elapsed, the mobile electronic device 1 may end the application. In this case, even if the user puts his hand in the imaging range of the infrared camera 4 again, the application is started from the beginning.

  As described above, in the portable electronic device 1 of the present embodiment, an interactive application whose progress is controlled according to a user input is executed. Specifically, the application proceeds in response to a gesture input performed in the side surface direction of the mobile electronic device 1. For example, as an example of an interactive application, a game is performed in which success / failure determination with respect to a gesture input is performed, and points are added according to the result of the determination of success / failure. The application is activated and started when the user holds his / her hand in the lateral direction of the portable electronic device 1 or performs a gesture using the hand. When the user removes the hand in the side direction of the mobile electronic device 1, the running application is temporarily stopped and a menu screen is displayed. On the menu screen, the user can select another desired application. When the user holds his / her hand again in the side surface direction of the portable electronic device 1 with the paused application selected, the paused application is resumed.

  In the above-described example, when the user removes the hand while executing the application, that is, when the user's hand is no longer detected in the infrared image captured by the infrared camera 4, the menu screen is displayed. In another example, for example, the menu screen may be displayed when the user moves (or moves away from) the right hand of the mobile electronic device 1 at a predetermined speed. Further, when the user performs a predetermined gesture, a menu screen may be displayed. For example, the menu screen may be displayed when the application being executed is paused when the user performs a specific gesture and the hand is moved in that state.

  FIG. 20 is a diagram illustrating an example of a gesture for displaying a menu screen according to another embodiment. For example, as illustrated in FIG. 20, when the user performs a gesture such as grasping an object in the right side surface direction of the mobile electronic device 1, the application being executed is temporarily stopped. In this state, when the user moves his hand in the right direction, a part of the menu screen comes out from the left end of the display 2. When the user further moves the hand to the right, the entire screen of the display 2 is switched to the menu screen.

  Further, in the above, when the user holds the hand over the infrared camera 4, the application selected on the menu screen is started (resumed). In other embodiments, the selected application may be initiated, for example, when the user makes a predetermined gesture or when the hand is moved at a predetermined speed to prevent malfunction. Moreover, when the right side surface of the portable electronic device 1 is gripped by the user, the selected application may be started. In this case, whether or not the right side surface of the portable electronic device 1 is gripped by the user may be determined based on luminance information of an image captured by the infrared camera 4, for example, or may be determined based on a change in luminance. May be. Moreover, the portable electronic device 1 may start (resume) an application, when a user's pulse is detected. For example, the user's pulse can be detected based on a change in luminance of an image captured by the infrared camera 4. A malfunction can be prevented by starting an application when a user's pulse is detected.

  Moreover, the conditions regarding the detection of an object may differ by the starting of an application and the start of the predetermined process in an application. In addition, the conditions regarding the detection of the object may be different between the suspension of the application and the termination of the application. For example, when a first condition relating to object detection is satisfied, the application is started. When a second condition that is stricter than the first condition is satisfied, predetermined processing in the application is started. May be. In this case, when the first condition is satisfied and the second condition is not satisfied, the predetermined process in the application is not started even when the application is activated. Further, for example, the application may be temporarily stopped when the first condition is satisfied, and the application may be ended when the second condition that is stricter than the first condition is satisfied. In this case, when the first condition is satisfied and the second condition is not satisfied, the application is not terminated even if the application is paused.

  For example, the application B is activated when the portable electronic device 1 detects an object, and the activation screen (initial screen) of the application B is displayed as shown in FIG. This activation screen is a screen before a predetermined process (game process) in application B is started. Then, after the startup screen is displayed, for example, when the mobile electronic device 1 detects the user's hand, the game process is started. Specifically, when any object (any object regardless of whether it is a hand) is detected in the infrared image captured by the infrared camera 4, the startup screen of the application B is displayed, and the mobile electronic device is displayed in that state. When the device 1 detects the user's hand, the game starts. In other words, the condition for the startup screen of application B to start up is that the user's hand does not need to be detected, that some object is detected, and the condition for starting the game processing in application B is as follows. The user's hand is detected. This is because even if the user's hand enters the imaging range of the infrared camera 4, even if the portable electronic device 1 can recognize that some object exists if the distance between the hand and the infrared camera 4 is too short, It is not possible to recognize whether it is a hand. In such a case, if the startup screen of the application B does not stand up and remains on the menu screen, it becomes difficult for the user to know what operation the application B starts up. For this reason, even if the portable electronic device 1 cannot recognize the user's hand (even if the second condition is not satisfied), if any object is detected (the first condition is satisfied). In this case, the portable electronic device 1 starts up the startup screen of the app B. However, since the game is a game using a gesture, the mobile electronic device 1 needs to recognize the user's hand in order to perform actual game processing. Therefore, the condition for starting the game is that the user's hand can be recognized. In this way, the application is started if the first condition related to object detection is satisfied, and after the application starts, if a second condition that is stricter than the first condition is satisfied, a predetermined process (game) in the application is performed. Processing) may be started.

  Similarly, when the application is paused and terminated, if the first condition relating to object detection is satisfied, the application is paused. After the application is paused, the second condition that is stricter than the first condition If satisfied, the app may be terminated.

  Further, the application to be executed is not limited to the above-described game. For example, a game in which a predetermined game object is displayed and the predetermined game object is controlled by the movement of the user's hand may be played. For example, the mobile electronic device 1 may control the movement of the game object based on the position of the image captured by the infrared camera 4 (for example, the barycentric position of an area having a luminance of a predetermined value or more).

  Moreover, in the said embodiment, when the portable electronic device 1 stopped detecting a user's hand during execution of an application, the application was paused and the menu screen was displayed. In another embodiment, when the mobile electronic device 1 no longer detects the user's hand during the execution of the application, the application may be paused and a screen as to whether to end the application may be displayed. If the user instructs to end the application on this screen, the application may be ended. In addition, even if there is no user instruction after the application is paused, the application may be terminated when a predetermined time has elapsed.

(Details of guide display processing)
Next, an example of specific processing performed in the mobile electronic device 1 will be described. First, with reference to FIG. 21, the detail of the process for performing the guide display mentioned above is demonstrated. FIG. 21 is a flowchart illustrating an example of processing for performing guide display. The control unit 14 (the CPU thereof) reads a program stored in the storage device at a predetermined timing (for example, a timing before the user's instruction or application is started) into the RAM, and executes the program as shown in FIG. The process shown in is performed. Note that the processing shown in FIG. 21 is repeatedly performed at predetermined time intervals (for example, once every 1/60 seconds).

  As shown in FIG. 21, in step S <b> 10, the control unit 14 acquires an infrared image from the infrared camera 4. Next, the control part 14 performs the hand detection process which detects the user's hand by analyzing the infrared image acquired at step S10 (step S11). If the hand can be detected as a result of the hand detection process (step S12: YES), the control unit 14 sets a fixed guide frame in the acquired infrared image (step S13). Specifically, the control unit 14 sets a guide frame having a certain size that covers the central region of the acquired infrared image.

  Then, the control part 14 produces | generates the outline image which shows the outline of the user's hand contained in the acquired infrared image (step S14). Specifically, when there are a plurality of objects in the acquired infrared image, the control unit 14 extracts only the object having the highest luminance and detects the contour of the extracted object. Then, the control unit 14 deletes pixel information other than the contour while leaving the detected contour. Next, the control part 14 sets the pixel information of the part which protruded from the guide frame set by step S13 among the produced | generated outline images to red (step S15). Subsequently, the control unit 14 synthesizes a guide frame image indicating the guide frame set in step S13 with the contour image (step S16). And the control part 14 produces | generates the image which looked at the synthetic | combination synthesized image from diagonal, and displays it on the display 2 (step S17). Specifically, the control unit 14 performs processing for the perspective display described with reference to FIG. 15, for example, to display an image obtained by rotating the synthesized image by a predetermined angle around the Y axis in FIG. 10 on the display 2. Is displayed at the right end of.

  On the other hand, if the hand cannot be detected as a result of the hand detection process in step S11 (step S12: NO), the control unit 14 displays an error on the display 2 (step S18). For example, the control unit 14 may display a character indicating that no hand has been detected on the display 2 and notify the user that the hand has not been detected by voice, vibration, or the like.

  After performing the process of step S17 or step S18, the control unit 14 determines whether or not to end the process shown in FIG. 21 (step S19). For example, the process shown in FIG. 21 may be ended according to a user instruction, or may be ended when a predetermined time has elapsed. When it is determined that the control unit 14 does not end the process shown in FIG. 21 (step S19: NO), the process of step S10 is executed again, and when it is determined that the process shown in FIG. 21 is ended (step S19: YES). ), The process shown in FIG.

(Details of menu display processing)
Next, the details of the process for displaying the menu screen described above will be described with reference to FIG. FIG. 22 is a flowchart illustrating an example of processing for displaying a menu screen. Note that the processing shown in FIG. 22 is performed by the control unit 14 (or its CPU) executing a program for displaying a menu. Further, the process shown in FIG. 22 is repeatedly performed at a predetermined time interval (for example, once every 1/60 seconds).

  As shown in FIG. 22, in step S <b> 20, the control unit 14 displays a menu screen on the display 2. For example, the control unit 14 acquires a list of applications that are currently executable in the mobile electronic device 1 and displays icons indicating the applications side by side on the screen. The application executable in the mobile electronic device 1 is, for example, an application stored in a storage device inside the mobile electronic device 1 or an application stored in an external storage device accessible from the mobile electronic device 1. Also good.

  Following step S20, the control unit 14 performs an application selection process (step S21). For example, the control unit 14 determines whether an icon corresponding to the application is selected (touched) by the user based on the position detected by the touch panel 3. When it is determined that an icon has been selected by the user, the control unit 14 selects an application corresponding to the selected icon.

  Next, the control unit 14 acquires an infrared image from the infrared camera 4 (step S22). Next, the control unit 14 performs a hand detection process for detecting the user's hand by analyzing the infrared image acquired in step S22 (step S23). If the hand can be detected as a result of the hand detection process (step S24: YES), the control unit 14 determines whether or not the application has been selected (step S25). Specifically, the control unit 14 determines whether or not an application has already been selected by the user in step S21. When the application has been selected (step S25: YES), the control unit 14 determines whether or not the selected application has already been activated (step S26). Specifically, when the application has already been started, the information on the application (the program itself and other data) is stored in the RAM, so the control unit 14 is based on the information stored in the RAM. It is determined whether the application has already been started.

  On the other hand, as a result of the hand detection process, when the hand cannot be detected (step S24: NO), or when the application has not been selected (step S25: NO), the control unit 14 performs the process of step S20. Do it again.

  When the application has not been activated (step S26: NO), the control unit 14 activates the application selected in step S21. When the application has already started (step S26: YES), the control unit 14 resumes the execution of the selected application (step S28).

  After the process of step S27 or step S28, the control unit 14 ends the process shown in FIG.

(Details of application control processing)
Next, with reference to FIG. 23, details of the process for controlling the above-described application will be described. FIG. 23 is a flowchart illustrating an example of the application control process. The process illustrated in FIG. 23 is performed, for example, when the user holds his / her hand on the right side surface of the mobile electronic device 1 as described above. Further, the process shown in FIG. 23 is repeatedly performed at a predetermined time interval (for example, once every 1/60 seconds).

  As shown in FIG. 23, in step S30, the control unit 14 starts (or resumes) the application. Specifically, when the application is not activated (when the application program is not read into the RAM or when the initial setting for starting the application is not performed), the control unit 14 activates the application. Then, the application is started in response to establishment of a predetermined condition (a predetermined time has elapsed or a user instruction has been given). When the application is paused (that is, when the menu screen is displayed after the application has already been started), the control unit 14 resumes the application. Specifically, the control unit 14 reads data for the application stored in the RAM, and resumes the application from the state where the application is temporarily stopped in step S38 described later.

  Following step S30, the control unit 14 acquires an infrared image from the infrared camera 4 (step S31). Next, the control unit 14 performs a hand detection process for detecting the user's hand by analyzing the infrared image acquired in step S31 (step S32). If the hand is detected as a result of the hand detection process (step S33: YES), the control unit 14 performs a gesture determination process for determining a gesture input performed by the user (step S34). Specifically, the control unit 14 determines which of the plurality of predetermined gestures has been performed based on the detected hand shape and movement.

  Subsequently, the control unit 14 performs application processing according to the determined gesture (step S35). For example, when the game application shown in FIG. 19 is performed, the control unit 14 displays characters, images, characters, and the like indicating instructions for the user on the display 2 and determines whether or not a gesture according to the instructions has been performed. judge. Then, the control unit 14 displays the determination result on the display 2. For example, when it is determined that the gesture according to the instruction has been performed, the control unit 14 adds the points and displays the added points on the display 2. In addition, the control unit 14 measures the elapsed time from the start of the application and displays it on the display 2. Note that the application process in step S35 may be performed based on, for example, an operation on the input button or the attitude of the mobile electronic device 1 in addition to (or instead of) the gesture input performed by the user.

  Following step S35, the control unit 14 determines whether or not to end the application (step S36). For example, when the remaining time indicating the time until the application ends is “0”, the control unit 14 determines to end the application. For example, when the user performs an operation for ending the application (button operation or gesture operation), the control unit 14 determines to end the application.

  When it is not determined to end the application (step S36: NO), the control unit 14 executes the process of step S31 again. On the other hand, when it is determined that the application is to be terminated (step S36: YES), the control unit 14 performs an application termination process (step S40) and terminates the process illustrated in FIG. For example, the control unit 14 releases the RAM area reserved for executing the application program as the application termination process. As a result, values (for example, the remaining time and the score) corresponding to the progress status of the application are cleared.

  On the other hand, if the user's hand could not be detected in the hand detection process of step S32 (step S33: NO), has the predetermined waiting time elapsed since the control unit 14 stopped detecting the user's hand? It is determined whether or not (step S37). When the predetermined standby time has not elapsed since the user's hand is no longer detected (step S37: NO), the control unit 14 executes the process of step S31 again. Note that the predetermined standby time may vary depending on the type of application, may be a fixed time, or may vary depending on the hand detection status.

  When a predetermined waiting time has elapsed since the user's hand is no longer detected (step S37: YES), the control unit 14 temporarily stops the running application (step S38). Specifically, the application is suspended while maintaining the RAM area reserved for executing the application. For example, when an application image is displayed on the display 2, the control unit 14 stores the image in the RAM. Further, the control unit 14 maintains a value (for example, remaining time, score, etc.) according to the progress of the application.

  When the application is temporarily stopped in step S38, the control unit 14 calls a program for performing the menu process described above (step S39), and ends the process shown in FIG. Thereby, the menu process shown in FIG. 22 is performed.

  Note that the processing shown in the flowchart is merely an example, and the order and contents of the processing may be appropriately changed. Moreover, the application mentioned above is only an example, and other arbitrary applications may be performed.

(Modification)
In addition, embodiment mentioned above is only an example, For example, the following modifications may be added.

  For example, the infrared camera 4 is not limited to the right side surface of the portable electronic device 1, but may be provided on any of the left side surface (T2 surface in FIG. 1), the upper side surface (T4 surface), and the lower side surface (T3 surface). Gesture input may be performed by the user in the side direction. Further, the infrared camera 4 may be provided on a plurality of side surfaces. For example, when the infrared camera 4 is provided on the left side surface of the mobile electronic device 1, the above-described guide display (the contour image 21 and the guide frame image 22) is displayed on the left side of the screen of the display 2. Further, the display position of the guide display may be different depending on the posture of the mobile electronic device 1. The portable electronic device 1 can detect the posture of the mobile device 1 by the posture detection unit 16, and can detect, for example, how many times it has been rotated about an axis perpendicular to the screen by an angular velocity sensor or an acceleration sensor. For example, as shown in FIG. 6A, when the infrared camera 4 is normally held so as to be positioned on the right side, when the user holds the portable electronic device 1 so that the left and right are reversed, the infrared camera 4 is positioned on the left side. To come. In this case, the portable electronic device 1 is gripped with the right hand, and gesture input is performed with the left hand. In such a case, the guide display is displayed on the left side of the display 2. Specifically, a parsed guide display is displayed on the left side of the display 2 so that the user can see the same view as when the user looks directly at the left hand. That is, the guide display is displayed on the same side as the side surface imaged by the infrared camera 4 in the display screen of the display 2.

  Moreover, in the said embodiment, it assumed that the portable electronic device 1 provided with the display 2 performs the said process. In other embodiments, the image may be displayed on an external display. The user holds the handheld input device provided with the infrared camera 4, and the user performs a gesture using the hand in the side direction of the input device, so that the guide display described above is displayed on the display device separated from the input device. An image, an application image, a menu screen, or the like may be displayed. An information processing system in which the handheld input device provided with the infrared camera 4, the display device, and the information processing device are separated may be constructed. In this case, for example, the guide display image and the application image described above are displayed on the display device, and the processing of the application is performed by the information processing device. For example, the information processing device acquires an image from the input device, recognizes a gesture based on the image, and performs application processing based on the recognition result of the gesture. Further, the information processing apparatus acquires an image from the input device and performs processing for the guide display (drawing of the contour image 21 and the guide frame image 22 and processing for parsing these images). Then, the information processing apparatus outputs an image as a result of processing for application and processing for guide display to the display device. The processing described above may be executed by each of the plurality of devices sharing the processing. Moreover, you may make the portable electronic device 1 function as a controller which receives a user's input. In this case, a display may or may not be provided. When the portable electronic device 1 functions as a controller, the portable electronic device 1 is connected to another information processing apparatus wirelessly or by wire. When an operation (an operation on an input button, an operation of moving the mobile electronic device 1 itself, the gesture operation, or the like) is performed on the mobile electronic device 1, operation data corresponding to the operation is transmitted to another information processing apparatus. . When receiving the operation data, the information processing apparatus performs processing according to the operation data. Processing according to operation data differs depending on the type of application to be executed.

  Further, the mobile electronic device 1 may have any shape. For example, the mobile electronic device 1 (display 2) may be vertically long instead of horizontally long, or may be square.

  Moreover, although the infrared camera 4 was used in the said embodiment, it replaces with the infrared camera 4 and the camera which can image the image (RGB image) of visible light may be used in other embodiment.

  In the above-described embodiment, an example has been described in which a portable electronic device is held with one hand and a gesture input is performed on the portable electronic device with the other hand. In another embodiment, the portable electronic device may be fixed to a user's arm (body), for example, a wristwatch-type device having a screen.

  The portable electronic device may have any shape, for example, a plate-like ellipse. For example, the portable electronic device may be foldable.

DESCRIPTION OF SYMBOLS 1 Portable electronic device 2 Display 3 Touch panel 4 Infrared camera 5 Distance sensor 6 Input button 7 Irradiation part 8 Projector 15 Communication part 21 Outline image 22 Guide frame image 41 Composite image

Claims (24)

A handheld electronic device,
A camera that images the lateral direction of the electronic device;
Obtaining means for obtaining an input image captured by the camera;
An electronic apparatus comprising: control means for performing control of at least one of start, pause, and restart of an application based on the input image obtained by the obtaining means.   The electronic device according to claim 1, wherein the application is an interactive application in which processing is performed according to an input by a user.   The electronic apparatus according to claim 1, further comprising an application control unit that controls the progress of the application being executed based on the input image acquired by the acquisition unit. Further comprising detection means for detecting an object based on the input image acquired by the acquisition means;
The electronic device according to claim 1, wherein the control unit performs the control of the application based on a detection result of the object by the detection unit.   The electronic device according to claim 4, wherein the detection unit detects a user's hand. The detection means detects a gesture made by a user's hand,
The electronic device according to claim 5, wherein the control unit performs the control of the application based on a gesture detected by the detection unit.   The electronic device according to claim 4, wherein the control unit starts the application when the detection unit detects the object.   The electronic device according to claim 4, wherein when the object is not detected by the detection unit, the control unit temporarily stops an application being executed.   The electronic device according to claim 8, wherein the control unit temporarily stops the application when a predetermined period has elapsed after the detection unit no longer detects the object.   The electronic device according to claim 8, wherein the control unit resumes the paused application when the object is detected by the detection unit after the currently running application is paused.   10. The electronic device according to claim 4, wherein when the object is not detected by the detection unit, the control unit starts or restarts a second application different from the application being executed. 11.   The electronic device according to claim 11, wherein when the object is detected by the detection unit, the control unit starts or restarts the application that was executed before the second application was started. The detection means detects whether or not the electronic device is gripped by a user based on the input image,
The electronic device according to claim 4, wherein the control unit starts the application when the detection unit detects that the electronic device is held by a user.   14. The electronic apparatus according to claim 4, further comprising application control means for controlling the progress of the application being executed based on the detection result of the object by the detection means. Further comprising distance calculation means for calculating a distance between the object and the camera based on the input image;
The electronic device according to claim 4, wherein the control unit performs the control of the application based on the distance calculated by the distance calculation unit. A speed calculating means for calculating the speed of the object based on the input image;
The electronic device according to claim 4, wherein the control unit performs the control of the application based on the speed calculated by the speed calculation unit.   The electronic device according to claim 1, further comprising notification means for notifying a user that the application is suspended when the application is suspended by the control means. A selection means for selecting any one of a plurality of applications according to a user input;
The electronic device according to claim 1, wherein the control unit starts an application that has already been selected by the selection unit.   The control unit according to any one of claims 1 to 18, wherein when the application is paused, the control unit receives an instruction by a user as to whether or not to end the application, and ends the application according to the instruction from the user. The electronic device described.   The electronic device according to claim 1, wherein when the application is temporarily stopped, the control unit ends the application according to a lapse of a predetermined time.   The electronic device according to claim 1, wherein the camera is an infrared camera.   The electronic device according to claim 1, wherein the camera is provided on a side surface of the electronic device. Obtaining means for obtaining an input image captured by the camera;
An application execution means for executing an interactive application that proceeds based on a user operation;
An electronic apparatus comprising: control means for temporarily stopping the application being executed based on the input image obtained by the obtaining means. A handheld electronic device,
A camera that images the lateral direction of the electronic device;
Obtaining means for obtaining an input image captured by the camera;
An electronic device comprising: an application activation unit that activates an application based on the input image acquired by the acquisition unit.

Images