JP6528193B2 - Electronics - Google Patents

Description

  The present invention relates to a hand-held electronic device capable of performing predetermined information processing.

  BACKGROUND Conventionally, there has been a portable game device provided with a display and a camera provided on the back of the display (e.g., Patent Document 1). In such a conventional game device, a captured image can be acquired by a camera, and an image based on a result of analysis of the acquired image can be displayed on a display.

Unexamined-Japanese-Patent No. 2006-271663

  However, in the above-mentioned conventional apparatus, an image obtained by capturing the back direction of the display using a camera is displayed on the display, and an improvement is made in that the captured image from the camera capturing another direction is displayed easily for the user. There was room for

  Therefore, one of the objects of the present invention is to provide an electronic device capable of displaying an image from a camera for imaging a predetermined direction in an easy-to-view manner for the user.

  In the present invention, the following configuration is adopted in order to solve the above problems.

  An electronic device according to an embodiment includes a camera, an acquisition unit, a display control unit, and a processing unit. The camera captures the side direction of the electronic device. Acquisition means acquires an input image captured by the camera. The display control means causes the display means to display an image in which the subject indicated by the input image acquired by the acquisition means is inclined in a predetermined direction. The processing means performs predetermined application processing based on the input image acquired by the acquisition means.

  According to the above, when displaying an image of a subject captured by a camera capturing an image in the side direction on the display device, it can be displayed by being tilted in a predetermined direction, and for example, display that is easy for the user to view can be performed.

  In addition, in another configuration, the display control means may cause the display means to display an image obtained by viewing the input image obliquely.

  According to the above, by displaying an image obtained by viewing the captured input image obliquely, an object present in the side direction of the electronic device can be displayed without a sense of incongruity, and the display can be easily viewed by the user.

  In addition, in another configuration, the display control unit may display the image in a manner that allows the user to recognize the relative position between the camera and an object included in the imaging range of the camera.

  According to the above, the position of the object present in the side direction of the electronic device can be displayed so as to be recognizable by the user.

  Further, in another configuration, the display control means may display the image in a manner that allows the user to recognize the position in the depth direction of the object included in the imaging range of the camera.

  According to the above, the position in the depth direction of the object present in the side direction of the electronic device can be displayed so as to be recognizable by the user.

  In another configuration, the display control unit may be configured such that the user directly looks at an object included in the imaging range of the camera and the image of the object displayed on the display unit looks the same. The image may be displayed on the display means.

  According to the above, the appearance of the object viewed from the user is the same as the appearance of the object displayed on the display device, which makes the user easy to see.

  In another configuration, the display control unit may display a guide image that allows the user to recognize whether an object included in the imaging range of the camera is at an appropriate position.

  According to the above, the guide image can make the user recognize whether or not the object is at the appropriate position, and can guide the user to the appropriate position.

  In addition, in another configuration, the display control unit may further cause the display unit to display a guide image indicating a predetermined area in the imaging range of the camera.

  According to the above, by displaying the guide image, for example, an object present in the side direction can be easily accommodated within the imaging range of the camera.

  Further, in another configuration, the display control means may cause the display means to display an image in which the guide image and the input image are inclined in a predetermined direction.

  According to the above, since the input image and the guide image are displayed being inclined in a predetermined direction, for example, the display can be made more easily viewable by the user, and the object can be easily set within the imaging range of the camera. .

  In addition, in another configuration, the display unit may be provided in the electronic device. The camera captures the side direction of the display means. The display control means causes the image to be displayed on the imaging direction side of the camera on the display screen of the display means.

  According to the above, the imaging direction of the camera and the display position of the image on the display screen of the display means can be made to coincide. For example, when the right side surface direction of the display device is captured by a camera, an image captured by the camera can be displayed on the right side of the display screen of the display device. Therefore, for example, the user can easily recognize the positional relationship of the object present in the side direction.

  In another configuration, the display control unit may display an image indicating the result of the predetermined application processing together with an image in which the subject indicated by the input image acquired by the acquisition unit is inclined in a predetermined direction. Good.

  According to the above, it is possible to make the user easily recognize the positional relationship between the subject and the camera and the result of the application processing based on the positional relationship.

  In addition, in another configuration, the display control unit may cause the display unit to display an image obtained by abstracting an object included in an imaging range of the camera.

  According to the above, the object included in the imaging range of the camera can be abstracted and displayed, and for example, the display can be easy for the user to view.

  Further, in another configuration, the display control means may cause the display means to display an image indicating an outline of an object included in an imaging range of the camera.

  Further, in another configuration, the display control means may cause the display means to display only a specific object among a plurality of objects included in an imaging range of the camera.

  According to the above, it is possible to display only a specific object, and it is possible to prevent the display from being obscured by the display of other objects.

  In another configuration, the apparatus may further include detection means for detecting a user's hand based on the input image. The processing means performs the application processing based on the detection result by the detection means.

  According to the above, the user's hand can be detected, and application processing can be performed based on the hand detection result.

  In another configuration, the detection means may detect a gesture performed by the user.

  According to the above, the gesture performed by the user can be detected, and for example, the application processing can be performed by the input using the gesture.

  Further, in another configuration, the image processing apparatus may further include a determination unit that determines whether at least a part of the object included in the input image is out of a predetermined area of the imaging range of the camera. The display control means causes the display means to display an image according to the determination result by the determination means.

  According to the above, it is possible to make the user recognize whether or not a part of the object is out of the predetermined area of the imaging range.

  Further, in another configuration, the display control means may display or hide the image according to an execution state of a predetermined application process performed by the processing means.

  According to the above, the image can be displayed or not displayed according to the execution status of the application or the user's selection. For example, the image can be displayed only before the start of the application or can be displayed only at the time of the tutorial .

  In another configuration, the electronic device may be a hand-held electronic device.

  In another configuration, the camera may be an infrared camera capable of receiving infrared light.

  In another embodiment, an input device, a camera for capturing a side direction of the input device, an acquisition unit for acquiring an input image imaged by the camera, and an input image acquired by the acquisition unit are used. An electronic device system comprising: display control means for displaying on a display means an image obtained by tilting a subject to be moved in a predetermined direction; and processing means for performing predetermined application processing based on an input image acquired by the acquisition means. It is also good. The system may be configured by a plurality of devices or may be configured by one device.

  Further, the electronic device according to one embodiment includes a camera, an acquisition unit, an input image display unit, a setting unit, an area display unit, a determination unit, and a display mode control unit. Acquisition means acquires an input image captured by the camera. The input image display means causes the display means to display the input image acquired by the acquisition means. The setting means sets a fixed area which is an area smaller than the imaging range of the camera and is fixed to the imaging range. The area display means causes the display means to display an area image indicating the fixed area. The determination means determines whether or not at least a part of the object included in the input image is out of the fixed area. The display mode control means controls the display mode of the image displayed on the display means based on the determination result by the determination means.

  According to the above, for example, it is possible to make the user recognize whether or not an object has come out of the fixed area, and to make the user recognize that the object deviates from the imaging range of the camera.

  In another configuration, the camera may capture the side direction of the electronic device.

  In addition, the electronic device according to one embodiment includes a camera, an acquisition unit, a detection unit, a processing unit, a condition determination unit, and a warning unit. Acquisition means acquires an input image captured by the camera. The detection means detects a specific object included in the input image acquired by the acquisition means. The processing means performs predetermined information processing when the specific object is detected by the detection means. The condition determining means determines whether or not a predetermined condition is satisfied based on the input image. The warning means gives a warning when it is judged by the condition judging means that the predetermined condition is satisfied, when the specific object is detected by the detection means. The processing means performs the predetermined information processing based on the input image even when the condition determination means determines that the predetermined condition is satisfied.

  According to the above, for example, even when a specific object is detected, when it is determined that the predetermined condition is satisfied based on the input image, a warning can be given, for example, The user can be warned in advance before the predetermined information processing based on the input image is not performed.

  In another configuration, the apparatus may further include a second warning unit that gives a warning different from the warning by the warning unit when the specific object is not detected by the detection unit.

  In another configuration, the apparatus may further include stop means for stopping the predetermined information processing by the processing means when the specific object is not detected by the detection means.

  According to an example of the present invention, when displaying an image captured by a camera capturing an image in the side direction on a display device, it can be displayed by being tilted in a predetermined direction, and for example, display that is easy for the user to view can be performed.

Front view of the portable electronic device 1 of the present embodiment Right side view of portable electronic device 1 Rear view of portable electronic device 1 Block diagram showing an example of the internal configuration of the portable electronic device 1 FIG. 6 is a view showing an example of the usage state of the portable electronic device 1, showing a state in which the user holds the portable electronic device 1 with both hands. A diagram showing a user performing a gesture input using the right hand in the right side direction of the portable electronic device 1 A diagram showing a user performing a gesture input using the right hand in the right side direction of the portable electronic device 1 The figure which shows an example of the image imaged by the infrared camera 4 at the time of gesture input as shown to FIG. 6A being performed. The figure which shows an example of the image imaged by the infrared camera 4 at the time of gesture input as shown to FIG. 6B being performed. The figure which shows an example of the guide display of this embodiment The figure which shows an example of the infrared image imaged with the infrared camera 4 in the state shown in FIG. A diagram showing an example of an image after the contour drawing process and the guide frame image drawing process when the image shown in FIG. 9 is acquired The figure which shows the state which the user moved the hand upwards (Y-axis direction) from the state shown in FIG. The figure which shows the state which the user moved the hand to the back direction (Z-axis direction) of a screen from the state shown in FIG. The figure which shows the synthetic | combination image 41 which synthesize | combined the outline image 21 and the guide frame image 22 in the state shown in FIG. The figure which shows the synthetic | combination image 41 which synthesize | combined the outline image 21 and the guide frame image 22 in the state shown in FIG. Schematic diagram showing an example of processing for performing perspective display The figure which shows an example of the menu screen displayed on the display 2 of the portable electronic device 1. Diagram showing application B selected by the menu screen started up A diagram showing an example of a screen immediately after the game of the started application B is started Diagram showing an example of screen transition while a game is being executed The figure which shows an example of the gesture for displaying the menu screen in other embodiment Flow chart showing an example of processing for performing guide display Flowchart showing an example of processing for displaying a menu screen Flow chart showing an example of application control processing

  Hereinafter, a portable electronic device according to an embodiment of the present invention will be described. The portable electronic device is a hand-held information processing apparatus which can be operated by the user by hand and may be, for example, a game device, a portable telephone (such as a smart phone), a tablet terminal, a camera, and a wristwatch type terminal Or any other device. FIG. 1 is a front view of a portable electronic device 1 of the present embodiment. FIG. 2 is a right side view of the portable electronic device 1. FIG. 3 is a rear view of the portable electronic device 1.

  As shown in FIG. 1, the portable electronic device 1 includes a display 2, a touch panel 3, an infrared camera 4, a distance measuring sensor 5, input buttons 6 (6 A to 6 D), an irradiation unit 7, and a projector 8. , Which are housed in the housing 10. The housing 10 (portable electronic device 1) has a plate-like shape, and has a size that can be gripped by the user with one hand or both hands.

  As the display 2, for example, a liquid crystal display device, an organic EL display device, or the like may be used, and any other display device may be used. The screen of the display 2 is provided so as to be exposed on the front surface (surface T5) of the housing 10. The touch panel 3 is provided on the screen of the display 2 and detects the position on the screen touched by the user. As the touch panel 3, a touch panel capable of single-point detection or a touch panel capable of multi-point detection may be used. For example, an arbitrary type such as an electrostatic capacitance system or a resistance film system may be used.

  The input buttons 6A to 6D receive input (depression) by the user. The input buttons 6A to 6D are provided at positions where the user's finger can reach when the user grips both ends of the portable electronic device 1. Specifically, when the user grips the portable electronic device 1 with the right hand, the input buttons 6A and 6C are in a position where the finger of the right hand reaches, the input button 6A is in a position where the thumb of the right hand reaches, and the input button 6C is It is provided at a position where the index finger or the middle finger of the right hand can reach. In addition, the input buttons 6B and 6D are in a position where the finger of the left hand reaches when the user grips the portable electronic device 1 with the left hand, the input button 6B is in a position where the thumb of the left hand reaches, and the input button 6D is the forefinger of the left hand Or it is provided in the position which a middle finger reaches. As shown in FIG. 1, the input buttons 6A and 6B are provided on the front surface (surface T5) of the housing 10, and the input buttons 6C and 6D are provided on the top surface (surface T4) of the housing 10. In addition to the input buttons 6A to 6D, a cross key for direction input, an analog stick, or the like may be provided as an input unit that receives an input by the user.

  The infrared camera 4 includes a lens and a sensor that is sensitive to light (infrared, specifically, near infrared). The sensor of the infrared camera 4 is an image sensor in which a plurality of elements sensitive to infrared light are arranged vertically and horizontally. Each element of the image sensor receives infrared light and converts it into an electrical signal to output a two-dimensional infrared image. .

  The distance measuring sensor 5 reflects a light (for example, infrared light) emitted from a light source provided to the distance measuring sensor 5 to an object, and the reflected light is received by the light receiving element to obtain a distance between the light and the object. Measure As the distance measuring sensor 5, any sensor such as a triangular distance measuring method or a TOF (Time Of Flight) method may be used. As a light source of the distance measurement sensor 5, an LED, a laser diode or the like which emits infrared light in a specific direction is used.

  The irradiation unit 7 irradiates infrared light at a predetermined time interval (for example, 1/60 second interval). The irradiation unit 7 emits infrared light in synchronization with the timing at which the infrared camera 4 captures an image. The irradiation unit 7 irradiates infrared rays to a predetermined range in the right side surface direction of the portable electronic device 1. The infrared light emitted by the irradiation unit 7 is reflected by the object, and the reflected infrared light is received by the infrared camera 4 to obtain an infrared image. The irradiation unit 7 may be used to capture an infrared image by the infrared camera 4 and to measure the distance by the distance measurement sensor 5. That is, the infrared camera 4 may capture an image using the infrared light from the irradiation unit 7 and the distance measuring sensor 5 may measure the distance.

  The projector 8 also has a light source for emitting visible light, and uses the light from the light source to project characters, images, etc. onto a projection surface (screen, user's hand, etc.).

  The infrared camera 4, the distance measurement sensor 5, the irradiation unit 7, and the projector 8 are provided on the side surface of the housing 10 (for example, right side surface: T1 surface). Specifically, the imaging direction (optical axis) of the infrared camera 4 is in the direction perpendicular to the right side surface. The detection direction of the distance measurement sensor 5 and the direction in which the projector 8 emits light are also directions perpendicular to the right side surface. That is, when the user holds the portable electronic device 1 with the left hand, the infrared camera 4 captures a space in the direction of the right side surface of the portable electronic device 1, and the distance measuring sensor 5 measures the direction of the right side surface of the portable electronic device 1. Measure the distance to an object present in space. In addition, the projector 8 projects an image or the like by emitting visible light in the same direction as the infrared camera 4 and the distance measuring sensor 5.

  An outer camera 9 is provided on the back surface (surface T6) of the portable electronic device 1 (FIG. 3). The outer camera 9 can typically capture a direction perpendicular to the imaging direction of the infrared camera 4 and can capture a direction perpendicular to the back surface. The outer camera 9 comprises a lens and an image sensor sensitive to visible light. The outer camera 9 captures a space in the back direction to obtain a color image (RGB image). It should be noted that a camera may be provided on the front in addition to the rear camera 9 on the rear, and the rear camera 9 on the rear is not provided, and the camera is provided on the front (the surface on which the screen of the display 2 is provided) It is also good.

  FIG. 4 is a block diagram showing an example of the internal configuration of the portable electronic device 1. As shown in FIG. 4, the portable electronic device 1 includes the vibrator 11, the microphone 12, the speaker 13, the control unit 14, the communication unit 15, the attitude detection unit 16, the GPS receiver 17, and the geomagnetic sensor 18 in addition to the above-described units. Prepare. In addition, the portable electronic device 1 includes a battery (not shown), and power is supplied from the battery. These components are housed in the housing 10.

  The control unit 14 includes the display 2, the touch panel 3, the infrared camera 4, the distance measurement sensor 5, the input button 6, the irradiation unit 7, the projector 8, the vibrator 11, the microphone 12, the speaker 13, the communication unit 15, the attitude detection unit 16, and GPS The receiver 17 and the geomagnetic sensor 18 are connected to each part to control each part.

  Specifically, the control unit 14 includes a CPU, a memory (RAM), and the like, and stores a predetermined program (for example, a non-volatile memory, a hard disk, etc.) (not shown) provided in the portable electronic device 1. For example, predetermined processing is performed based on game processing, image processing, and an application program for performing various calculations. For example, the control unit 14 acquires an image from the infrared camera 4 and analyzes the image, or calculates a distance to an object based on a signal from the distance measurement sensor 5, or from the touch panel 3 or the input button 6. Perform processing according to the input signal of Then, the control unit 14 generates an image based on the result of the predetermined processing, and outputs the image to the display 2. The program for performing the predetermined process may be downloaded from the outside via the communication unit 15.

  The vibrator 11 operates based on an instruction from the control unit 14 to vibrate the entire portable electronic device 1. The vibrator 11 is provided at a predetermined position where the vibration is easily transmitted to the user's hand (for example, a position biased to either the central portion in the housing 10 or to the left or right).

  The microphone 12 and the speaker 13 are used to input / output voice. The communication unit 15 is used to communicate with another device by a predetermined communication method (for example, a wireless LAN or the like). Further, the posture detection unit 16 is, for example, an acceleration sensor or an angular velocity sensor, and detects the posture of the portable electronic device 1.

  The GPS receiver 17 receives a signal from a GPS (Global Positioning System) satellite, and the mobile electronic device 1 can calculate the position of the mobile electronic device 1 by receiving the signal. For example, when a predetermined operation (for example, a gesture input or a button input using an infrared camera 4 described later, an operation such as shaking the mobile electronic device 1) is performed at a specific position, the mobile electronic device 1 An object associated with a specific position may be displayed. For example, when a game is played in the mobile electronic device 1, when the mobile electronic device 1 is in a specific position, an object corresponding to the specific position may be made to appear in the game.

  Further, the geomagnetic sensor 18 is a sensor capable of detecting the direction and magnitude of magnetism. For example, the portable electronic device 1 determines whether or not it is facing a specific direction based on the detection result of the geomagnetic sensor 18, and a predetermined operation (such as the above gesture input) is performed in the specific direction. , Objects may be displayed. For example, when a game is played in the portable electronic device 1, an object corresponding to a specific orientation may appear in the game. In addition, the portable electronic device 1 may combine GPS information using the GPS receiver 17 and azimuth information using a geomagnetic sensor. For example, when the portable electronic device 1 is at a specific position and faces a specific orientation, the object corresponding to the specific position and the specific orientation is displayed, or the object is made to appear in the game. It is also good.

  FIG. 5 is a view showing an example of the usage state of the portable electronic device 1, and a view showing a state where the user holds the portable electronic device 1 with both hands. As shown in FIG. 5, the user grips the left and right ends of the portable electronic device 1. When the user grips the left and right ends, the user can press the input button 6B with the thumb of the left hand, and can press the input button 6A with the thumb of the right hand. Also, when the user grips the left and right ends, the user can move the index finger or middle finger of the left hand and can press the input button 6D with the index finger or middle finger of the left hand, and move the index finger or middle finger of the right hand It is possible to press the input button 6C with the index finger or the middle finger of the right hand.

  Next, input to the portable electronic device 1 will be described. In the present embodiment, the user can perform a gesture input to the portable electronic device 1 using the right hand while holding the portable electronic device 1 with the left hand.

  6A and 6B are diagrams showing how the user performs a gesture input using the right hand in the direction of the right side surface of the portable electronic device 1. FIG. FIG. 7A is a view showing an example of an image captured by the infrared camera 4 when a gesture input as shown in FIG. 6A is performed. FIG. 7B is a diagram showing an example of an image captured by the infrared camera 4 when a gesture input as shown in FIG. 6B is performed.

  As shown in FIGS. 6A and 6B, the user performs various gesture inputs using the right hand at a position in the direction of the right side surface of the portable electronic device 1. The portable electronic device 1 captures an infrared image with the infrared camera 4 provided on the right side surface of the portable electronic device 1 and analyzes the captured infrared image to determine the gesture input of the user.

  Specifically, when the image shown in FIG. 7A or 7B is acquired from the infrared camera 4, the portable electronic device 1 sets a specific object (specifically, a human hand) included in the acquired image. Detection is performed, and the type of gesture performed by the user is determined based on the shape or the like of the specific object. For example, the portable electronic device 1 determines whether an object having a predetermined shape exists in the acquired image by pattern matching or the like. For example, when the image shown in FIG. 7A is obtained, the portable electronic device 1 can recognize “Goo” from the number of fingers, and when the image shown in FIG. 7B is obtained, "Per" can be recognized. And the portable electronic device 1 performs the process according to the kind of gesture so that it may mention later.

  Here, the gesture to be determined is a gesture or gesture using a part or the whole of the body such as the user's hand or face, and the portable electronic device 1 recognizes a stationary state of the hand or the like as a gesture input. You may also recognize a series of actions using a hand as gesture input. The portable electronic device 1 may also recognize a gesture input performed in a state where the user holds an object. In this case, the portable electronic device 1 may recognize the stationary state or the operating state of only the object held by the user as a gesture input, or gesture the entire stationary state or the operating state including the user's hand and the object. It may be recognized as input.

  When a plurality of objects are included in the infrared image captured by the infrared camera 4, the portable electronic device 1 detects only a specific object of the plurality of objects, and processes based on the detected objects. You may For example, when imaging a user's hand using the infrared camera 4, the hand and an object behind it may be included in the infrared image. In this case, the portable electronic device 1 may recognize only the user's hand. For example, the portable electronic device 1 extracts only an image of an object having the highest average luminance among images of a plurality of objects included in the infrared image captured by the infrared camera 4 and analyzes the image of the object. The user's hand may be recognized. Alternatively, only the user's hand may be recognized from images of a plurality of objects included in the infrared image captured by the infrared camera 4 based on pattern matching or the like.

  In the present embodiment, the user performs various operations on the portable electronic device 1 by performing a gesture using the right hand in, for example, the right side direction of the portable electronic device 1. First, the user needs to recognize whether or not the right hand of the user's right hand is normally imaged by the infrared camera 4 disposed on the right side surface of the portable electronic device 1 and that gesture input is possible. Thus, in the present embodiment, guide display (for guiding the user's hand to a position suitable for gesture input) for causing the user to recognize the position where gesture input can be performed is performed. For example, before the application described later is executed, guide display is performed.

(Guide display)
FIG. 8 is a view showing an example of the guide display of the present embodiment. FIG. 8 shows an example of guide display in the case where the user's right hand is within the imaging range of the infrared camera 4 and the user's right hand is imaged by the infrared camera 4 from the front of the infrared camera.

  As shown in FIG. 8, on the right side of the screen of the display 2, an outline image 21 and a guide frame image 22 are displayed as a guide display for causing the user to recognize the correct gesture input position. The contour image 21 is an image showing the contour of the image of the user's hand included in the infrared image captured by the infrared camera 4. The guide frame image 22 is an image for causing the user to recognize the correct gesture input position, and is an image showing a narrower range than the imaging range imaged by the infrared camera 4.

  As shown in FIG. 8, for example, the right direction (long side direction) of the portable electronic device 1 is defined as an X axis, the upper direction (short side direction) as a Y axis, and the back direction of the screen of the display 2 as a Z axis. If a straight line extending from the infrared camera 4 in the X-axis direction is approximately at the center of the palm of the user's right hand, and the user's right hand is at an appropriate position in the X-axis direction, the infrared camera 4 captures all of the user's right Be done.

  FIG. 9 is a view showing an example of an infrared image captured by the infrared camera 4 in the state shown in FIG. FIG. 10 is a view showing an example of an image after the contour drawing process and the guide frame image drawing process when the image shown in FIG. 9 is acquired.

  As shown in FIG. 9, the infrared image 40 captured by the infrared camera 4 includes an image of the user's right hand. A contour drawing process and a guide frame image drawing process are performed on the captured infrared image 40 to generate a composite image 41 shown in FIG. Specifically, as shown in FIG. 10, when the infrared image 40 is captured by the infrared camera 4, the portable electronic device 1 detects the outline of the right hand included in the infrared image 40 by edge detection, for example, and detects an outline other than the outline. The contour image 21 is generated by erasing the pixel information in the inside. The portable electronic device 1 also draws the guide frame image 22 so as to overlap the generated outline image 21. Specifically, the guide frame image 22 is an image that is fixed at the image center of the infrared image 40 captured by the infrared camera 4 and indicates a rectangular frame having a certain size.

  A composite image 41 obtained by combining the contour image 21 and the guide frame image 22 is displayed on the display 2 while being tilted in a predetermined direction. Specifically, as shown in FIG. 8, the portable electronic device 1 parses the composite image 41 and displays the composite image (image obtained by viewing the composite image 41 obliquely) on the display 2. Note that a specific example of the process of parsing will be described later.

  Here, when the user moves the right hand from the state shown in FIG. 8, the image displayed on the display 2 changes. FIG. 11 is a diagram showing a state in which the user has moved his hand upward (Y-axis direction) from the state shown in FIG. FIG. 12 is a diagram showing a state in which the user has moved his hand in the back direction (Z-axis direction) of the screen from the state shown in FIG.

  As shown in FIG. 11, when the center of the user's hand is above the center of the imaging range of the infrared camera 4, an image in which a part of the outline image 21 protrudes above the guide frame image 22 is displayed on the display 2. Is displayed on. Further, as shown in FIG. 12, when the center of the user's hand is on the left side (the Z axis direction side in FIG. 12) with respect to the center of the imaging range of the infrared camera 4, the outline image on the left of the guide frame image 22 An image in which a part of the image 21 is projected is displayed on the display 2. Here, as shown in FIG. 11 and FIG. 12, the portion of the contour image 21 that is out of the guide frame image 22 is displayed in a display mode different from the portion that fits in the guide frame image 22. For example, a portion of the outline image 21 that protrudes from the guide frame image 22 is different in color or displayed as a thick line.

  FIG. 13 is a view showing a composite image 41 obtained by combining the outline image 21 and the guide frame image 22 in the state shown in FIG. FIG. 14 is a view showing a synthesized image 41 obtained by synthesizing the outline image 21 and the guide frame image 22 in the state shown in FIG.

  As shown in FIG. 13, when the user's hand exists above the center of the infrared image (the center of the imaging range of the infrared camera 4), a part of the user's hand protrudes above the guide frame image 22 . Further, as shown in FIG. 14, when the user's hand exists on the left side of the center of the infrared image, a part of the outline of the user's hand protrudes to the left of the guide frame image 22. These portions that have run out are displayed in a display mode (eg, red) different from other portions (eg, black) that do not run out. After such a composite image 41 is generated, these images are parsed and displayed on the display 2 (the composite image 41 is displayed in perspective as shown in FIGS. 11 and 12).

  In this manner, the user can easily recognize the position in the depth direction by displaying the image subjected to the perspective (image obtained by viewing the composite image 41 obliquely), and the user's hand is in the imaging range of the infrared camera 4. It can be easily recognized whether it is in or out. That is, since the image with the perspective is the same as when the user directly looks at his hand, it is easy for the user to recognize the depth direction, and at which position in the depth direction the hand is currently located Can be easily recognized.

  In addition, since the guide frame image 22 fixed in the imaging range of the infrared camera 4 is displayed, the user can easily adjust the position of the hand so that the hand is in the guide frame image 22. In addition, since the part protruding from the guide frame image 22 is displayed in a mode different from the part in the guide frame image 22, the user can easily recognize that the hand is out of the guide frame image 22. , Easy to adjust the position of the hand.

  The guide frame image 22 is smaller than the infrared image captured by the infrared camera 4. Therefore, even if the user's hand extends beyond the guide frame image 22, when the infrared image captured by the infrared camera 4 includes the user's hand (finger), the portable electronic device 1 performs the gesture performed by the user. It can be recognized. By displaying the guide frame image 22, it is possible to warn the user that the user's hand will come out of the imaging range of the infrared camera 4 before it can not recognize the gesture performed by the user.

  Further, in the present embodiment, the right side surface direction of the portable electronic device 1 is imaged by the infrared camera 4, and the imaged image is displayed on the right side of the screen of the display 2. That is, the side surface direction of the portable electronic device 1 is imaged by the infrared camera 4, and an image is displayed on the same side of the screen of the display 2 as the side surface direction imaged by the infrared camera 4. Therefore, the user can easily recognize the position of his / her hand intuitively.

  Thus, in the present embodiment, guide display (the outline image 21 and the guide frame image 22) is performed. This allows the user to recognize the correct gesture input position. That is, by performing the guide display as described above, it is possible to make the user recognize whether or not the hand is at an appropriate position.

  Here, an example of processing for performing perspective display as illustrated in FIGS. 8, 11 and 12 will be described.

  FIG. 15 is a schematic diagram showing an example of processing for performing perspective display. As shown in FIG. 15, the contour image 21 and the guide frame image 22 are arranged at predetermined positions in the virtual space. For example, each point forming the contour image 21 and each point forming the guide frame image 22 are plotted on the yz plane of the virtual space defined by the x axis, the y axis, and the z axis. Each of these points is translated in the x-axis direction to form a contour image 21 'and a guide frame image 22'. By projection-transforming the moved points (the outline image 21 ′ and the guide frame image 22 ′), the display 2 displays the image subjected to the perspective. Specifically, the virtual camera 50 is disposed on the yz plane, the imaging direction (cz axis direction) of the virtual camera 50 is set parallel to the z axis, and the upper direction (cy axis direction) of the virtual camera 50 is the y axis Set parallel to An image obtained by capturing the contour image 21 ′ and the guide frame image 22 ′ with such a virtual camera 50 is displayed on the display 2.

  Note that the process for perspective display is not limited to the process shown in FIG. For example, the composite image 41 obtained by combining the contour image 21 and the guide frame image 22 is rotated by a predetermined angle around the vertical axis (Y axis in FIG. 8), and the rotated image is displayed on the display 2 An image in which a perspective is applied to the composite image 41 may be displayed.

  Further, in the above embodiment, the contour image 21 indicating the contour of the hand image included in the infrared image captured by the infrared camera 4 is displayed on the display 2, but the infrared image captured by the infrared camera 4 instead of the contour May be displayed on the display 2 as it is. Further, the infrared image picked up by the infrared camera 4 may be abstracted and displayed on the display 2 in perspective, instead of being displayed on the display 2 as it is. The infrared image captured by the infrared camera 4 is not limited to displaying the outline of the hand as described above, for example, by displaying a bone (a line segment connecting the feature points) connecting the captured feature points of the hand May be abstracted and displayed, or the infrared image may be abstracted by another method.

  Further, in the above embodiment, the user is informed of the fact that the hand has protruded from the guide frame by displaying the portion of the image of the hand that has protruded from the guide frame image 22 in a specific mode different from other portions. The In another embodiment, when the user's hand extends out of the guide frame, the user may be warned by displaying the entire hand image (or the extended portion) in a specific manner. Also, when the user's hand protrudes from the guide frame image 22, a warning is given to the user by displaying the guide frame image 22 in a specific mode different from the normal, in addition to or instead of the hand image. It is also good. Here, the specific aspect is not limited to the one in which the color is different from the normal or the thickness of the line is changed as described above. For example, it may be made different from normal by blinking display. In addition, when the user's hand protrudes from the guide frame, light using another image, characters, sound, vibration (vibration with respect to the hand holding the portable electronic device 1 (left hand), projector 8 (which may be an LED lamp) The user may be warned, etc.

  Further, in the above embodiment, the infrared image captured by the infrared camera 4 is displayed on the display 2 after being scanned and displayed on the display 2. However, in another embodiment, the captured image (or an image obtained by abstracting the captured image) You may display without parsing it. That is, as long as the user's relative position of the hand with respect to the portable electronic device 1 can be displayed in a manner that can be recognized, it may be displayed in any manner. For example, the range image indicating the imaging range imaged by the infrared camera 4 is displayed on the display, and the image indicating the position of the user's hand is superimposed (or not superimposed) on the range image to display the imaging range. It may also indicate at which position the user's hand is located. For example, a range image indicating the position in the depth direction is displayed so that the user can easily recognize the position in the depth direction, and an image indicating the position of the hand is displayed by overlapping (or not overlapping) the range image. You may Specifically, for example, from the infrared image captured by the infrared image 4, a predetermined position (for example, the center of gravity) of the user's hand in the captured image is calculated. Then, a range image indicating the range in the left-right direction (Z-axis direction in FIG. 9) of the infrared image is displayed, and an image indicating the position of the hand is displayed at the calculated position. This allows the user to recognize where the user's hand is located in the depth direction (Z-axis direction).

  In addition, although the guide display described above is performed before the execution of the application described later, the guide display may or may not be performed depending on the execution status of the information processing performed by the portable electronic device 1. It is also good. For example, guide display may be always performed during execution of an application described later, or display / non-display of the guide display may be switched by an instruction from the user. In addition, when an instruction to execute a program for performing guide display is issued by the user, the guide display may be performed. Further, the above-described guide display may be performed only at the time of setting of the portable electronic device 1 or at the time of a tutorial for teaching the user how to use the portable electronic device 1.

  Further, in the above-described embodiment, when the user's hand protrudes from the guide frame image 22 as a pre-warning until the user's hand can not be recognized, the part that has protruded is displayed in a specific manner. Specifically, even if it is possible to recognize the user's hand from the infrared image captured by the infrared camera 4, if the user's hand is out of the guide frame, the guide frame image 22 as described above. The user is warned in advance by displaying the part that has come off from the display in a specific display mode. In this case, when the user's hand can not be recognized from the infrared image actually taken by the infrared camera 4, another warning may be issued. For example, a second warning different from the above-described prior warning using the guide frame image 22 may be performed using voice, characters, images, vibrations, light, and the like. In addition, in the case where a warning is given in advance using the above-mentioned guide display while an application as described later is being executed, the second hand can not be recognized when the user's hand can not be recognized from the infrared image captured by the infrared camera 4. After giving a warning (or not giving a second warning), the application may be terminated or the application may be suspended as described later.

  Further, in the above embodiment, even when the user's hand is detected, when at least a part of the user's hand is out of the guide frame, the user is warned. In other embodiments, the conditions for giving a warning are not limited to this. For example, in another embodiment, it is determined based on an image captured by the infrared camera 4 whether or not a predetermined condition in which the user's gesture can not be recognized is satisfied, and the predetermined condition is satisfied. You may give a warning. For example, the predetermined condition may be that the image captured by the infrared camera 4 is too dark (or too bright), that the level of noise included in the image is a predetermined level or more, or the like. For example, if the average brightness of the entire image is less than a predetermined value, the user's gesture can not be determined if the brightness is lower than that, so a warning may be issued in such a case.

  Further, the shape and size of the guide frame are merely an example, and a guide frame of any other shape and size may be set and displayed as a guide frame image. Moreover, not only a frame image but other arbitrary images may be displayed. That is, any image may be displayed as long as the user can recognize whether an object (for example, the user's hand) included in the imaging range of the infrared camera 4 is at an appropriate position. For example, when the hand is at the proper position, a predetermined image is displayed at the edge of the screen, and when the hand is not at the proper position (when the hand is within the imaging range and out of the proper range) A predetermined image may be displayed blinking.

(Application control)
Next, control of an application based on an infrared image captured by the infrared camera 4 in the portable electronic device 1 will be described. FIG. 16 is a view showing an example of a menu screen displayed on the display 2 of the portable electronic device 1. FIG. 17 is a diagram showing how the application B selected by the menu screen is activated.

  As shown in FIG. 16, a list of applications that can be executed on the portable electronic device 1 is displayed on the menu screen. For example, the mobile electronic device 1 transmits and receives a game application for performing a predetermined game, an application for playing music and video, an application for calling, an application for capturing and displaying an image, and an e-mail, a message, etc. It may be possible to execute any application such as an application for browsing, a browser application for browsing web pages on the Internet, and an application for creating or editing a document. Moreover, the program for setting the portable electronic device 1 may be executable from the menu screen.

  The user selects an icon corresponding to the desired application from among the icons indicating the plurality of applications displayed on the menu screen. For example, the user may select an application using the input button, or may select an application by performing a touch operation on the touch panel 3 using a finger or a touch pen. Further, the user may be able to select a desired application from a plurality of applications displayed on the menu screen by making a gesture on the right side surface of the portable electronic device 1.

  For example, when the user brings the hand close to the right side in a state where the application B is selected as shown in FIG. 16, the application B is activated as shown in FIG. Specifically, the portable electronic device 1 analyzes an infrared image captured by the infrared camera 4 to detect an object (for example, a user's hand). When an object is detected, the portable electronic device 1 activates the selected application B. Specifically, the CPU of the control unit 14 of the portable electronic device 1 reads the program of the application B from the storage device (for example, non-volatile memory etc.) inside the portable electronic device 1 into the RAM and starts the execution of the program . Alternatively, the program of the application B may be stored outside the mobile electronic device 1, and the mobile electronic device 1 may acquire the program via the communication unit 15 and read the program into the RAM.

  In addition, as conditions for starting the selected application, not only that the portable electronic device 1 detected the object but that the portable electronic device 1 detected the user's hand, the portable electronic device 1 specified the user That the right side surface of the portable electronic device 1 is held by the user, that the portable electronic device 1 detects a specific object, and the distance between the portable electronic device 1 and the specific object is a predetermined value It may be that the object is moving at a predetermined speed or more.

  For example, by analyzing the infrared image captured by the infrared camera 4, the portable electronic device 1 can detect that the user grips the portable electronic device 1 or the distance between the portable electronic device 1 and the object. In addition, the portable electronic device 1 determines the relative distance between the portable electronic device 1 and the specific object based on the size of the specific object included in the infrared image acquired from the infrared camera 4. Good. For example, the portable electronic device 1 may determine that the user is held by the user based on the luminance information of the infrared image captured by the infrared camera 4, or the distance between the portable electronic device 1 and the object may be calculated. Good. As the distance between the object and the portable electronic device 1 decreases, the luminance of each pixel of the infrared image increases. For this reason, the portable electronic device 1 can determine that the user grips the region, for example, in the case where the region having the luminance equal to or more than the predetermined value in the captured infrared image is equal to or more than the predetermined ratio.

  Further, the portable electronic device 1 can calculate the relative distance to the object based on the luminance of the object included in the infrared image, and calculates the velocity of the object based on the change of the relative distance be able to. The portable electronic device 1 may activate (start) the application based on the distance and speed calculated in this manner.

  The activated application B is started when a predetermined condition is established. For example, when the application B is a game application, the game is started on condition that the user has given a predetermined instruction (instruction using a button, a touch panel, a gesture, or the like) to start the game. Alternatively, the game may be started on condition that the portable electronic device 1 detects the user's hand. When the application B is a game using a gesture, the game can not be performed unless the user's hand is detected, and thus the detection of the user's hand may be the condition for starting the game. Alternatively, the game may be started on condition that a predetermined time has elapsed since the start of the application B.

  FIG. 18 is a diagram showing an example of the screen immediately after the game of the started application B is started. As shown in FIG. 18, when the game is started, the display 2 shows a display indicating the remaining time until the game is over and a display showing the points acquired during the game. The application B is, for example, a game aiming to obtain more points within a predetermined time, may be a game played by the user alone, or may be another portable electronic device 1 (or other device using the communication unit 15). The game may be a game in which a large number of people cooperate with each other to obtain points by interactively communicating with the device), or a game in which the numbers are competed by a large number of people. For example, the application B is a game in which the score is added when the user performs a gesture as instructed on the display 2. In such a game, when the game is started after the application B is started, an image shown in FIG. 18 is displayed.

  FIG. 19 is a diagram showing an example of screen transition while the game is being executed. As shown in FIG. 19A, when 30 seconds have elapsed from the start of the game, the display indicating the remaining time is “30” and the score is “75”. In addition, an instruction for the user is displayed on the screen, and the user performs a gesture according to the instruction. As shown in (B), when 2 seconds have passed and 32 seconds have passed since the game started, the display indicating the remaining time is “28” and the score is also added by the user performing a gesture according to the instruction. Ru. As described above, in this game, for example, instructions are displayed one after another according to a predetermined music rhythm, and when a gesture corresponding to the displayed instruction is performed, points are added.

  Here, as shown in FIG. 19C, when the user's hand is out of the imaging range of the infrared camera 4 and the infrared camera 4 can not detect the user's hand, the game of the application B is temporarily stopped. And return to the menu screen. For example, when the predetermined standby time elapses after the user's hand is not detected, the game screen of application B is switched to the menu screen. At this time, the game is paused internally, and the progress of the game is temporarily stopped. However, the game being executed is not ended, and data relating to the execution of the game remains in the RAM. For example, a value (for example, remaining time or score) indicating the progress of the game is stored in the RAM, and during the game, the value indicating the progress is in accordance with the passage of time (or the user's operation). To increase. When the progress of the game is stopped, the increase (or decrease) of the value indicating the progress is also stopped. In this menu screen, the user can also select another desired application.

  Then, as shown in (D) of FIG. 19, when the user's hand again enters the imaging range of the infrared camera 4 in a state where the application B is selected, the portable electronic device 1 detects the user's hand. Resume the game. Specifically, the game of application B starts again from the state of being paused. For example, when the game is paused when the remaining time is 28 seconds and the score is 80 points, the game is resumed from the state where the score is 80 points with the remaining time 27 seconds.

  More specifically, the portable electronic device 1 resumes the game of the application B immediately after detecting the user's hand (or after a period shorter than the standby time has elapsed). The portable electronic device 1 waits for a predetermined time when temporarily stopping the application, but when restarting the application, the user's intention is achieved by immediately resuming the application without waiting for the predetermined time. You can control app pause / resume. That is, in an application that performs a gesture using a hand, the hand may be momentarily detached from the imaging range of the infrared camera 4 without the user's intention. In such a case, pausing the running application immediately suspends the application temporarily against the user's intention. However, since the waiting time is set when pausing the application, it is possible to prevent the suspension of the application that the user does not intend. On the other hand, when resuming the application from the menu screen, the user intentionally puts his hand in the imaging range of the infrared camera 4. For this reason, the portable electronic device 1 resumes the application immediately without providing the above-mentioned waiting time when restarting the application.

  The portable electronic device 1 may resume the application after a predetermined waiting time has elapsed since the user's hand was detected, as in the case of temporarily stopping the application. Further, the portable electronic device 1 may suspend / resume the application immediately without waiting for a predetermined time, even when the application is temporarily suspended or restarted. When the application is suspended as shown in FIG. 19C and the application is not resumed even after a predetermined time has elapsed, the portable electronic device 1 may end the application. In this case, even if the user puts his hand in the imaging range of the infrared camera 4 again, the application is started from the beginning.

  As described above, in the portable electronic device 1 of the present embodiment, an interactive application in which the progress is controlled in accordance with the user's input is executed. Specifically, the application progresses in response to a gesture input performed in the side direction of the portable electronic device 1. For example, as an example of an interactive application, success or failure determination with respect to a gesture input is performed, and a game is performed in which points are added according to the result of the success or failure determination. The application is started and started by the user holding a hand in the side direction of the portable electronic device 1 or performing a gesture using the hand. In addition, when the user puts his hand in the side direction of the portable electronic device 1, the application being executed is paused and a menu screen is displayed. The menu screen allows the user to select another desired application. When the user holds the hand in the side direction of the portable electronic device 1 again with the suspended application selected, the suspended application is resumed.

  In the example described above, the menu screen is displayed when the user moves his hand while executing the application, that is, when the user's hand is not detected in the infrared image captured by the infrared camera 4. In another example, for example, the menu screen may be displayed when the user brings the right hand close to (or away from) the right side surface of the portable electronic device 1 at a predetermined speed. In addition, when the user makes a predetermined gesture, a menu screen may be displayed. For example, the menu screen may be displayed when the application being executed is paused when the user makes a specific gesture and the hand is moved in that state.

  FIG. 20 is a diagram showing an example of a gesture for displaying a menu screen according to another embodiment. For example, as shown in FIG. 20, when the user makes a gesture of grasping an object in the direction of the right side of the portable electronic device 1, the application being executed is suspended. In this state, when the user moves the hand in the right direction, a part of the menu screen appears to be pulled from the left end of the display 2. Then, when the user further moves the hand in the right direction, the entire screen of the display 2 is switched to the menu screen.

  Also, in the above, when the user places the hand over the infrared camera 4, the application selected on the menu screen is started (restarted). In other embodiments, the selected application may be launched, for example, when the user makes a predetermined gesture or when the hand is moved at a predetermined speed, to prevent malfunction. In addition, when the user grips the right side surface of the portable electronic device 1, the selected application may be started. In this case, whether or not the user holds the right side surface of the portable electronic device 1 may be determined based on, for example, luminance information of an image captured by the infrared camera 4 or based on a change in luminance. It may be done. In addition, the portable electronic device 1 may start (resume) an application when it detects a pulse of the user. For example, the pulse of the user can be detected based on the change in the brightness of the image captured by the infrared camera 4. Malfunction can be prevented by starting the application when the user's pulse is detected.

  In addition, the conditions regarding detection of an object may be different between the activation of the application and the start of predetermined processing in the application. In addition, the condition regarding detection of an object may be different between the suspension of the application and the termination of the application. For example, the application is activated when the first condition for detecting an object is satisfied, and starts a predetermined process in the application when a second condition stricter than the first condition is satisfied. May be In this case, if the first condition is satisfied and the second condition is not satisfied, predetermined processing in the application is not started even if the application is started. In addition, for example, the application may be suspended when the first condition is satisfied, and the application may be terminated when the second condition stricter than the first condition is satisfied. In this case, if the first condition is satisfied and the second condition is not satisfied, the application is not terminated even if the application is suspended.

  For example, the application B is activated when the mobile electronic device 1 detects an object, and the start screen (initial screen) of the application B is displayed as shown in FIG. The start screen is a screen before the predetermined process (game process) in the application B is started. Then, after the start-up screen is displayed, for example, when the portable electronic device 1 detects a user's hand, game processing is started. Specifically, when an object (some object regardless of whether it is a hand or not) is detected in the infrared image captured by the infrared camera 4, the start screen of the application B is displayed, and the portable electronic device is displayed in that state When the device 1 detects the user's hand, the game starts. That is, the condition for the start-up screen of application B to rise up is that the user's hand does not have to be detected, but some object is detected, and the condition for the game processing in application B to be started , The user's hand is to be detected. This is because even if the user's hand falls within the imaging range of the infrared camera 4, if the hand and the infrared camera 4 are too close to each other, the portable electronic device 1 can recognize that there is any object, I can not recognize if it is a hand or not. In such a case, if the start screen of the application B does not rise and the menu screen remains, it is difficult for the user to understand what operation the application B will start. Therefore, even if the portable electronic device 1 can not recognize the user's hand (even if the second condition is not satisfied), it detects the first object (if the first condition is satisfied). If the mobile electronic device 1 launches the launch screen of the application B). However, since the game is a game using a gesture, the portable electronic device 1 needs to recognize the user's hand in order to perform actual game processing. Therefore, the recognition of the user's hand is regarded as the condition for starting the game. As described above, the application is started if the first condition regarding detection of an object is satisfied, and the second condition stricter than the first condition is satisfied after the application is started, and the predetermined process (game Processing) may be started.

  Even when the application is paused and terminated, the application is paused if the first condition for detecting an object is satisfied, and the second condition is stricter than the first condition after the application is paused. If satisfied, you may exit the app.

  Also, the application to be executed is not limited to the game described above. For example, a predetermined game object may be displayed, and a game in which the user's hand controls the predetermined game object may be played. For example, the portable electronic device 1 may control the movement of the game object based on the position of the image captured by the infrared camera 4 (for example, the position of the center of gravity of the area having a predetermined luminance or more).

  Further, in the above embodiment, when the portable electronic device 1 does not detect the user's hand during the execution of the application, the application is paused to display the menu screen. In another embodiment, when the portable electronic device 1 does not detect the user's hand during execution of the application, the application may be paused, and a screen as to whether or not to end the application may be displayed. The application may be terminated when the user instructs the termination of the application on this screen. In addition, even if the application is suspended and there is no instruction from the user, the application may be terminated when a predetermined time has elapsed.

(Details of guide display process)
Next, an example of specific processing performed in the portable electronic device 1 will be described. First, with reference to FIG. 21, the details of the process for performing the above-described guide display will be described. FIG. 21 is a flowchart showing an example of processing for performing guide display. The control unit 14 (of the CPU) reads a program stored in the storage device into the RAM at a predetermined timing (for example, a timing before a user's instruction or an application is started), and executes the program. Perform the process shown in. The process shown in FIG. 21 is repeatedly performed at a predetermined time interval (for example, once in 1/60 seconds).

  As shown in FIG. 21, in step S <b> 10, the control unit 14 acquires an infrared image from the infrared camera 4. Next, the control unit 14 performs a hand detection process of detecting the user's hand by analyzing the infrared image acquired in step S10 (step S11). As a result of the hand detection process, when the hand can be detected (step S12: YES), the control unit 14 sets a fixed guide frame to the acquired infrared image (step S13). Specifically, the control unit 14 sets a guide frame of a certain size that covers the central region of the acquired infrared image.

  Subsequently, the control unit 14 generates an outline image indicating the outline of the user's hand included in the acquired infrared image (step S14). Specifically, when a plurality of objects exist in the acquired infrared image, the control unit 14 extracts only the object with the highest luminance, and detects the contour of the extracted object. Then, the control unit 14 erases pixel information other than the contour while leaving the detected contour. Next, the control unit 14 sets the pixel information of the portion of the generated contour image that is out of the guide frame set in step S13 to red (step S15). Subsequently, the control unit 14 combines the outline image with the guide frame image indicating the guide frame set in step S13 (step S16). Then, the control unit 14 generates an image obtained by obliquely viewing the combined image, and displays the image on the display 2 (step S17). Specifically, the control unit 14 performs the processing for perspective display described with reference to FIG. 15, for example, to display the image obtained by rotating the composite image by a predetermined angle around the Y axis in FIG. 10. Display at the right end of

  On the other hand, when the hand can not be detected as a result of the hand detection process of step S11 (step S12: NO), the control unit 14 displays an error on the display 2 (step S18). For example, the control unit 14 may display a character indicating that a hand has not been detected on the display 2 and notify the user that the hand has not been detected by voice, vibration or the like.

  After performing the process of step S17 or step S18, the control unit 14 determines whether to end the process illustrated in FIG. 21 (step S19). For example, the process illustrated in FIG. 21 may end in response to a user's instruction, or may end in response to the passage of a predetermined time. If it is determined that the control unit 14 does not end the process shown in FIG. 21 (step S19: NO), the process of step S10 is executed again, and if it is determined that the process shown in FIG. 21 is ended (step S19: YES). And end the processing shown in FIG.

(Details of menu display process)
Next, with reference to FIG. 22, the details of the process for displaying the menu screen described above will be described. FIG. 22 is a flowchart illustrating an example of processing for displaying a menu screen. The process shown in FIG. 22 is performed by the control unit 14 (CPU thereof) executing a program for displaying a menu. Further, the process shown in FIG. 22 is repeatedly performed at a predetermined time interval (for example, once in 1/60 seconds).

  As shown in FIG. 22, in step S <b> 20, the control unit 14 displays a menu screen on the display 2. For example, the control unit 14 acquires a list of currently executable applications in the portable electronic device 1, and arranges and displays icons indicating each application on a screen. An application executable in the portable electronic device 1 is, for example, an application stored in a storage device inside the portable electronic device 1 or an application stored in an external storage device accessible from the portable electronic device 1 It is also good.

  Following step S20, the control unit 14 performs application selection processing (step S21). For example, based on the position detected by the touch panel 3, the control unit 14 determines whether the icon corresponding to the application is selected (touched) by the user. If it is determined that the icon has been selected by the user, the control unit 14 selects an application corresponding to the selected icon.

  Next, the control unit 14 acquires an infrared image from the infrared camera 4 (step S22). Next, the control unit 14 performs a hand detection process of detecting the user's hand by analyzing the infrared image acquired in step S22 (step S23). As a result of the hand detection process, when the hand can be detected (step S24: YES), the control unit 14 determines whether the application has been selected (step S25). Specifically, the control unit 14 determines whether or not the application has been selected by the user in step S21. If an application has been selected (step S25: YES), the control unit 14 determines whether the selected application has already been activated (step S26). Specifically, when the application has already been started, the information on the application (the program itself and other data) is stored in the RAM, so the control unit 14 determines based on the information stored in the RAM. , It is determined whether the application has already been started.

  On the other hand, when the hand can not be detected as a result of the hand detection process (step S24: NO), or when the application has not been selected (step S25: NO), the control unit 14 performs the process of step S20. Do it again.

  When the application has not been started (step S26: NO), the control unit 14 starts the application selected in step S21. If the application has already been started (step S26: YES), the control unit 14 resumes the execution of the selected application (step S28).

  After the process of step S27 or step S28, the control unit 14 ends the process illustrated in FIG.

(Details of application control process)
Next, the details of the process for controlling the application described above will be described with reference to FIG. FIG. 23 is a flowchart illustrating an example of the application control process. The process shown in FIG. 23 is performed, for example, when the user holds his hand over the right side surface of the portable electronic device 1 as described above. Further, the process shown in FIG. 23 is repeatedly performed at a predetermined time interval (for example, once in 1/60 seconds).

  As shown in FIG. 23, in step S30, the control unit 14 starts (or restarts) an application. Specifically, when the application is not activated (when the application program is not read into the RAM, or when the initial setting for starting the application is not performed), the control unit 14 activates the application. After that, the application is started in response to establishment of a predetermined condition (a predetermined time has elapsed or a user instruction has been made). In addition, when the application is paused (that is, when the application is already started and then paused and the menu screen is displayed), the control unit 14 resumes the application. Specifically, the control unit 14 reads data for the application stored in the RAM, and resumes the application from a state in which the application is suspended in step S38 described later.

  Following step S30, the control unit 14 acquires an infrared image from the infrared camera 4 (step S31). Next, the control unit 14 analyzes the infrared image acquired in step S31 and performs a hand detection process of detecting the user's hand (step S32). As a result of the hand detection process, when the hand can be detected (step S33: YES), the control unit 14 performs a gesture determination process for determining a gesture input performed by the user (step S34). Specifically, the control unit 14 determines which of a plurality of predetermined gestures has been performed, based on the detected hand shape and movement.

  Subsequently, the control unit 14 performs an application process according to the determined gesture (step S35). For example, when the game application shown in FIG. 19 is performed, the control unit 14 causes the display 2 to display characters, images, characters, etc. indicating an instruction for the user, and determines whether a gesture according to the instruction has been performed. judge. Then, the control unit 14 displays the determination result on the display 2. For example, when it is determined that the gesture according to the instruction has been performed, the control unit 14 adds points and displays the added points on the display 2. Further, the control unit 14 measures an elapsed time from the start of the application and displays the time on the display 2. In addition to (or instead of) the gesture input performed by the user, the application processing in step S35 may be performed based on, for example, an operation on the input button or the posture of the portable electronic device 1.

  Following step S35, the control unit 14 determines whether to end the application (step S36). For example, when the remaining time indicating the time until the end of the application becomes “0”, the control unit 14 determines to end the application. Further, for example, when an operation for ending the application (an operation by a button operation or a gesture) is performed by the user, the control unit 14 determines to end the application.

  If it is not determined to end the application (step S36: NO), the control unit 14 executes the process of step S31 again. On the other hand, when it is determined to end the application (step S36: YES), the control unit 14 performs an application end process (step S40), and ends the process shown in FIG. For example, the control unit 14 releases the area of the RAM secured for the execution of the application program as the application termination process. As a result, a value (for example, the above-mentioned remaining time or score) according to the progress status of the application is cleared.

  On the other hand, if the user's hand can not be detected in the hand detection process in step S32 (step S33: NO), whether the control unit 14 has detected the user's hand and then a predetermined standby time has elapsed It is determined whether or not it is (step S37). If the predetermined standby time has not elapsed since the user's hand was not detected (step S37: NO), the control unit 14 executes the process of step S31 again. Note that the predetermined waiting time may be different depending on the type of application, may be a fixed time, or may be changed according to the detection condition of the hand.

  If the predetermined waiting time has elapsed since the user's hand was not detected (step S37: YES), the control unit 14 suspends the application being executed (step S38). Specifically, the application is suspended while maintaining the RAM area reserved for the execution of the application. For example, when the image of the application is displayed on the display 2, the control unit 14 stores the image in the RAM. In addition, the control unit 14 maintains a value (for example, remaining time, score, and the like) according to the progress status of the application.

  When the application is temporarily stopped in step S38, the control unit 14 calls a program for performing the above-described menu processing (step S39), and ends the processing illustrated in FIG. Thus, the menu processing shown in FIG. 22 is performed.

  The process shown in the flowchart is merely an example, and the order and contents of the process may be appropriately modified. Also, the application described above is merely an example, and any other application may be performed.

(Modification)
The embodiment described above is merely an example, and the following modifications may be added, for example.

  For example, the infrared camera 4 is not limited to the right side surface of the portable electronic device 1, but may be provided on any of the left side surface (T2 surface in FIG. 1), the upper side surface (T4 surface), and the lower side surface (T3 surface) Gesture input may be performed by the user in the side direction of. Further, the infrared camera 4 may be provided on a plurality of side surfaces. For example, when the infrared camera 4 is provided on the left side surface of the portable electronic device 1, the above-described guide display (the outline image 21 and the guide frame image 22) is displayed on the left side of the screen of the display 2. Further, the display position of the guide display may be different according to the attitude of the mobile electronic device 1. The portable electronic device 1 can detect the attitude of the mobile electronic device 1 by the attitude detection unit 16, and can detect, for example, how many times it has been rotated about an axis perpendicular to the screen by using an angular velocity sensor or an acceleration sensor. For example, as shown in FIG. 6A, normally, the infrared camera 4 is held to be positioned on the right side, but when the user holds the portable electronic device 1 so that the left and right are reversed, the infrared camera 4 is positioned on the left side You will come to In this case, the portable electronic device 1 is held by the right hand, and gesture input is performed by the left hand. In such a case, the guide display is displayed on the left side of the display 2. Specifically, a guide display with a perspective is displayed on the left side of the display 2 in the same manner as when the user directly looks at the left hand. That is, the guide display is displayed on the same side of the display screen of the display 2 as the side direction taken by the infrared camera 4.

  Moreover, in the said embodiment, it assumed that the portable electronic device 1 in which the display 2 was provided performs the said process. In another embodiment, the image may be displayed on an external display. The user holds the hand-held input device provided with the infrared camera 4 and the user performs a gesture using a hand in the side direction of the input device to display the above-described guide display on the display device separated from the input device. An image, an image of an application, a menu screen or the like may be displayed. An information processing system may be constructed in which the hand-held input device provided with the infrared camera 4, the display device, and the information processing device are separated. In this case, for example, the image of the guide display or the image of the application described above is displayed on the display device, and the processing of the application is performed on the information processing device. For example, the information processing apparatus acquires an image from an input device, recognizes a gesture based on the image, and performs processing of an application based on a recognition result of the gesture. In addition, the information processing apparatus acquires an image from the input device and performs processing for displaying the guide (processing for drawing the contour image 21 and the guide frame image 22 and performing processing on these images). Then, the information processing apparatus outputs an image as a result of the processing of the application and the processing for guide display to the display device. The above-described processing may be executed by each of the plurality of devices sharing the processing. Also, the mobile electronic device 1 may function as a controller that receives user input. In this case, a display may or may not be provided. When the portable electronic device 1 functions as a controller, the portable electronic device 1 is connected to another information processing apparatus wirelessly or by wire. When an operation (an operation for an input button, an operation for moving the mobile electronic device 1 itself, the above gesture operation, etc.) is performed on the mobile electronic device 1, operation data corresponding to the operation is transmitted to another information processing apparatus . When the information processing device receives the operation data, the information processing device performs a process according to the operation data. The processing according to the operation data differs depending on the type of application to be executed.

  Further, the shape of the portable electronic device 1 may be any shape. For example, the portable electronic device 1 (display 2) may be vertically long instead of horizontally long, or may be square.

  Moreover, although the infrared camera 4 was used in the said embodiment, it replaces with the infrared camera 4 in another embodiment, and the camera which can pick_up the image (RGB image) of visible light may be used.

  Further, in the above embodiment, an example has been described in which the portable electronic device is held by one hand and the gesture input is performed to the portable electronic device by the other hand. In another embodiment, the portable electronic device may be fixed to the user's arm (body), such as a watch-type device with a screen.

  In addition, the shape of the portable electronic device may be any shape, for example, a plate-like elliptical shape. For example, the portable electronic device may be foldable.

Reference Signs List 1 portable electronic device 2 display 3 touch panel 4 infrared camera 5 distance measuring sensor 6 input button 7 irradiation unit 8 projector 15 communication unit 21 outline image 22 guide frame image 41 composite image

Claims (23)

An input device ,
A camera for imaging the side direction of the input device ;
Acquisition means for acquiring an input image captured by the camera;
Display control means for causing the display means to display an image obtained by tilting an object included in the input image acquired by the acquisition means in a predetermined direction;
And processing means for performing predetermined application processing based on the input image acquired by the acquisition means .
The electronic device system , wherein the display control means displays a guide image for causing a user to recognize whether the object included in the imaging range of the camera is at an appropriate position for the predetermined application processing. . The electronic device system according to claim 1, wherein the display control means causes the display means to display an image obtained by obliquely viewing the input image. 3. The electronic device system according to claim 1, wherein the display control unit displays the image in a manner that allows a user to recognize the relative position between the camera and an object included in an imaging range of the camera. The electronic device system according to any one of claims 1 to 3, wherein the display control means causes the image to be displayed in a mode in which a user can recognize the position of the object included in the imaging range of the camera in the depth direction. The display control means displays the image so that the user looks directly at an object included in the imaging range of the camera and the image of the object displayed on the display means looks the same. The electronic device system according to any one of claims 1 to 4, which is displayed on the means. The guide image is shown to picture image a predetermined region in the imaging range of the camera, an electronic device system according to any one of claims 1 to 5. The electronic device system according to claim 6 , wherein the display control means causes the display means to display an image in which the guide image and the input image are inclined in a predetermined direction. The display means is provided to the electronic device.
The camera captures the side direction of the display means,
The electronic device system according to any one of claims 1 to 7 , wherein the display control means causes the image to be displayed on an imaging direction side of the camera among display screens of the display means. 9. The display control means according to any one of claims 1 to 8 , wherein an image showing a result of the predetermined application processing is displayed together with an image obtained by tilting the subject indicated by the input image acquired by the acquisition means in a predetermined direction. The electronic device system described in. The electronic device system according to any one of claims 1 to 9 , wherein the display control means causes the display means to display an image obtained by abstracting an object included in an imaging range of the camera. Wherein the display control unit, an image showing the outline of an object included in the imaging range of the camera is displayed on the display unit, an electronic device system according to claim 1 0. The electronic device system according to any one of claims 1 to 11, wherein the display control means causes the display means to display only a specific object among a plurality of objects included in an imaging range of the camera. It further comprises detection means for detecting the user's hand based on the input image,
It said processing means based on the detection result by the detection means, performs the application process, the electronic device system according to claim 1 1 2. The sensor detects a gesture made by the user, the electronic device system according to claim 1 3. The image processing apparatus further includes a determination unit that determines whether at least a part of an object included in the input image is out of a predetermined area of an imaging range of the camera.
Wherein the display control unit, wherein an image corresponding to the determination result by the determination means to be displayed on the display unit, an electronic device system according to any one of claims 1 to 1 4. Wherein the display control unit, according to the execution status or user's selection of a predetermined application processing performed by the processing means, to show or hide the image, electronic according to claim 1 1 5 Equipment system . The electronic device system according to any one of claims 1 to 16 , wherein the electronic device is a handheld electronic device. The electronic device system according to any one of claims 1 to 17 , wherein the camera is an infrared camera capable of receiving infrared light. An electronic device ,
A camera for imaging the side direction of the electronic device ;
Acquisition means for acquiring an input image captured by the camera;
Display control means for causing the display means to display an image obtained by tilting an object included in the input image acquired by the acquisition means in a predetermined direction;
And processing means for performing predetermined application processing based on the input image acquired by the acquisition means .
Wherein the display control unit, the object included in the imaging range of the camera, whether in position to display a guide image for recognizing the user for the predetermined application processing, electronic equipment . With the camera,
Acquisition means for acquiring an input image captured by the camera;
Input image display means for causing the display means to display the input image acquired by the acquisition means;
A setting unit configured to set a fixed area that is smaller than an imaging range of the camera and fixed to the imaging range;
Area display means for displaying an area image showing the fixed area on the display means;
A determination unit that determines whether at least a part of an object included in the input image is out of the fixed area;
When it is determined by the determination means that at least a part of the object is out of the fixed area , the display mode of the image displayed on the display means that at least a part of the object is out of the fixed area electronic device system comprising a display mode controller for controlling the display mode shown, the. The electronic device system includes an input device
The camera images the side surface direction of the input device, an electronic device system according to claim 2 0. With the camera,
Acquisition means for acquiring an input image captured by the camera;
A detection unit that detects a specific object included in the input image acquired by the acquisition unit;
Processing means for performing predetermined information processing when the specific object is detected by the detection means;
Condition determination means for determining whether or not a predetermined condition is satisfied based on the input image;
In the case where the specific object is detected by said detecting means, when it is determined that the predetermined condition is satisfied by the condition determining means, a first warning means for performing a first warning,
And second warning means for performing a second warning different from the first warning by the first warning means when the specific object is not detected by the detection means .
The electronic device system , wherein the processing means performs the predetermined information processing based on the input image even when the condition determination means determines that the predetermined condition is satisfied. Further comprising an electronic device system according to claim 2 2 wherein when a particular object is not detected, stopping means for stopping the predetermined information processing by said processing means by said detecting means.

Images