JP2013164701A - Electronic device using user interaction and method for controlling the same, and electronic device control system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an electronic device using user interaction which occurs in a space adjacent to the electronic device and a method for controlling the electronic device and an electronic device control system.SOLUTION: The control method of an electronic device using input interaction includes: a step of recognizing one or more interactions from a space adjacent to the electronic device; and a step of controlling the electronic device in accordance with one or more interactions.

Description

  The present invention relates to an electronic device using user interaction, a control method thereof, and an electronic device control system, and more specifically, an electronic device controlled by user interaction generated in a space adjacent to the electronic device and a control method thereof. And an electronic device control system.

  Recently, with the rapid development of communication technology, the functions of electronic devices have been gradually expanded, thereby providing more various user interfaces (UI) and various functions using the user interfaces. In electronic devices, various input methods for controlling various functions are provided.

  In a general electronic device, in order to control the electronic device, that is, to execute various functions given to the electronic device, the electronic device can be controlled through key input to a key provided in the electronic device. Alternatively, in the case of an electronic device provided with a touch screen, the electronic device can be controlled through touch input to a specific area of the touch screen. Thus, conventionally, it is possible to control an electronic device by directly generating an input to a specific input means for controlling the electronic device.

  However, as described above, in general control of an electronic device, control of each function provided by the electronic device is simply performed by key input or touch input. Therefore, each function of the electronic device has a limit due to restrictions controlled only by key input or touch input, and each function can be limited only by a uniform input method such as key input or touch input. There is a problem with the simplicity being controlled. Further, conventionally, there is a problem that an input unit for controlling one electronic device does not support a function of outputting the result by merging simultaneous inputs of different types of input devices.

The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to control user interaction generated in a space adjacent to an electronic device using user interaction. An electronic device that is controlled by multiple user interactions input in a space adjacent to the electronic device, and that recognizes one or more interactions occurring in the space adjacent to the electronic device and is controlled by the one or more interactions, and A control method and an electronic device control system are provided.
Also, an object of the present invention is to provide a multimodal interface that can easily and intuitively use various functions provided in an electronic device, and in a space adjacent to the electronic device, a planar interaction and a spatial interaction are provided. To provide an electronic device that recognizes one or more interactions and is easily and intuitively controlled by at least one of a plane interaction and a spatial interaction, a control method thereof, and an electronic device control system.

  In order to achieve the above object, an electronic device using input interaction according to an aspect of the present invention includes a device that recognizes one or more interactions that occur in a space adjacent to the electronic device, and the device recognizes And a control unit that controls the electronic device by one or more interactions.

  In order to achieve the above object, an electronic device control system using input interaction according to an aspect of the present invention is a first device for recognizing planar interaction by a user gesture performed on a plane in a space adjacent to the electronic device. A second device for recognizing a spatial interaction by a user gesture performed on a space in a space adjacent to the electronic device, and the planar interaction and the spatial interaction are classified, and the corresponding to at least one of the classified interactions And a processing means for controlling the electronic device.

  In order to achieve the above object, a method of controlling an electronic device using input interaction according to an aspect of the present invention includes recognizing one or more interactions from a space adjacent to the electronic device, and the one or more interactions. And controlling the electronic device in response to the interaction.

The device may detect one or more of a planar interaction and a spatial interaction that occur in a space adjacent to the electronic device. The device may be classified into a device that recognizes the planar interaction by a user gesture and a device that recognizes a spatial interaction by a user gesture.
When the device includes a plurality of devices, the devices may be configured of the same type or different types. Further, the device has a recognition range in which an interaction by a user gesture is recognized, and the recognition range is divided into a plurality of pieces corresponding to the configuration of the device, and overlapped between the divided recognition ranges. A region is included.
When the device detects an interaction within a recognition range, the device provides information for identifying a user gesture corresponding to the interaction. In addition, the device provides information corresponding to planar interaction and spatial interaction from the recognition range of the overlapping region, and provides information corresponding to planar interaction or spatial interaction from the individual recognition range other than the overlapping region. And
The processing means including the control unit classifies the interaction by analyzing information provided by the device. The processing means may classify any one of the planar interaction and the spatial interaction according to information provided by one or more devices, and may control the electronic device in accordance with the classified interaction. Features. The processing unit may control the electronic apparatus in response to the planar interaction and the spatial interaction when the planar interaction and the spatial interaction are generated together by information provided by one or more devices.

  According to the present invention, the electronic device includes one or more devices that recognize one or more user interactions from a space adjacent to the electronic device, and the user performs a user gesture set on the space adjacent to the electronic device. The electronic device can be controlled simply and intuitively. In addition, since an execution function for one or more user interactions recognized through one or more devices can be defined by user settings, various controls on the electronic device can be performed by the input user interactions.

  In addition, according to the present invention, when controlling an electronic device, the present invention is not limited to conventional restrictive and uniform input means and input methods, but can be realized through intuitive control and intuitive control by user gestures using different devices. Control becomes possible. That is, according to the present invention, it is possible to recognize a user gesture performed in a space adjacent to the electronic device through different devices and control the electronic device using the interaction by the user gesture. Thereby, the user can improve user convenience and affinity by electronic device control.

  Further, according to the present invention, for example, when a performance function using an electronic device is executed, the musical instrument can be played by user gesture interaction in a space adjacent to the electronic device, and various musical instrument objects (drum, guitar, trumpet) Etc.) can be played intuitively and intuitively. As described above, according to the present invention, it is possible to provide a variety of user electronic device control inputs in that the electronic devices are controlled using an input method other than key input or screen touch input. effective. That is, according to the present invention, when the electronic device is controlled through diversification of inputs, the user can perform control by various methods and complex control, and the electronic device is restricted and uniform. By improving a simple control method, it is possible to provide reality and improve convenience when using an electronic device.

1 is a diagram schematically illustrating an electronic apparatus according to an embodiment of the present invention. 1 is a diagram schematically illustrating an electronic apparatus according to an embodiment of the present invention. 1 is a diagram schematically illustrating an electronic apparatus according to an embodiment of the present invention. It is a figure which shows schematically the structure of the electronic device by one Embodiment of this invention. It is a schematic diagram for explaining a recognition range for recognizing a plane and spatial interaction in an electronic device according to an embodiment of the present invention. It is a schematic diagram for explaining an operation method of an electronic device by a user gesture in each recognition range according to an embodiment of the present invention. It is a schematic diagram for explaining an operation method of an electronic device by a user gesture in each recognition range according to an embodiment of the present invention. It is a schematic diagram for explaining an operation method of an electronic device by a user gesture in each recognition range according to an embodiment of the present invention. It is a figure which shows the control method using the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows schematically the whole operation | movement which controls the electronic device using the user interaction by one Embodiment of this invention. It is a figure which shows one operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows one operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows one operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows one operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the other operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the other operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the further another operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the further another operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the further another operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention. It is a figure which shows the further another operation example which processes the said function corresponding to the user interaction in the electronic device by one Embodiment of this invention.

  Hereinafter, specific examples of embodiments for implementing an electronic device using the user interaction according to the present invention, a control method thereof, and an electronic device control system will be described in detail with reference to the drawings. Here, in the drawings, the same components are denoted by the same reference numerals as much as possible. Further, detailed descriptions of known functions and configurations that may obscure the subject matter of the present invention are omitted.

  The electronic device of the present invention includes at least one device that recognizes one or more interactions that occur in a space adjacent to the electronic device. In the present invention, the function of the electronic device is controlled by one or more interactions recognized by at least one device.

  In addition, the electronic device of the present invention includes a plane interaction by a user gesture performed on a plane around the electronic device, a space interaction by a user gesture performed on a space, and a fusion interaction in which a plane interaction and a space interaction are performed together. And the function corresponding to the interaction is processed.

  According to the present invention, a user's gesture (hand shape, hand movement, moving direction, etc.) is detected using a device included in the electronic device, and the detected gesture is classified according to a predetermined definition. And used as a meaningful input for electronic equipment. Also, embodiments of the present invention use at least one device to sense and sort user gestures, and the classified inputs are defined to be used for electronic device control interactions.

  Hereinafter, an electronic device that uses user interaction and a method that controls the electronic device using user interaction according to an embodiment of the present invention will be described. However, the electronic device of the present invention and its control operation are not limited to the contents described below, and can be applied to various embodiments based on the following embodiments.

  1 to 3 are diagrams schematically illustrating an electronic apparatus according to an embodiment of the present invention.

  Referring to FIGS. 1 to 3, FIG. 1 shows an example of a portable terminal which is one of electronic devices used in the present invention, and FIG. 2 shows one of electronic devices used in the present invention. FIG. 3 shows an example of a television which is one of the electronic devices used in the present invention.

  Here, the mobile terminal of FIG. 1 is an electronic device including all mobile communication terminals that operate based on communication protocols corresponding to various communication systems, and is a PMP (Portable Multimedia Player). ), Digital broadcasting players, PDAs (Personal Digital Assistants), music players (for example, MP3 players), portable game terminals, smartphones (Smart Phones), and other information communication devices, multimedia devices, and application devices for them. Including. In addition, the guidance system of FIG. 2 provides various guidance information requested by users in museums, exhibition halls, amusement parks, streets, etc., and is a screen monitor fixedly mounted on a wall surface, pillar, ground, or the like. , Media pole, etc. 3 shows an example of a large electronic device, and includes a personal computer, a personal computer monitor, a notebook personal computer, a notebook personal computer monitor, and the like in addition to such a television.

  As shown in FIGS. 1 to 3, the electronic device according to the present embodiment has means for providing an output corresponding to the user's input, and such an electronic device is only a small device portable by the user. It also includes medium and large equipment that is fixed in a specific space. Therefore, hereinafter, a case where the electronic device is a portable terminal will be described as a representative example. However, as described above, the electronic device of the present invention includes all devices in various forms.

  The electronic apparatus according to the present invention senses one or more user interactions that occur in a space adjacent to the electronic apparatus, and processes function control based on the user interactions. To this end, the electronic device of the present invention includes one or more devices to which one or more user interactions are input. Hereinafter, a configuration example for an electronic apparatus including one or more devices will be described with reference to FIG.

  FIG. 4 is a diagram schematically showing the configuration of an electronic device according to an embodiment of the present invention. In particular, FIG. 4 shows an electronic device as an example of one form of a portable terminal that can use user interaction. However, the electronic device of the present invention is not necessarily limited to a portable terminal, and is described above. As described above, all of a variety of small-sized devices and medium-sized devices can be used.

  Referring to FIG. 4, the electronic apparatus according to the present embodiment includes one or more devices. For example, the electronic apparatus includes a first device 410, a second device 430, and a third device 450. In the present embodiment, each of the first to third devices 410, 430, and 450 represents a unit that recognizes one or more interactions corresponding to a user gesture input in a specific space adjacent to the electronic device.

  In this embodiment, each device 410, 430, 450 includes all forms of recognition means that can track user gestures and thereby generate result values. For example, the devices 410, 430, and 450 include recognition means such as a proximity sensor, an infrared sensor, an illuminance sensor, and a camera sensor. In the case of the example in FIG. 4, the electronic apparatus shows a case where the first device 410 and the second device 430 are camera sensors and the third device 450 is a proximity sensor. That is, the electronic apparatus of FIG. 4 shows a portable terminal including a device composed of two camera sensors 410 and 430 and one proximity sensor 450.

  Here, the position of each device 410, 430, 450 in FIG. 4 is not necessarily limited to the form of FIG. For example, in FIG. 4, a first device 410 and a second device 430 corresponding to a camera sensor are provided on each of the left and right sides of the electronic device, and a third device 450 corresponding to a proximity sensor is provided inside the electronic device. The case where it is formed is shown as an example. However, the first device 410 and the second device 430 are respectively provided on the upper and lower sides of the electronic apparatus, and the third device 450 is one or both of the first device 410 and the second device 430. Of course, it can be configured in various ways when manufacturing an electronic device, such as being arranged adjacent to the device.

  In the example of FIG. 4, the electronic apparatus is illustrated as including a number of different devices. However, the electronic apparatus of the present invention is not necessarily limited to such a device configuration. That is, the electronic apparatus of the present invention can be composed of one device. In addition, the electronic apparatus of the present invention can include a plurality of devices of the same type, such as the first device 410 and the second device 430. In addition, the electronic apparatus of the present invention can be a combination of different types of devices such as the first device 410, the second device 430, and the third device 450. The configuration of such a device can be variously provided depending on the form of the electronic device.

  For example, the electronic device may be configured by any one of recognition means such as a proximity sensor, an infrared sensor, an illuminance sensor, and a camera sensor, or a combination of one proximity sensor and one camera sensor. The device can be configured with a single sensor or a combination of a number of camera sensors.

  In the following description of the embodiment of the present invention, as shown in the example of FIG. 4, camera sensors are provided on the left and right sides of the electronic device, respectively, and one proximity sensor is provided inside the electronic device. The operation of the present invention will be described with reference to the portable terminal.

  That is, in the following, as shown in FIG. 4, a user performed around an electronic device using the first device 410 and the second device 430 corresponding to the camera sensor and the third device 450 corresponding to the proximity sensor. Recognize gestures (eg, hand shape, hand movement, hand movement direction, etc.). Further, a user gesture recognized by at least one device is processed as an interaction input for controlling the electronic device.

  Here, in the present embodiment, the third device 450 corresponding to the proximity sensor determines a state in which a user gesture generated in a space adjacent to the electronic device is close to the electronic device (proximity state), and sets a reference for this. provide. Further, the third device 450 is used for applications such as proximity recognition of a body (for example, a user's hand) and a region on a space adjacent to an electronic device. That is, the third device 450 senses the proximity of an object from the electronic device up to a specific distance, and in this embodiment, senses the proximity of the user's hand and takes charge of determining the function of the electronic device. . That is, the third device 450 generates a control signal using spatial interaction by a user gesture performed on a space adjacent to the electronic device.

  In the present embodiment, the first device 410 and the second device 430 corresponding to the camera sensor sense a user gesture and measure a movement direction, a movement speed, a gesture shape (such as a hand shape) of the user gesture, and the like. Used to do. In particular, the first device 410 and the second device 430 generate a control signal using planar interaction by a user gesture performed on a space adjacent to the electronic device.

  Here, in the case where the first device 410 and the second device 430 and the third device 450 generate the spatial interaction and the planar interaction by the user gesture together, the present invention defines the spatial interaction and the planar interaction as a fusion interaction, It is assumed that a control signal is generated by the fusion interaction. That is, according to the present embodiment, when a specific user gesture is sensed by one of the first device 410 and the second device 430 and is sensed together by the third device 430, The device 430 controls the functions of the electronic device by classifying the interaction by the user gesture according to the stage, that is, the proximity level.

  For example, when a spatial interaction is recognized through a proximity sensor, the electronic device is controlled to perform the corresponding function mapped by a spatial interaction user gesture, and when a planar interaction is recognized through a camera sensor, it is mapped by a planar interaction user gesture. When the electronic device is controlled to perform the function and the fusion interaction is recognized through the camera sensor and the proximity sensor, the electronic device is controlled to perform the function mapped by the user gesture of the fusion interaction. Such an example will be described with reference to drawings and tables according to embodiments described later.

  As described above, various configurations for the operation of the present invention include devices such as a proximity sensor, a camera sensor, an infrared sensor, an illuminance sensor, and the like that detect an interaction by a user's gesture, and an electronic device including the device, Then, it can be composed of a processing unit (for example, a control unit, an execution application corresponding to each function, etc.) that processes function control using an interaction detected from at least one device as an input. Here, the control unit is in charge of controlling the electronic device using the interaction according to the present embodiment, and the execution application includes a musical instrument performance application, an image viewer application, a camera function related application, and the like as shown in an example described later. Such an application performs an action defined by interaction with one or more devices.

  FIG. 5 is a schematic diagram for explaining a recognition range for recognizing a plane and space interaction in an electronic apparatus according to an embodiment of the present invention.

  First, one or more devices included in the electronic apparatus according to the present embodiment are used to detect a gesture such as a user's hand movement, and the detected gesture is classified according to a predetermined definition. Use as a meaningful input. Here, in the present embodiment, at least one device is used to sense and classify gestures, and the classified input is defined to be used for control interaction of an electronic device.

  Referring to FIG. 5, the recognition ranges for recognizing the interaction with one or more devices defined in the present embodiment are the plane recognition ranges 510 and 530 in which the first device 410 and the second device 430 recognize the plane interaction. And the third device 450 divides it into a spatial recognition range 550 in which spatial interaction is recognized.

  In the present embodiment, the plane recognition ranges 510 and 530 indicate ranges for the first device 410 and the second device 430 to detect the user's plane interaction from the space adjacent to the electronic device, and the space recognition range 550 is the first range. 3 shows a range for the device 450 to detect a user's spatial interaction from the space adjacent to the electronic device. Here, as shown in the examples of FIGS. 4 and 5, the plane recognition ranges 510 and 530 are divided into left and right regions according to the positions of the first device 410 and the second device 430 with respect to the electronic device, respectively. It is arranged to recognize each gesture with the right hand.

  Here, in the present embodiment, the user's gesture is recognized within the recognition range of at least one of the first device 410 to the third device 450, and the interaction with the gesture is determined. In addition, the electronic device is controlled by interaction.

  For example, assuming that the user performs a musical instrument performance using an electronic device, the user generates an interaction by a gesture input set in a recognition range 510 to 530 in a space adjacent to the electronic device. Then, the first device 410 and / or the second device 430 recognizes the user's gesture in the plane recognition range 510 and / or 530 and generates a plane interaction with the user's gesture, thereby processing the function by the musical instrument performance. Further, the third device 450 recognizes a gesture entering below the limit point with reference to the limit point (boundary line) of the space recognition range 550, and generates a space interaction that strikes the instrument, thereby playing the musical instrument. Process the function by. In addition, by recognizing a user's gesture in the plane recognition range and the spatial recognition range of the first device 410 and / or the second device 430 and the third device 450, and generating a fusion interaction therewith, the function by playing the musical instrument can be achieved. Process.

  As described above, according to the present embodiment, it is possible to perform control so that electronic devices are distinguished by interaction by at least one device. This can be summarized as shown in Table 1 below.

  As shown in Table 1, the control unit of the electronic device detects an interaction only in the plane recognition range 510 of the first device 410, that is, outside the spatial recognition range 550 of the third device 450 (for example, a limit) When an interaction is detected in the plane recognition range 510 of the upper end of the point), the user's gesture measured in the plane recognition range 510 is analyzed, and the electronic device is controlled by a function predefined in the gesture.

  In addition, when an interaction is detected only in the plane recognition range 530 of the second device 430, the control unit of the electronic device, that is, the plane outside the space recognition range 550 of the third device 450 (for example, the upper limit point). When an interaction is detected in the recognition range 530, the user's gesture measured in the plane recognition range 530 is analyzed, and the electronic device is controlled by a function predefined in the gesture.

  In addition, when the interaction is detected only in the space recognition range 550 of the third device 450, that is, the control unit of the electronic device, that is, the plane inside the space recognition range 550 of the third device 450 (for example, the lower end of the limit point). When an interaction is detected in a region other than the recognition ranges 510 and 530, the user's gesture measured in the space recognition range 550 is analyzed, and the electronic device is controlled by a function predefined in the gesture.

  Further, when an interaction is detected in the plane recognition range 510 of the first device 410 and the space recognition range 550 of the third device 450, the control unit of the electronic device, that is, inside the space recognition range 550 of the third device 450 (for example, When the interaction is detected in the region where the plane recognition range 510 overlaps (the lower limit point), the user's gesture measured in the region where the space recognition range 550 and the plane recognition range 510 overlap is analyzed, and the gesture The electronic device is controlled by a predefined function.

  When the interaction is detected in the plane recognition range 530 of the second device 430 and the space recognition range 550 of the third device 450, that is, the inside of the space recognition range 550 of the third device 450 (for example, If the interaction is detected in the region where the plane recognition range 530 overlaps (the lower end of the limit point), the user's gesture measured in the region where the space recognition range 550 and the plane recognition range 530 overlap is analyzed, and the gesture The electronic device is controlled by a predefined function.

  In addition, when an interaction is detected in the plane recognition ranges 510 and 530 of the first device 410 and the second device 430 and the space recognition range 550 of the third device 450, the control unit of the electronic device, that is, the space of the third device 450. An area where the plane recognition range 510 overlaps inside the recognition range 550 (for example, the lower end of the limit point) and a plane recognition range 530 inside the space recognition range 550 of the third device 450 (for example, the lower limit point). When an interaction is detected in each of the overlapping regions, the user's gestures measured in the region where the space recognition range 550 and the plane recognition range 510 overlap and the region where the space recognition range 550 and the plane recognition range 530 overlap are displayed. Analyze and control the electronic device by a function predefined in the gesture.

  In addition, as described above, in the present embodiment, the function is classified according to the measured user gestures and information defined for the gestures, which are classified into planar interaction, spatial interaction, and fusion interaction corresponding to each recognition range. Process control. As described above, in the present embodiment, interaction with respect to two or more levels is enabled by dividing and using a space by using a recognition range of one or more devices.

  Hereinafter, specific embodiments of the present invention described above will be described with reference to FIGS.

  6 to 8 are schematic views for explaining a method of operating an electronic device using a user gesture in each recognition range according to an embodiment of the present invention.

  Referring to FIG. 6, a case in which a user gesture is sensed by the first device 410 and / or the second device 430 outside the spatial recognition range 550 of the third device 450, that is, in a non-overlapping region, and operations according thereto are illustrated. That is, the user can generate a gesture defined in advance in an area that does not overlap with the space recognition range 550 of at least one plane recognition range of the first device 410 and the second device 430. Here, the gesture includes hand movements corresponding to up, down, left and right directions with respect to the electronic device, various forms of hands, various forms of hand movements, and the like. Accordingly, the control unit of the electronic device analyzes the gesture by the interaction recognized by the first device 410 and / or the second device 430, and processes the function control mapped thereto.

  Referring to FIG. 7, a case where a user gesture is sensed by the first device 410 and / or the second device 430 inside the spatial recognition range 550 of the third device 450, that is, in an overlapping region, and operations according thereto are illustrated. That is, the user can generate a gesture defined in advance in an area that overlaps the space recognition range 550 of at least one plane recognition range of the first device 410 and the second device 430. Here, the gesture includes hand movements corresponding to up, down, left and right directions with respect to the electronic device, various forms of hands, various forms of hand movements, and the like. Accordingly, the control unit of the electronic device analyzes the gesture by the interaction recognized by the first device 410 and / or the second device 430 and the third device 450, and processes the function control mapped thereto.

  Referring to FIG. 8, a case where a gesture that crosses the spatial recognition range (limit point) of the third device 450, that is, a recognition range is changed from a non-superimposed region to a superposed region is detected, and an operation performed thereby. That is, the user generates a predefined gesture in which an interaction enters an overlapping area from an area that does not overlap with the space recognition range 550 of at least one plane recognition range of the first device 410 and the second device 430. Can do. Here, the gesture includes an action of a hand entering from the upper side to the lower side (for example, an act of hitting) with reference to the limit point of the space recognition range 550. Accordingly, the control unit of the electronic device senses an interaction that has been moved and changed to a region that overlaps the recognition range of the third device 450 in the recognition range of the first device 410 and / or the second device 430, and the control unit Analyzes gestures by interaction and processes the function control mapped to it.

  The operation of FIG. 8 will be described by taking the case of performing an instrument performance function as an example. The hitting of the instrument can be controlled by an operation in which user interaction enters from the outside of the limit point of the space recognition range 550 to the inside. Here, each of the first device 410 and the second device 430 can detect a change in the frame due to the entry of the user gesture, and recognize the hitting strength according to the degree of the change. Also, the frame change of the first device 410 and / or the second device 430 is analyzed for a gesture entering from the outside of the limit point of the space sensing range 550, thereby determining whether the percussion instrument is struck strongly or weakly. Can be segmented. Such an operation is equally applied to the left side and the right side, and can be used as different sound inputs. An example of such an operation will be described with reference to the drawings described later.

  FIG. 9 is a diagram illustrating a control method using user interaction in an electronic device according to an embodiment of the present invention.

  Referring to FIG. 9, first, the controller of the electronic device activates a mode for electronic device control using user interaction in response to a user request (operation 901). Here, the mode activation can be adaptively activated / deactivated according to a user request, or can always be in an activated state according to a setting method. Therefore, the mode activation step can be omitted in this embodiment.

  Next, the control unit of the electronic device detects the interaction from the space adjacent to the electronic device (step 903). Here, as described with reference to FIGS. 4 to 8 described above, one or more interactions detected through one or more devices are input to the control unit of the electronic device.

  Next, the control unit of the electronic device classifies the input interaction (step 905). That is, the control unit of the electronic device determines the recognition range of the interaction recognized through one or more devices. That is, the control unit determines whether the interaction is an interaction recognized in the individual recognition range of one or more devices or an interaction recognized in the overlap recognition range for at least two devices. In addition, the control unit identifies a user gesture measured in the recognition range where the interaction is recognized.

  Next, the control unit of the electronic device determines a function set by the input interaction (step 907). That is, a function defined in advance for the user gesture identified by the interaction is determined. Then, the control unit of the electronic device processes the determined function control (step 909).

  FIG. 10 is a diagram schematically illustrating an overall operation of controlling an electronic device using user interaction according to an embodiment of the present invention.

  Referring to FIG. 10, first, when a user gesture is generated in a space adjacent to the electronic device, the control unit of the electronic device detects an interaction by the user gesture from one or more devices (step 1010). In addition, when an interaction is detected from one or more devices, the control unit of the electronic device processes a procedure for identifying the interaction (operation 1020).

  Here, the interaction can be variously indicated according to the form in which the electronic apparatus includes the device. For example, when the electronic device is configured only by a device that recognizes a spatial interaction, such as a proximity sensor, only the spatial interaction is detected by the device. In addition, when the electronic device is configured only by a device that recognizes a planar interaction such as a camera sensor, only the planar interaction is detected by the device. In addition, when an electronic device is configured with a large number of different types of devices such as proximity sensors and camera sensors so that both spatial interaction and planar interaction can be individually recognized, the space varies depending on the device. Sensing at least one of an interaction and a planar interaction.

  When the control unit of the electronic device identifies that the identified interaction is a planar interaction (step 1031), as described with reference to FIG. 4 to FIG. 9 described above, the control unit of the electronic device processes the function control by the planar interaction (step 1033). ). At this time, the control unit of the electronic device can process the function control for each type of planar interaction. For example, an interaction set so as to correspond to each user gesture (hand movement, hand shape, movement direction, etc.) generated within the recognition range of the camera sensor is tracked, and the function by the interaction is controlled.

  Also, when the control unit of the electronic device identifies that the identified interaction is a spatial interaction (step 1041), as described with reference to FIGS. 4 to 9 described above, the function control by the spatial interaction is processed ( 1043). At this time, the control unit of the electronic device can process the function control for each type of spatial interaction. For example, an interaction set so as to correspond to each user gesture (hand movement, proximity degree, etc.) generated within the recognition range of the proximity sensor is tracked, and the function by the interaction is controlled.

  When the control unit of the electronic device identifies that the interaction to be identified is a fusion interaction (step 1051), as described with reference to FIGS. 4 to 9, the function control by the fusion interaction is processed (see FIG. 4). Step 1053). The fusion interaction indicates an interaction when a spatial interaction and a planar interaction occur together. That is, a user gesture generated in a region where a recognition range where spatial interaction is detected and a recognition range where planar interaction is detected can be detected, and this is processed as a fusion interaction. At this time, the control unit of the electronic device can process the function control for each type of fusion interaction. For example, it tracks the interaction set to correspond to each user gesture (hand movement, degree of approach, hand shape, direction of movement, etc.) that occurs within the area where the recognition ranges of proximity sensor and camera sensor overlap. Control the function by interaction.

  Next, as described above, the control unit of the electronic device detects a change in the interaction during the function control by the planar interaction, the spatial interaction, and the fusion interaction (step 1060), that is, a user gesture is detected. When the recognition range or the gesture form is changed, the process proceeds to step 1020, and the above-described steps are continued.

  Hereinafter, embodiments of the operation according to the present invention and examples of the screens will be described. However, embodiments of the present invention are not limited to the contents described below, and can be applied to various embodiments based on the following embodiments.

  11 to 14 are diagrams illustrating an example of an operation for processing the function corresponding to user interaction in the electronic device according to the embodiment of the present invention. Here, in the following description, it is assumed that the first device 410 and the second device 430 correspond to camera sensors, and the third device 450 corresponds to a proximity sensor.

  Referring to FIGS. 11 to 14, an operation example is shown in which, when an album application is executed, an interaction is classified according to a proximity level detected by a proximity sensor, and the set function is controlled. That is, FIGS. 11 to 14 classify the interaction by the user gesture generated inside or outside with respect to the boundary (limit point) of the proximity sensor, thereby controlling the mapped function. Hereinafter, with reference to FIG. 11 to FIG. 14, an interaction classification according to the proximity level and a method for controlling various functions will be described.

  First, referring to FIG. 11, when an arbitrary object (for example, an image) is presented by executing an album application, when an interaction occurs within the recognition range of the proximity sensor, a user gesture (for example, left or right hand) As an example, the function defined in (Move) is an object navigation function.

  That is, as shown in FIG. 11, the control unit of the electronic device recognizes a user gesture input inside the recognition range of the proximity sensor, and the user gesture is one of up, down, left, and right within the recognition range of the camera sensor. Recognize that it has moved in one direction. As a result, the control unit fuses the interaction by the proximity sensor and the interaction by the camera sensor, and determines that the gesture is a fusion interaction. In addition, the control unit executes a function defined in advance by the moving direction of the user gesture, that is, the gesture to be tracked in the fusion interaction. That is, the navigation function for the presented object corresponding to the movement direction of the gesture is processed.

  As described above, when a fusion interaction is generated by the proximity sensor and the camera sensor, that is, when a gesture by the camera sensor is sensed together while a gesture by the proximity sensor is sensed, the controller of the electronic device performs the fusion interaction. The function control for navigating between objects by the gesture detected by the camera sensor is processed.

  Next, referring to FIG. 12, when an arbitrary object (for example, an image) is presented by executing the album application, when an interaction occurs within the recognition range of the proximity sensor, a user gesture (for example, a hand) As an example, the function defined in (up / down movement) is an enlargement / reduction function.

  That is, as illustrated in FIG. 12, the control unit of the electronic device recognizes the user gesture input inside the proximity sensor recognition range, and the user gesture approaches the electronic device inside the proximity sensor recognition range. It is recognized that it has been changed or changed. Accordingly, the control unit determines that the gesture is a spatial interaction due to a proximity level change by the proximity sensor. In addition, the control unit executes a function defined in advance by the proximity level of the user gesture, that is, the gesture to be tracked in the spatial interaction. In other words, the enlargement / reduction function for the presented object is processed corresponding to the degree that the gesture is close to the electronic device. Here, when the gesture is out of the recognition range (limit point) of the proximity sensor, it is needless to say that the operation of restoring the object to the basic size set as a basic setting can be performed by the setting method.

  As described above, when the gesture by the proximity sensor is sensed or when the gesture by the camera sensor is sensed together with the gesture by the proximity sensor sensed, the control unit of the electronic device is sensed by the proximity sensor. Function control for enlarging / reducing an object by a gesture is processed.

  Next, referring to FIG. 13, when an arbitrary object (for example, an image) is presented by the execution of the album application, when an interaction occurs outside the recognition range of the proximity sensor, a user gesture (for example, a hand) As an example, the function defined in (Move Up / Down) is a category navigation function.

  That is, as shown in FIG. 13, no interaction by the proximity sensor occurs for a user gesture input outside the recognition range of the proximity sensor. That is, the user gesture is detected only within the recognition range of the camera sensor, and an interaction is generated accordingly. In such a case, when the control unit of the electronic device recognizes a gesture that moves in one of the upper, lower, left, and right directions within the recognition range of the camera sensor, it determines the interaction by the camera sensor as a planar interaction, and the user interacts with the planar interaction. A function (for example, category navigation control) mapped to the movement direction of the gesture, that is, the gesture to be tracked is executed. That is, the navigation function for the category presented corresponding to the movement direction of the gesture is processed.

  Therefore, as described above, when a plane interaction is generated by the camera sensor, the function control for performing navigation between categories is processed by the plane interaction. At this time, contents (for example, images) belonging to the category are also changed and provided by changing the category.

  Next, referring to FIG. 14, when an arbitrary object (for example, an image) is presented by executing the album application, when an interaction occurs outside the recognition range of the proximity sensor, a user gesture (for example, a hand) As an example, the function defined in (Rotation) is the object rotation function.

  That is, as shown in FIG. 14, the interaction by the proximity sensor does not occur for a user gesture input outside the recognition range of the proximity sensor. That is, the user gesture is detected only within the recognition range of the camera sensor, and an interaction is generated accordingly. In such a case, when the control unit of the electronic device recognizes a gesture that rotates clockwise or counterclockwise within the recognition range of the camera sensor, it determines the interaction by the camera sensor as a planar interaction, and the user gesture by the planar interaction, that is, A function (for example, object rotation control) mapped in the rotation direction of the gesture to be tracked is executed. That is, the rotation function for the presented object corresponding to the rotation direction of the gesture is processed.

  Accordingly, as described above, when a plane interaction is generated by the camera sensor, the function control for rotating the object by the plane interaction is processed. At this time, the effect of rotating the object in the direction corresponding to the clockwise or counterclockwise rotation of the gesture is adaptively reflected and provided.

  As described above, according to the operation example of the present invention, it is possible to determine a spatial range by proximity using a proximity sensor. User-defined, for example, the gesture direction is recognized by the camera sensor inside the proximity range and used as the interaction, and the gesture shape / motion is recognized by the camera sensor outside the proximity sensor and used in the interaction. Various operations are possible.

  FIGS. 15 and 16 are diagrams illustrating another example of operations for processing the function corresponding to user interaction in the electronic device according to the embodiment of the present invention. Here, in the following, the first device 410 and the second device 430 correspond to the camera sensor, the third device 450 corresponds to the proximity sensor, and the camera sensor that is the first device 410 has the function of the camera body. The camera sensor that is executed and is the second device 430 will be described as operating as a function for user interaction in the present embodiment. Of course, the functions between the first device 410 and the second device 430 can be exchanged and used.

  Referring to FIGS. 15 and 16, an example of an operation of controlling the function by classifying the interaction according to the proximity level detected by the proximity sensor is shown. In particular, FIG. 15 and FIG. 16 show an example of performing a self-photographing function among functions using a camera, and it is needless to say that such an operation can also be applied during an image call.

  Referring to FIG. 15 and FIG. 16, the functions related to self-photographing are classified by the user gesture generated inside or outside based on the recognition range (limit point) of the proximity sensor, and the user gesture corresponding to each interaction. An example of the operation for controlling is shown. That is, FIGS. 15 and 16 illustrate an operation example in which, when the self-photographing application is executed, the interactions detected by the proximity sensor and the remaining other camera sensors are classified, and the set self-photographing function is controlled.

  First, referring to FIG. 15, a case where an interaction by a user gesture occurs inside the proximity sensor recognition range 550 during execution of the self-photographing function is shown. As shown in FIG. 15, the control unit of the electronic device recognizes the user interaction input inside the proximity sensor recognition range 550, and the user gesture approaches the electronic device inside the proximity sensor recognition range 550. It is recognized that it has been changed or changed. Thereby, the control unit determines that the gesture is an interaction due to a proximity level change by the proximity sensor. In addition, the control unit executes a function defined in advance according to the proximity level of the user gesture in the interaction, that is, the gesture to be tracked. For example, the zoom-in / zoom-out function of the subject by self-photographing is processed in accordance with the degree to which the gesture is close to the electronic device.

  As described above, when the gesture by the proximity sensor is sensed or when the gesture by the camera sensor is sensed together with the gesture by the proximity sensor sensed, the control unit of the electronic device is sensed by the proximity sensor. The zoom-in / zoom-out function control by self-photographing is processed by the gesture.

  That is, in FIG. 15, when a user interaction is generated by the camera sensor, a user interaction is generated by the proximity sensor, or a user interaction is generated only by the proximity sensor, self-photographing is performed according to the proximity level of the user interaction by the proximity sensor. The zoom in / zoom out function can be controlled. At this time, the zoom-in or zoom-out function is processed according to the proximity level generated within the recognition range 550 of the proximity sensor, that is, the strength of the recognized recognition value. Referring to FIG. 15, when the user's hand approaches the electronic device side from the limit point of the proximity sensor recognition range 550, the zoom-in function is processed, and the user's hand is far from the limit point of the recognition range 550 in the electronic device. If this happens, the zoom-out function is processed.

  Next, referring to FIG. 16, a case where an interaction by a user gesture occurs outside the proximity sensor recognition range 550 during execution of the self-photographing function is shown. As shown in FIG. 16, the user gesture is detected only within the recognition range 550 of the camera sensor, and an interaction occurs accordingly. In such a case, when the control unit of the electronic device recognizes the gesture generated in the recognition range 550 of the camera sensor, the control unit executes a function mapped according to the user gesture, that is, the form of the gesture to be tracked by the interaction with the camera sensor. For example, when the gesture has a hand shape corresponding to “OK”, the photographing function by self-photographing is processed corresponding to the shape of the hand.

  That is, in FIG. 16, when user interaction is generated by the camera sensor, various additional functions by self-photographing can be processed according to the form of the gesture tracked by the camera sensor.

  On the other hand, the function control between the camera sensor and the proximity sensor such as self-photographing is not necessarily limited to the above example, and the function control information between the user gesture and the device can be changed by the user setting. Such an example can be shown in Table 2 below.

  As shown in Table 2 above, in the embodiment of the present invention, an interaction, a gesture, and a function corresponding thereto can be set by user settings. For example, for the same zoom-in / zoom-out function, sense fusion interaction by proximity sensor and camera sensor, or sense planar interaction or spatial interaction by any one device, A gesture for generating an interaction can also be set according to a user's request. Also, the same gesture and the function based on the same gesture can be set in various ways depending on the user's convenience, such as whether to perform by spatial interaction or planar interaction.

  17 to 20 are diagrams showing still another example of operations for processing the function corresponding to the user interaction in the electronic device according to the embodiment of the present invention. Here, in the following description, it is assumed that the first device 410 and the second device 430 correspond to camera sensors, and the third device 450 corresponds to a proximity sensor.

  Here, the electronic devices shown in FIGS. 17 to 20 are devices for identifying user interaction, and two camera sensors are provided on the sides of the plane of the electronic device, respectively. An example in which a proximity sensor is provided on the side is shown. An example of such a configuration corresponds to an example of various electronic device forms according to the present invention, and the electronic device according to the present invention is not necessarily limited to the form shown in FIGS. .

  First, FIG. 17 shows an operation example in which a user inputs a gesture when executing a musical instrument performance application in an electronic device. As shown in FIG. 17, each of the left hand and the right hand of the user is located on each recognition range of the two camera sensors 410 and 430. At this time, each of the two camera sensors 410 and 430 continuously senses a hand movement with respect to the left hand and the right hand.

  Further, the recognition range (limit point) of the proximity sensor 450 is regarded as a virtual surface of the percussion instrument, and it is defined that the percussion instrument is hit when an input entering from the outside to the inside of the recognition range (limit point) is detected. In addition, each camera sensor 410, 430 can sense the moving speed of the user gesture, which can be defined as the strength of playing the instrument.

  In such an example, as shown in the example of FIG. 17, assuming that a musical instrument performance application is executed on an electronic device, the two camera sensors 410 and 430 and the proximity sensor 450 each move through left and right hand movements. It is used for playing a musical instrument, tracking the frame change sensed by each camera sensor 410, 430, and measuring the speed of hand movement.

  On the other hand, FIG. 17 shows an example in which two camera sensors 410 and 430 are used. However, when two or more camera sensors are provided and each recognition range is divided, a large number of inputs are used. Can accept.

  Hereinafter, an operation for executing a musical instrument performance function in an electronic device using the user interaction of the present invention will be described.

  18 to 20, first, the electronic device displays a user interface (UI) corresponding to execution of the musical instrument performance application on the screen. The user interface is a graphic user interface (GUI) corresponding to an actual musical instrument form. Or the GUI which simplified and showed the actual musical instrument form may be sufficient. 18 to 20 show examples in which the form of the musical instrument corresponding to the user request is simplified and shown in the GUI. For example, FIGS. 18 to 20 show GUIs corresponding to the case where the user selects a drum among percussion instruments.

  First, as shown in FIG. 18, in the case of hitting a musical instrument, only an action entering inside from the recognition range of the proximity sensor 450 is recognized and processed as an interaction. At this time, each of the camera sensors 410 and 430 senses a change in the frame and processes an interaction depending on the degree of the change. That is, the strength by the user gesture is classified. In addition, frame changes are analyzed by the camera sensors 410 and 430 with respect to the movement of the gesture entering from the outside of the recognition range of the proximity sensor 450, thereby distinguishing whether the percussion instrument was hit strongly or weakly. Such an operation is equally applied to the left and right sides, and can be used as different sound inputs.

  Next, as shown in FIG. 19, repetitive weak input in musical instrument performance is defined as a wave action inside the recognition range of the proximity sensor 450, and a wave gesture is detected in the recognition range where the proximity sensor and the camera sensor overlap. When such an interaction occurs, it is processed as a continuous weak input in the musical instrument performance function.

  Next, as shown in FIG. 20, the musical instrument may continue to sound due to its characteristics, and thus defines an interaction that mute the current sound. In the example of FIG. 20, an input for controlling the mute function is defined as a wave operation performed outside the recognition range of the proximity sensor 450, and interaction is generated by the camera sensors 410 and 430 outside the recognition range of the proximity sensor 450. Then, this is processed as a mute input in the musical instrument performance function. In such an operation, only one of the user interactions is muted, or both the left and right are muted.

  As described above, according to the present invention, electronic devices can be controlled by user interaction using one or more devices. For example, the camera sensor can continuously detect a movement of a hand movement and use the movement as each interaction. Further, the proximity sensor can be used as a role of defining a proximity range and partitioning an area in space. According to the present invention, an extended interaction can be provided by introducing the concept of space (proximity level) by combining two sensors as compared to an interaction method using a single sensor.

  As mentioned above, although embodiment of this invention was described referring drawings, this invention is not limited to the above-mentioned embodiment, In the range which does not deviate from the technical scope of this invention, various changes are implemented. It is possible.

410 First device 430 Second device 450 Third device 510, 530 Plane recognition range 550 Spatial recognition range

Claims (15)

An electronic device that uses input interaction,
A device for recognizing one or more interactions occurring in a space adjacent to the electronic device;
An electronic device comprising: a control unit that controls the electronic device by one or more interactions recognized by the device. The device is
A first device for recognizing planar interaction by a user gesture performed on a plane around the electronic device;
The electronic device according to claim 1, further comprising: a second device that recognizes a spatial interaction by a user gesture performed on a space around the electronic device.   The electronic device according to claim 2, wherein the first device and the second device are configured with the same type or different types. The device has a recognition range in which an interaction by a user gesture is recognized,
The electronic device according to claim 2, wherein the recognition range is divided into a large number corresponding to the configuration of the device, and an overlapping region is formed between the divided recognition ranges.   The device provides information corresponding to planar interaction and spatial interaction from the recognition range of the overlapping region, and provides information corresponding to planar interaction or spatial interaction from an individual recognition range other than the overlapping region. The electronic device according to claim 4.   The electronic device according to claim 5, wherein the control unit divides planar interaction and spatial interaction according to information from the device, and controls the electronic device in accordance with at least one divided interaction. . An electronic device control system using input interaction,
A first device for recognizing planar interaction by a user gesture performed on a plane in a space adjacent to the electronic device;
A second device for recognizing spatial interaction by a user gesture performed on the space in a space adjacent to the electronic device;
An electronic device control system comprising: processing means for classifying the planar interaction and the spatial interaction and controlling the electronic device in response to at least one of the classified interactions.   The electronic device control system according to claim 7, wherein the first device and the second device are devices of the same type or different types of devices. The first device has a recognition range for the planar interaction, the second device has a recognition range for the spatial interaction;
The electronic device control system according to claim 8, wherein the recognition range of the first device and the recognition range of the second device have overlapping regions. A method for controlling an electronic device using input interaction,
Recognizing one or more interactions from a space adjacent to the electronic device;
Controlling the electronic device in response to the one or more interactions. Recognizing the one or more interactions comprises:
Sensing a user gesture from a space adjacent to the electronic device;
The method according to claim 10, further comprising: recognizing one or more interactions corresponding to the user gesture. Recognizing the one or more interactions comprises:
12. The method of controlling an electronic device according to claim 11, wherein at least one of a plane interaction and a spatial interaction is recognized according to a recognition range in which the user gesture is generated in a space adjacent to the electronic device.   The recognition range in which the user gesture is generated is divided into a recognition range for the planar interaction and a recognition range for the spatial interaction, and a recognition range for the planar interaction and a recognition range for the spatial interaction. The method for controlling an electronic device according to claim 12, further comprising a region where is superimposed. The step of controlling to control the electronic device includes:
13. The electronic device control according to claim 12, further comprising: identifying one of the planar interaction and the spatial interaction, and controlling the electronic device in response to the identified interaction. Method. The step of controlling to control the electronic device includes:
13. The method of controlling an electronic device according to claim 12, further comprising the step of controlling the electronic device corresponding to the planar interaction and the spatial interaction when the planar interaction and the spatial interaction occur together.

Images