JP2016072698A - Information processing device, display device, tv set, information processing device control program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, a display device, and a TV set which allow a user to perform simple and various operation instructions.SOLUTION: A command determination device 10 comprises: an IR command receiver 100 for receiving an IR command from a display device 20; an image receiver 110 for receiving an image captured by a camera 30 from the camera 30; a gesture recognition unit 120 which performs gesture recognition; a control command determination unit 130 which determines a control command with respect to the display device 20 in accordance with the IR command and the gesture; and a control command transmission unit 140 which causes the display device 20 to execute the control command.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus that transmits a control instruction to a display device that displays an image.

  In recent years, electronic devices that perform input operations using a remote controller such as a television have become more and more rich in functions. For example, a digital television (smart television) having functions as an Internet browser, VOD (video on demand) and the like has been developed.

  On the other hand, as the functions of the electronic device become richer in this way, the operation of the electronic device becomes more complicated and the number of buttons on the remote controller tends to increase. In this case, when the user performs an input operation, he / she once looks away from an electronic device such as a television, confirms a button on the remote controller, and performs the input operation. Therefore, the user cannot receive information during an input operation on a television or the like. In addition, since the number of buttons on the remote controller increases as described above, the user may feel stress because the position of the button to be operated is unknown.

  On the other hand, various techniques have been developed to simplify the operation of electronic devices. For example, Patent Document 1 discloses a technique for determining an operation signal for operating an electronic device from pressing of an operation key of a remote controller and movement of a hand holding the remote controller. Patent Documents 2 and 3 describe techniques for recognizing a movement of a remote controller as a gesture and controlling an electronic device according to the gesture instead of pressing a button.

Japanese Patent Laid-Open No. 11-285083 (released on October 15, 1999) JP 2008-300895 A (released on December 11, 2008) JP 2010-045730 A (released February 25, 2010)

  However, when the technique disclosed in Patent Document 1 is applied, it is necessary to use a dedicated remote controller that includes a sensor for specifying the movement of a hand having a remote controller such as an acceleration sensor. Therefore, for example, when the remote control is lost or when the remote control breaks down, it is necessary to prepare a dedicated remote control again. In addition, when applying the techniques disclosed in Patent Documents 2 and 3, a remote control that can arrange only a limited number of buttons limits the types of input operations related to button presses. It is necessary to increase the types of gestures to be recognized. For this reason, the user needs to acquire a large number of gestures, and there is a problem that the convenience of operation by the gestures is impaired.

  The present invention has been made in view of the above problems, and its purpose is to provide an information processing device, a display device, a television, a control program, a recording medium, and the like that allow a user to simply and perform various types of operation instructions. Is to realize.

  In order to solve the above-described problem, an information processing apparatus according to an aspect of the present invention includes a command receiving unit that receives a command corresponding to a type of input operation performed on an operating device for a display device, and When the command accepting unit accepts the command, the gesture recognizing unit for recognizing the predetermined motion of the operating device from the video of the operating device taken by the camera, and the gesture recognizing unit recognizes the predetermined motion. In this case, the control command determining means for determining a control command corresponding to the command is different from the control command when the gesture recognition unit does not recognize the predetermined movement, and the control command is sent to the display device. And a control command instruction means to be executed.

  According to an aspect of the present invention, there is an effect that it is possible to allow a user to easily perform various types of operation instructions.

It is a block diagram which shows the principal part structure of the television (TV) which concerns on Embodiment 1 of this invention. It is a figure which shows the operation | movement outline | summary of the said TV. 2A shows an example of an input operation using a remote controller, and FIG. 2B shows an example of the operation of the TV after the user performs the input operation shown in FIG. 2A. . It is a figure which shows an example of a data structure of the control command determination information stored in the memory | storage part of the command determination apparatus contained in the said TV. It is a figure which shows an example of input operation and gesture with respect to a remote control, and the screen transition of the said TV according to the said input operation and gesture. It is a figure which shows another example of the input operation and gesture with respect to a remote control, and the screen transition of the said TV according to the said input operation and gesture. It is a figure which shows another example of the input operation and gesture with respect to a remote control, and the screen transition of the said TV according to the said input operation and gesture. It is a figure which shows another example of the input operation and gesture with respect to a remote control, and the screen transition of the said TV according to the said input operation and gesture. It is a flowchart which shows the flow of the process in the said command determination apparatus. It is a flowchart which shows the flow of a process in the command determination apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the principal part structure of TV which concerns on Embodiment 3 of this invention. It is a figure which shows an example of the screen which TV which concerns on Embodiment 3 of this invention displays.

Embodiment 1
Hereinafter, Embodiment 1 of the present invention will be described with reference to FIGS. First, an outline of the operation of the television (TV) 1 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an outline of the operation of the TV 1. 2A shows an example of an input operation using the remote controller, and FIG. 2B shows an example of the operation of the TV 1 after the user performs the input operation shown in FIG. 2A. Show. A broken line in FIG. 2A indicates a range in which the camera 30 can capture an image.

≪Configuration of remote control≫
A remote controller (operation device) 2 is a remote controller for controlling the TV 1. The remote controller 2 is provided with an input unit such as buttons and switches for accepting user input operations on the TV 1. When input is performed via the input unit, the remote controller 2 transmits an infrared (IR) signal corresponding to the type of the input operation. In other words, the remote controller 2 transmits an IR signal corresponding to each input unit included in the own device. In addition, the remote controller 2 transmits an IR signal related to the button at a predetermined time interval while the button is pressed.

  The remote controller 2 may be a remote controller dedicated to the TV 1 or a general-purpose remote controller for operating a television or electronic device including the TV 1. In the present embodiment, hereinafter, it is assumed that the remote controller 2 includes a button as an input unit, and the user performs an input operation on the TV 1 by pressing the button.

  The remote controller 2 may transmit a signal other than infrared light as long as the signal indicates a command. In this case, the light receiving unit 200 described later is configured to receive a signal transmitted from the remote controller 2.

≪Configuration of TV (TV) ≫
The TV 1 according to the present embodiment is an electronic device that displays an image, which includes a display device 20, a camera 30, and a command determination device 10 to be described later. The TV 1 is assumed to include members for realizing functions generally provided in a television receiver in addition to the above-described units. For example, the TV 1 may include an output unit (not shown) for an audio output function. The TV 1 determines its own process control according to the type of button pressed on the remote controller 2 and whether or not the user has performed a gesture using the remote controller 2 following the pressing of the button. In other words, the TV 1 determines its processing control according to the command (IR command) indicated by the IR signal corresponding to the pressed button and whether or not the gesture is recognized when the IR command is received. Note that when the gesture is performed, the TV 1 may further determine its own process control according to the type of the gesture.

  Note that the gesture recognition may be performed during a period in which it is possible to recognize a gesture following the button press (a gesture estimated to be performed in association with the button press). In the present embodiment, hereinafter, the TV 1 performs gesture recognition when the user continues to press the button on the remote controller 2 for a predetermined time or longer (the button is pressed long).

  As shown in FIG. 2A, when the user presses a button on the remote controller 2, the remote controller 2 continues to transmit IR signals related to the pressed button at predetermined time intervals while the button is pressed. . The TV 1 receives the IR signal at the light receiving unit 200. When the user presses and holds the button on the remote controller 2, the TV 1 starts gesture recognition. In this state, when the user performs a gesture of shaking the remote controller 2 to the right, the TV 1 recognizes the gesture from the video of the camera 30.

  Then, the TV 1 controls itself according to the received IR signal and the recognized gesture. For example, as shown in FIG. 2B, the TV 1 changes the image to be displayed on its own device (display control is performed so as to change the screen).

  By the way, in FIG. 2A, the TV 1 has its own screen, an area α for displaying contents such as TV programs and moving images, and an area β for displaying a screen for operating the TV 1 (remote control screen). The display is divided into areas. However, the TV 1 does not have to divide and display its own screen in this way. For example, the TV 1 does not display the remote control screen (without securing the area β), accepts an input operation from the remote control in a state where the content is displayed on the entire screen, recognizes the gesture, and performs the input operation and the gesture recognition. In response to this, the screen shown in FIG. 2B may be displayed. The same applies to the drawings referred to hereinafter.

  Next, the principal part structure of TV1 is demonstrated with reference to FIG. FIG. 1 is a block diagram showing a main configuration of the TV 1 according to the present embodiment. In FIG. 1, the remote controller 2 is also shown for explaining the configuration of the TV 1. As shown in FIG. 1, the TV 1 includes a display device 20, a camera 30, and a command determination device (information processing device) 10. A white arrow in the figure indicates that the camera 30 captures an image of the remote controller 2 as a subject.

≪Display device≫
The display device 20 is a device that displays an image in accordance with a control command received from the command determination device 10. The display device 20 is a device that receives an IR signal transmitted from the remote controller 2 and transmits it to the command determination device 10. More specifically, the display device 20 includes a light receiving unit 200, an IR command transmission unit (command transmission unit) 210, a control command reception unit (control command reception unit) 220, a display control unit (display control unit) 230, A display (display unit) 240.

  The light receiver 200 receives the IR signal transmitted from the remote controller 2 and transmits the IR signal to the IR command transmitter 210.

  The IR command transmission unit 210 specifies an IR command from the IR signal received from the light receiving unit 200 and transmits the IR command to the command determination device 10.

  The control command receiving unit 220 receives a control command that is a response to the IR command from the command determining device 10 and transmits the control command to the display control unit 230. The control command receiving unit 220 receives not the control command itself from the command determining device 10 but information (code or the like) specifying the control command, specifies the control command from the information, and displays the specified control command as the display control unit. 230 may be transmitted.

  Display control unit 230 causes display 240 to display an image in accordance with the control command received from control command receiving unit 220. Note that the image data to be displayed on the display 240 may be image (video) data stored in the TV 1 or a recording medium such as a recorder connected to the TV 1, or the TV 1 receives a receiving unit (not shown). It may be image (video) data obtained from a carrier wave received at, or image (video) data acquired by TV 1 from a network such as the Internet via a communication unit (not shown).

  The display 240 displays an image according to the display control of the display control unit 230. Although the specific configuration of the display 240 is not particularly limited, for example, a liquid crystal display or an organic EL display can be used as the display 240.

≪Camera≫
The camera 30 captures an image from the remote controller 2 in accordance with a capturing instruction from the command determining device 10. As shown in FIGS. 2A and 2B, the camera 30 is provided so as to face a direction in which the user points the remote controller 2 when performing an input operation, such as the front of the TV 1, in order to photograph the remote controller 2. It is desirable. The camera 30 transmits the captured video to the image receiving unit 110 of the command determining device 10.

  In the present embodiment, the camera 30 is built in the TV 1. However, the camera 30 is provided independently of the TV 1, and the camera 30 and the image receiving unit 110 of the command determination device 10 are connected to be communicable. Also good.

  Further, the camera 30 may always take a picture and transmit the video to the command decision apparatus 10 instead of receiving a photography instruction from the command decision apparatus 10.

  In addition, the camera 30 only needs to be able to capture a video with an image quality that allows gesture recognition (described later) in the gesture recognition unit 120. Further, the camera 30 may shoot a continuous image (still image) instead of shooting a video, and transmit the continuous image to the image receiving unit 110.

≪Command decision device≫
The command determination device 10 receives an IR command from the display device 20 and a video captured by the camera 30 from the camera 30 and determines a control command for the display device 20 according to the IR command and a gesture recognized from the video. It is a device to do. More specifically, the command determination device 10 includes a storage unit 190, an IR command reception unit 100 (command reception unit), an image reception unit 110, a gesture recognition unit (gesture recognition unit) 120, and a control command determination unit (control command). Determination means) 130, a control command transmission unit (control command instruction unit) 140, and a gesture recognition start determination unit 150.

  The storage unit 190 stores various types of information of the command determination device 10 including control command determination information. Here, the “control command determination information” is information that associates each IR command, that is, a gesture that is a target of gesture recognition while receiving each IR command (while pressing the button) with each button of the remote controller 2. Furthermore, the control command determination unit 130 associates the control command determined by the combination of the IR signal type and the recognizable gesture type with information.

≪Control command decision information≫
Here, the data structure of the control command determination information will be described in detail with reference to FIG. FIG. 3 is a diagram illustrating an example of a data configuration of control command determination information stored in the storage unit 190. The control command determination information only needs to be information that can uniquely determine the control command from the combination of the IR command type and the gesture type, and the data structure is not limited to the table format.

  As illustrated, the control command determination information includes a “button” column, an “IR signal” column, a “gesture” column, and a “control command” column. In the control command determination information, information in the “gesture” column and the “control command” column is associated with information in the “IR signal” column.

  The “IR signal” column stores the value (code) of the IR signal transmitted when the remote controller 2 presses the button. The “button” column stores information indicating a button corresponding to the IR signal indicated by the “IR signal” column (a button for transmitting the IR signal described in the “IR signal” column when pressed). The “gesture” column stores information indicating the type of gesture subject to gesture recognition when the IR signal indicated by the “IR signal” column is received. The “control command” column stores information indicating a control command corresponding to a combination of information in the “IR signal” column and the “gesture” column.

  By the way, the IR signal is a signal determined according to the type of the pressed button as described above. In other words, there is a one-to-one correspondence between IR signal (type) and button type. Therefore, from the information in the “IR command” column, the type of button pressed by the remote controller 2, that is, the information in the “button” column can be specified. Therefore, the information in the “button” column is not essential information.

  The control command determination information is read by the gesture recognition start determination unit 150 and the control command determination unit 130 described later.

  The IR command receiving unit 100 receives an IR command from the IR command transmitting unit 210 of the display device 20 and transmits the IR command to the gesture recognition start determining unit 150 and the control command determining unit 130. As described above, the remote controller 2 transmits an IR signal related to the button at a predetermined time interval while the button is pressed, and the IR command transmission unit 210 of the display device 20 receives the IR command from the IR signal. I have identified. Therefore, the IR command receiving unit 100 receives IR commands at predetermined time intervals while the button is pressed.

  The gesture recognition start determination unit 150 determines the start and end of the gesture recognition process in the TV 1 (command determination device 10) from the IR command received from the IR command reception unit 100.

  When the gesture recognition start determination unit 150 receives the IR command of the remote controller 2 from the IR command reception unit 100, the gesture recognition start determination unit 150 first determines whether the input operation (pressed button) related to the IR command corresponds to gesture recognition. . More specifically, the gesture recognition start determination unit 150 first refers to the control command determination information stored in the storage unit 190 so that the IR command received by the IR command reception unit 100 is described in the control command determination information. Check if it is.

  When the IR command is described in the control command determination information, that is, when the gesture to be recognized is associated with the IR command while receiving the IR command, the gesture recognition start determining unit 150 sends the gesture to the remote controller 2. It is determined that the pressed button is a button corresponding to gesture recognition. On the other hand, when the received IR command is not described in the control command determination information, the gesture recognition start determining unit 150 determines that the button pressed on the remote controller 2 is a button that does not support gesture recognition.

  When the button pressed on the remote controller 2 corresponds to gesture recognition, the gesture recognition start determination unit 150 further determines whether or not the button is pressed for a long time. Specifically, the gesture recognition start determination unit 150 receives the button A IR command (the same IR command) from the IR command reception unit 100 continuously for a predetermined time (first time) or longer. Is pressed for a long time, and if the IR command for button A is not continuously received for a predetermined time or more, it is determined that button A is not pressed for a long time (normal button press (short press)).

  The gesture recognition start determination unit 150 also receives an IR command indicating a repeat signal indicating that the IR command reception unit 100 is not the same signal but the same signal transmitted last time. 100 may be regarded as accepting the same IR command.

  In addition, the predetermined time can be arbitrarily set to a time longer than the time required for short-pressing the button with the remote controller 2 (a time that can be estimated that the button is long-pressed, for example, about 200 ms). .

  In this way, the gesture recognition start determination unit 150 makes the gesture recognition unit 120 perform gesture recognition when the button is pressed for a long time, thereby determining whether or not the user has pressed the button with the intention of performing the gesture. Can be estimated and gesture recognition can be performed.

  Here, when the button is pressed for a long time, the gesture recognition start determination unit 150 starts the gesture recognition mode. Specifically, the gesture recognition start determination unit 150 transmits an image capturing instruction to the camera 30 and transmits a gesture recognition start instruction to the gesture recognition unit 120. In addition, the gesture recognition start determination unit 150 notifies the gesture recognition unit 120 of the type of gesture corresponding to the button A.

  The gesture recognition start determination unit 150 also instructs the gesture recognition unit 120 to end gesture recognition when the IR command reception by the IR command reception unit 100 is interrupted or when the gesture recognition unit 120 fails to recognize the gesture. Send.

  The image receiving unit 110 receives the video captured by the camera 30 and transmits it to the gesture recognition unit 120.

  The gesture recognition unit 120 performs gesture recognition based on the video received from the image reception unit 110. The gesture recognition unit 120 starts and ends gesture recognition in accordance with the gesture recognition start instruction and end instruction received from the gesture recognition start determination unit 150.

  More specifically, when the movement of the remote controller 2 included in the video received from the image receiving unit 110 is a predetermined movement, the gesture recognition unit 120 recognizes the predetermined movement as a gesture. In addition, when there are a plurality of gestures corresponding to the pressed button, the gesture recognition unit 120 also identifies which gesture is the recognized gesture and the type of the gesture.

  The gesture recognition unit 120 notifies the control command determination unit 130 of the result of gesture recognition (whether or not a gesture has been recognized and its type when the gesture is recognized). When the gesture cannot be recognized within a certain time (for example, 500 ms), the gesture recognition unit 120 may transmit a recognition result indicating that the gesture has not been recognized to the control command determination unit 130.

  Note that the gesture recognition method of the gesture recognition unit 120 is not particularly limited as long as the gesture using the remote control 2 can be recognized from the image of the remote control 2. As a gesture recognition method, various conventionally known gesture recognition methods can be used.

  The control command determination unit 130 determines a control command according to the type of IR command received by the IR command reception unit 100 and the result of gesture recognition notified from the gesture recognition unit 120.

  More specifically, when the gesture recognition unit 120 recognizes a gesture corresponding to the IR command, the control command determination unit 130 determines the control command differently from the case where the gesture recognition unit 120 does not recognize the gesture. Furthermore, it is preferable that the control command determination unit 130 determines the control command according to the combination of the IR command and the gesture type. In this case, the control command determination unit 130 refers to the control command determination information in the storage unit 190 to determine a control command corresponding to the IR command and the type of gesture.

  In addition, when the result of gesture recognition was not received (when the gesture recognition unit 120 did not perform gesture recognition), or the result of gesture recognition did not recognize the gesture (the gesture was not performed, or the gesture If it indicates that the recognition has failed), the control command is determined according to the IR command. For example, the control command determination unit 130 may use the control command of the display device 20 indicated by the IR command as it is as the control command.

  In addition, if the control command which the control command determination part 130 determines is a control command concerning TV1, the content of the command will not be specifically limited. For example, the control command determination unit 130 may determine a control command for controlling data acquisition performed by the TV 1 via a communication unit (not shown), or instruct to change the volume of the audio output of the TV 1. A control command may be determined.

  However, the control command determined by the control command determination unit 130 is preferably a display control command for the display 240 of the TV 1 (display device 20). Hereinafter, in this embodiment, the control command is a control command related to display control of the display 240 of the display device 20.

  The control command transmission unit 140 receives the control command determined by the control command determination unit 130 and transmits the control command to the control command reception unit 220 of the display device 20.

  Next, screen transition of the TV 1 (control in the display device 20) according to an input operation on the remote controller 2 and a gesture using the remote controller 2 will be described with reference to FIGS. 4 to 7 are diagrams illustrating examples of input operations and gestures on the remote controller and the TV screen transitions according to the input operations and gestures, respectively.

  For example, as shown in FIG. 4A, it is assumed that the user performs a gesture of shaking the remote controller 2 in the horizontal (left / right) direction while pressing and holding a numeric button of the remote controller 2. In this case, the control command determination unit 130 of the command determination device 10 includes the IR signal indicated by the numeric buttons (8 channels (8ch in the example in the figure)) and the gesture recognized by the gesture recognition unit 120 (the gesture shaken in the horizontal direction). The control command is specified by searching the control command determination information for the combination of the above.

  According to FIG. 3, the control command determined when the remote control 2 is shaken in the horizontal direction while pressing and holding the 8ch button is the “display the program guide of the 8ch broadcasting station” control command. Therefore, the control command determination unit 130 determines the control command as a control command to be transmitted to the display device 20, and the control command transmission unit 140 transmits the control command to the control command reception unit 220 of the display device 20.

  As a result, the display control unit 230 of the display device 20 acquires image data indicating the 8ch broadcast station program guide according to the control command, and causes the display 240 to display the 8ch broadcast station program guide image.

  In the control command determination information, a plurality of types of gestures may correspond to one type of button (IR command). For example, as shown in FIG. 4B, when the user shakes the remote control 2 in the vertical (vertical) direction while pressing and holding the BS button of the remote control 2, the control command determination unit 130 displays the control command determination information. By referring to this, the command “display BS broadcast program guide” is determined as a control command to be transmitted to the display device 20. On the other hand, when the user shakes the remote controller 2 in the horizontal direction while pressing and holding the BS button of the remote controller 2, the control command determination unit 130 transmits a “display BS BS recording list” command to the display device 20. Decide as a command.

  As a result, according to the control command, the display control unit 230 of the display device 20 displays the BS broadcast program list when the remote control 2 is swung up and down, and the BS broadcast recording list when the remote control 2 is swung left and right. It is displayed on the display 240.

  Furthermore, the gesture recognition unit 120 may recognize a plurality of gestures during a period during which gesture recognition is performed. Then, the control command determination unit 130 may sequentially specify the control command corresponding to the IR signal indicated by the long-pressed button and the gesture for each recognized gesture.

  For example, as shown in FIG. 5, it is assumed that a gesture of continuously shaking the remote controller 2 in the same direction (lateral direction) as the button while continuously pressing the button (left and right cursor keys) is performed. In this case, as shown in the figure, the display device 20 of the TV 1 displays on the display 240 an image showing a screen scrolled by one screen in the same direction as the pressed cursor key (left and right direction).

  Also, as shown in the figure, it is assumed that a gesture of continuously shaking the remote controller 2 in the same direction (vertical direction) as the button while continuously pressing the button (up and down cursor keys) is performed. Also in this case, the display device 20 of the TV 1 causes the display 240 to display an image indicating a screen scrolled by one screen in the same direction as the pressed cursor key (up and down direction).

  For example, as shown in FIG. 6A, when the program guide (A1) is displayed on the display 240, the remote controller 2 is shaken up and down while pressing and holding the up and down cursor keys (for example, the down key). When the gesture is performed, the control command determination unit 130 may transmit a command for scrolling the displayed program guide (A1) by one screen in the direction indicated by the cursor key to the display device 20 as a control command ( (A2) is displayed for one screen of the program table in the time zone not currently displayed.

  Similarly, as shown in FIG. 6B, when the program guide (B1) is displayed on the display 240, a gesture of shaking the remote controller 2 left and right while pressing and holding the left and right cursor keys (for example, the right key). Then, the control command determination unit 130 may transmit a command for scrolling the displayed program guide (B1) by one screen in the direction indicated by the cursor key to the display device 20 as a control command ( (One screen (B2) of the program table of a channel not currently displayed may be displayed).

  In addition, as shown in FIG. 7, when the recording list (C1) is displayed on the display 240, when a gesture of shaking the remote controller 2 up and down while holding down the up and down cursor keys (for example, the down key) is performed, The control command determination unit 130 may transmit a command for scrolling the displayed program guide (C1) by one screen in the direction indicated by the cursor key to the display device 20 as a control command (as shown in the figure). The image (C2) on the next page of the list is displayed).

  On a conventional TV, even when the cross key is pressed down while a screen that can scroll (page feed) such as a program guide or recording list is displayed, the cursor in the program guide only moves. It was. Therefore, in order to scroll the screen, it is necessary to continuously move the cursor downward by repeatedly pressing the cursor key of the remote controller 2 or to press a button for scrolling the screen provided separately.

  On the other hand, in the TV 1 according to the present embodiment, as described above, the screen can be scrolled easily by performing a gesture while pressing the cursor key. Thus, the TV 1 can improve user operability over the conventional television. Further, since the above-described buttons for scrolling the screen need not be provided separately on the remote controller 2, the number of buttons on the remote controller 2 can be minimized.

  Further, for example, when the user shakes the remote control 2 while pressing and holding the playback button of the remote control 2 while the TV 1 is reproducing a moving image, the control command determination unit 130 determines the type of the button, the type of gesture, Accordingly, the control command “fast-forward moving image” may be determined as a command to be transmitted to the display device 20, and the display device 20 may fast-forward the display of the moving image on the display 240 in accordance with the control command. Similarly, when the user shakes the remote control 2 to the left while pressing and holding the play button, the control command determination unit 130 may determine that the “rewind video” control command is a control command to be transmitted to the display device 20. Good.

  Furthermore, in the above-described case, when the user performs a gesture of shaking the remote controller 2 left and right a plurality of times, the fast forward or rewind speed may be increased according to the number of times of the gesture.

  Finally, the flow of processing in which the command determination device 10 according to the present embodiment determines a control command will be described with reference to FIG. FIG. 8 is a flowchart showing a flow of processing for determining a control command in the command determination device 10.

  When the user presses a button on the remote controller 2, the remote controller 2 transmits an IR command corresponding to the button. In the following description, it is assumed that the button pressed by the user is “button A”. The light receiving unit 200 of the display device 20 receives the IR signal of the button A transmitted from the remote controller 2 and transmits it to the IR command transmission unit 210. The IR command transmission unit 210 transmits the command indicated by the IR signal of the button A (IR command of the button A) to the IR command reception unit 100.

  The command determination device 10 stands by until the IR command receiving unit 100 receives an IR command (NO in S10). Here, when the IR command receiving unit 100 receives the IR command of the button A (YES in S10), the IR command receiving unit 100 transmits the IR command to the gesture recognition start determining unit 150.

  When the gesture recognition start determination unit 150 receives the IR command for the button A, it first determines whether the button A is a button corresponding to gesture recognition (S12). When the button A corresponds to gesture recognition (YES in S12), the gesture recognition start determination unit 150 further determines whether or not the button A is pressed and held (S14). When the button A is pressed for a long time (YES in S14), the gesture recognition start determination unit 150 causes the gesture recognition unit 120 to start gesture recognition (S16). More specifically, the gesture recognition start determination unit 150 transmits a shooting instruction to the camera 30 and transmits a gesture recognition start instruction to the gesture recognition unit 120. In addition, the gesture recognition start determination unit 150 identifies the gesture corresponding to the button A by referring to the storage unit 190 and notifies the gesture recognition unit 120 of the gesture.

  When button A does not support gesture recognition (NO in S12) or when button A is not pressed for a long time (NO in S14), gesture recognition start determination unit 150 transmits the shooting instruction and start instruction. Instead, the control command determination unit 130 determines the control command corresponding to the IR command, for example, the command itself indicated by the IR command as the control command (S18).

  When the camera 30 receives a shooting instruction from the gesture recognition start determination unit 150, the camera 30 takes a video of the remote controller 2 and transmits the shot video to the image receiving unit 110. The image receiving unit 110 transmits the received video to the gesture recognition unit 120.

  When the gesture recognizing unit 120 receives a start instruction from the gesture recognition start determining unit 150, the gesture recognizing unit 120 recognizes whether or not a gesture corresponding to the button A has been performed in accordance with the start instruction. Note that when the gesture recognition unit 120 recognizes the gesture, the gesture recognition unit 120 also specifies the type of the recognized gesture.

  If the gesture corresponding to the button A can be recognized within a predetermined time (YES in S20), the gesture recognition unit 120 notifies the control command determination unit 130 of the fact that the gesture has been recognized and the type of the gesture. The control command determination unit 130 determines a control command according to the IR command of the button A received from the IR command reception unit 100 and the type of gesture recognized by the gesture recognition unit 120 while referring to the control command determination information. (S22). Thereafter, the gesture recognition unit 120 and the control command determination unit 130 perform S20 to S20 while the IR command reception unit 100 continues to receive the IR command, that is, while the user holds down the button A (YES in S26). 22 is repeated.

  On the other hand, if the gesture recognition unit 120 cannot recognize the gesture corresponding to the button A within a predetermined time (NO in S20), the gesture recognition unit 120 notifies the control command determination unit 130 that the gesture has not been recognized. Then, the control command determining unit 130 determines a control command corresponding to the IR command, for example, an instruction itself indicated by the IR command as a control command (S24).

  When the control command is determined in S22, or when the control command is determined in S24 and the IR command is not accepted (the user stops pressing the button A) (NO in S26), the gesture recognition start determination unit 150 performs the gesture. A gesture recognition end instruction is transmitted to the recognition unit 120, and the gesture recognition unit 120 ends the gesture recognition according to the end instruction (S28).

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIG. For convenience of explanation, members having the same functions as those described in the embodiment are given the same reference numerals, and descriptions thereof are omitted.

  The gesture recognition start determination unit 150 of the command determination device 10 according to the present invention temporarily receives an IR command related to the same button, which is received by the IR command receiving unit 100 when the gesture recognition unit 120 is performing gesture recognition. If the interrupted time is less than or equal to a certain time (second time), it may be considered that the IR command is received without interruption (even if the gesture recognition by the gesture recognition unit 120 is continued). Good).

  Hereinafter, the flow of processing in the command determination apparatus 10 according to the second embodiment will be described with reference to FIG. In FIG. 9, the same processes as in FIG.

  FIG. 9 is a flowchart showing the flow of processing in the command determination device 10 according to the second embodiment of the present invention. FIG. 9 differs from the flowchart shown in FIG. 8 in that it includes steps S50 to S62.

  The command determination device 10 according to the present embodiment includes a key buffer (not shown) that temporarily stores the type of button indicated by the IR command received by the IR command receiving unit 100. Then, when the button A is pressed for a long time (YES in S14), the gesture recognition start determination unit 150 stores information indicating the pressed button (button A) in the key buffer (S50) and gesture recognition. The unit 120 starts gesture recognition (S16).

  In addition, when the control command is determined in S22 and the reception of the IR command is interrupted (NO in S26), the gesture recognition start determination unit 150 takes a fixed time (second time) waiting (WAIT) time (S52). . When the IR command reception unit 100 receives the IR command again during WAIT (YES in S54), the gesture recognition start determination unit 150 identifies the type of button indicated by the received IR command. In the following description, the button indicated by the IR command received again is referred to as button B. The gesture recognition start determination unit 150 stores information indicating the identified button (button B) in the key buffer (S56). Next, the gesture recognition start determination unit 150 compares information indicating the buttons stored in the key buffer in S50 and S56. Specifically, the gesture recognition start determination unit 150 determines whether or not the button A and the button B are the same button (S58).

  When button A = button B (YES in S58), the gesture recognition start determination unit 150 regards the IR command related to button A as being received without interruption by the IR command receiving unit 100, and the gesture recognition unit 120 recognizes the gesture. To continue. That is, the gesture recognition unit 120 continues the processing from S60.

  On the other hand, when the button A and the button B are different buttons (NO in S58), the gesture recognition start determination unit 150 regards that the long press of the button A has ended, and the control command determination unit 130 determines the IR command related to the button B. A control command is determined according to (IR command received again) (S60). In addition, the gesture recognition start determination unit 150 transmits a gesture recognition end instruction to the gesture recognition unit 120, and the gesture recognition unit 120 ends the gesture recognition according to the end instruction (S28). Finally, the gesture recognition start determination unit 150 clears the key buffer storing the information indicating the buttons A and B (S62).

  Note that the WAIT time of the gesture recognition start determination unit 150 is arbitrarily set (for example, about 500 ms) within a time that is estimated to be required for the user to perform one (one type) gesture, for example. be able to. It can be estimated that the reason why the acceptance of the IR command is interrupted (NO in S26) occurs, for example, when the user moves the remote controller 2 too much when gesturing and the light receiving unit 200 cannot receive the IR signal. It is.

[Embodiment 3]
In addition, when the IR command receiving unit 100 receives an IR command, the command determining device according to the present invention displays a gesture that can be combined with the IR command on at least one of a character and an image on the display 240 of the display device 20. You may let them.

  Hereinafter, the command determination device 40 (information processing device) according to the third embodiment will be described with reference to FIGS. FIG. 10 is a block diagram illustrating a main configuration of the TV 3 including the command determination device 40 according to the present embodiment.

  The command determination device 40 is different from the command determination device 10 in that it includes a guidance display command section (guidance display means) 160. When the IR command receiving unit 100 receives the IR command for a certain time (first time) (the user presses and holds the button on the remote controller 2), that is, the gesture recognizing unit 120 recognizes the gesture. When started, the gesture corresponding to the accepted IR command (button type) is identified by referring to the control command determination information in the storage unit 190, and the character or image (gesture guidance) indicating the gesture is displayed. A control command (guidance display command) to be displayed on 240 is transmitted to control command transmission unit 140.

  The control command transmission unit 140 transmits the guidance display command received from the guidance display command unit 160 to the control command reception unit 220, and the control command reception unit 220 transmits the guidance display command to the display control unit 230. Then, the display control unit 230 causes the display 240 to display the gesture guidance corresponding to the button type in accordance with the guidance display command.

  More specifically, the guidance display command changes the display (arrangement and size, etc.) of at least some objects included in the image displayed on the display 240 according to the control command, or adds a new object to the image. Command to be displayed.

  (A)-(d) of FIG. 11 is a figure which shows an example of the screen which TV3 displays. In FIG. 11, an object indicated by an arrow indicates gesture guidance. For example, as shown in FIGS. 11A to 11C, the guidance display command unit 160 displays the gesture guidance (arrow) on the display 240 so as to be superimposed on the image currently displayed on the display 240. The guidance display command to instruct may be transmitted to the control command transmission unit 140.

  Further, the behavior of the TV 1 (control indicated by the control command) when the gesture is performed according to the guidance may be described in the vicinity of the gesture guidance (arrow object) in FIG. Further, when any of the gestures is performed while the gesture guidance is displayed on the display 240, the guidance display command unit 160 changes the display mode of the gesture guidance (the color, size, and size of the arrow object). Or the shape may change, blink, or the arrow may disappear).

  Further, for example, as shown in FIG. 11D, the guidance display command unit 160 reduces the image currently displayed on the display 240 on the display 240 and displays the gesture guidance in the remaining space. The guidance display command to instruct may be transmitted to the control command transmission unit 140.

  When the gesture recognition unit 120 starts the gesture recognition, the guidance display command unit 160 displays a command for displaying at least one of a character and an image indicating that the gesture recognition has started on the display 240 (start gesture recognition start). Notification command) may be transmitted to the control command transmission unit 140.

  In addition, when the gesture recognition unit 120 ends the gesture recognition, the guidance display command unit 160 issues a command (gesture recognition end notification command) for displaying at least one of a character and an image indicating the end on the display 240. You may transmit to the control command transmission part 140. FIG.

  For example, the guidance display command unit 160 transmits a “display red object on display 240” command as a gesture recognition start notification command to the control command transmission unit 140, and a “display red object display” command as a gesture recognition end notification command. The “stop” command may be transmitted to the control command transmission unit 140.

  In addition, the guidance display command unit 160 issues a command “make the peripheral color of the image displayed on the display 240 blue (enclose the image in a blue frame)” as a gesture recognition start notification command. May be sent to.

  Further, the guidance display command unit 160 may transmit a “change the color of the background image displayed on the display 240” command to the control command transmission unit 140 as a gesture recognition start command. The “background image” here refers to an image displayed on the back side of an object (window, icon, etc.) among images displayed on the display 240.

  Note that when the above-described gesture recognition start notification command (and gesture recognition end notification command) is transmitted to the control command transmission unit 140, the guidance display command unit 160 may transmit the guidance display command to the control command transmission unit 140. However, it does not have to be transmitted. Furthermore, the guidance display command unit 160 does not have to specify a gesture corresponding to the IR command (button type) when the guidance display command is not transmitted.

  Furthermore, the guidance display command, the gesture recognition start notification command, and the gesture recognition end notification command receive voice (melody, sound effect, guidance voice, etc.) from an output unit (not shown) instead of displaying characters or images on the display 240. It may be a command to be output.

  For example, the guidance display command unit 160 may transmit a command “output a voice guidance of a gesture corresponding to the accepted IR command” to the control command transmission unit 140 as a guidance display command. Then, the control command transmission unit 140 transmits the guidance display command to the output unit (or an output control unit that controls the output unit), and causes the output control unit to output the voice guidance according to the instruction of the guidance display command. Also good.

  In addition, the guidance display command unit 160 transmits a command to “output the melody set by the user from the output unit every predetermined time” to the control command transmission unit 140 as a gesture recognition start notification command, and “ A command to stop the reproduction of the melody may be transmitted to the control command transmission unit 140.

  Further, the guidance display command unit 160 outputs, as a gesture recognition end notification command, a sound effect (warning sound) indicating that the gesture recognition is about to end immediately before the gesture recognition ends (for example, 5 seconds before). May be transmitted to the control command transmission unit 140.

  In addition, the guidance display command unit 160 outputs a short high tone when the gesture recognition starts, and continues to receive the IR command at a predetermined time interval while the gesture recognition continues (that is, the IR command reception unit 100 continues to receive the IR command at a predetermined time interval). A command for instructing the output unit to continuously output a low tone every predetermined time (for example, 1 second) may be transmitted to the control command transmission unit 140.

[Modification]
In each of the above embodiments, the command determination device 10 or 40 according to the present invention is a device built in the TV 1 or 3. However, the command determination device 10 or 40 may be a device provided separately from the TV 1 or 3. For example, the command determination device 10 or 40 is an external device attached to the TV 1 and may be connected to the display device 20 and the camera 30 by wire or wirelessly.

  In addition, the TV 1 according to each of the above embodiments may be any electronic device as long as it can be operated using a remote controller such as a video or a personal computer. The remote controller 2 may be a portable information device such as a smartphone having a remote control function for operating an electronic device.

[Example of software implementation]
The command determination devices 10 and 40 and the control blocks of the display device 20 (particularly the control blocks of the command determination devices 10 and 40) are realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. Alternatively, it may be realized by software using a CPU (Central Processing Unit).

  In the latter case, the command determination devices 10 and 40 and the display device 20 are recorded such that a CPU that executes instructions of a program that is software that realizes each function, the program and various data can be read by a computer (or CPU). In addition, a ROM (Read Only Memory) or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Summary]
The information processing apparatus (command determining apparatus 10 or 40) according to the aspect 1 of the present invention has a command corresponding to the type of input operation performed on the operation apparatus (remote control 2) for the display apparatus (display apparatus 20). (IR command) receiving command (IR command receiving unit 100), and when the command receiving means receives the command, from the video of the operating device taken by the camera (camera 30), A gesture recognition unit (gesture recognition unit 120) for recognizing a predetermined movement (gesture), and the gesture recognition unit when the gesture recognition unit recognizes the predetermined movement and the gesture recognition unit does not recognize the predetermined movement. Control instruction determining means for determining a control instruction (control command) corresponding to the command, which is different from the control instruction And a control command determining unit 130), the control command instruction means for executing the control command to the display device (control command transmitting unit 140), the.

  According to the above configuration, the gesture recognizing unit recognizes the predetermined movement as a gesture when the user moves the controller device with the predetermined movement when receiving a command corresponding to the type of the input operation. Here, the timing at which the command receiving means receives the command can be regarded as substantially simultaneous with the input operation to the controller device. Therefore, the gesture recognizing unit can recognize the gesture performed when the input operation to the operation device is performed.

  Then, when the gesture recognition unit recognizes the gesture, the control command determination unit can determine the control command different from the control command when the gesture is not recognized. In other words, the control command determination means, in addition to the input operation to the controller device, when the user performs only the input operation (when the input operation is performed but no gesture is performed), The control command can be determined by distinguishing from the case where the gesture is performed.

  Therefore, even if the types of input operations are limited on the operating device (even if the types of members such as buttons and switches that accept input operations are limited), the control command determination means combines the input operations and gestures. By determining the control command, various types of operation instructions can be easily performed on the display device.

  In the above configuration, the controller device only needs to be configured to transmit a command corresponding to the input operation. Therefore, the information processing apparatus can obtain the above-described effects by using a general-purpose remote controller or the like without using a dedicated operation device for the display device.

  The information processing apparatus according to aspect 2 of the present invention is the information processing apparatus according to aspect 1, wherein the gesture recognition unit specifies the recognized type of the predetermined movement, and the control instruction determination unit includes the command and the predetermined movement. The control command may be determined according to the combination with the type.

  According to the above configuration, the control command determination unit can further determine the control command according to the type of gesture recognized by the gesture recognition unit. Therefore, the control command determining means can assign more operation instructions to one member that receives an input operation in the operating device.

  The information processing apparatus according to aspect 3 of the present invention is the information processing apparatus according to aspect 1 or 2, wherein the gesture recognition unit is the same for a time equal to or longer than a predetermined first time after the command reception unit receives the command. When the command is continuously received, the predetermined movement may be recognized.

  According to the above configuration, the gesture recognition unit continuously inputs the same input for a certain time (first time) on the operation device even if a member for instructing the start of gesture recognition is not provided on the operation device. Gesture recognition can be performed when an operation is performed.

  Note that the first time is related to the input operation when a normal input operation (an input operation that the user does not intend to combine a gesture (such as a short press of a button on the operation device)) is performed on the operation device. It can be arbitrarily set to a time longer than the time (for example, a time that can be estimated to be a long press on the button of the operating device) or longer.

  By setting the first time in this way, when the gesture recognition unit of the information processing apparatus receives the same command (the same input operation) for the time equal to or longer than the first time, the user performs the input operation. It is possible to perform gesture recognition assuming that the user intends to perform a gesture after.

  The information processing apparatus according to aspect 4 of the present invention is the information processing apparatus according to any one of the aspects 1 to 3, wherein the gesture recognizing unit performs the predetermined process while the command receiving unit continuously receives the same command. Motion recognition may be performed.

  According to the above configuration, the gesture recognition unit recognizes a gesture while the same input operation is continuously performed on the operation device. Therefore, the gesture recognition means can determine the start and end timing of gesture recognition from the input operation even if a member for instructing start and end of gesture recognition is not provided on the remote controller.

  In the information processing apparatus (command determining apparatus 40) according to aspect 5 of the present invention, in the above aspect 4, the gesture recognizing unit is configured such that when the same command received by the command receiving unit is temporarily interrupted, When the interrupted time is equal to or shorter than a predetermined second time, it may be considered that the same command is received without interruption.

  According to the above configuration, the gesture recognizing unit outputs the command for some reason, for example, when the command receiving unit cannot receive a command from the operating device due to the user moving the operating device too much during the gesture. Even if reception of a command is temporarily interrupted without being accepted, if the same command can be received again within the second time, it is not determined that the command has been interrupted, and gesture recognition can be continued.

  Therefore, the gesture recognizing unit can prevent the gesture recognition from ending even though the user does not intend.

  Note that the second time can be arbitrarily set to a time within a time estimated to be required for the user to perform one (one type) gesture, for example.

  The information processing apparatus according to Aspect 6 of the present invention is the information processing apparatus according to any one of Aspects 1 to 5, wherein when the command receiving means receives the command, the character and the image showing the predetermined movement that can be combined with the command Guidance display means (guidance display command section 160) for displaying at least one of the above on the display device may be provided.

  According to the above configuration, when the input operation is performed on the operation device, the guidance display unit can be combined with the input operation (command corresponding to the input operation) when the control command determination unit determines the control command. Is presented to the user using at least one of characters and images.

  Thereby, the user can confirm the gesture performed with input operation with a display apparatus. Therefore, the guidance display means can improve the convenience when the user performs a gesture.

  The display device (display device 20) according to aspect 7 of the present invention is a light receiving unit (light receiving unit 200) with respect to the information processing device (command determining device 10) according to any one of the above aspects 1 to 6. Command transmission means (IR command transmission unit 210) for transmitting a command (IR command) corresponding to an infrared signal received from the operation device (remote control 2), and the control command (IR command transmission unit 210) as a response to the command. A control command receiving unit (control command receiving unit 220) for receiving a control command) and a display control unit (display control unit 230) for displaying an image on the display unit (display 240) according to the control command received by the control command receiving unit. And).

  According to the said structure, a display apparatus has an effect similar to the said information processing apparatus.

  A television (TV1 or TV3) according to aspect 8 of the present invention includes an information processing apparatus (command determination apparatus 10) according to any one of aspects 1 to 6 and a display apparatus (display apparatus) according to aspect 7. 20) and the camera (camera 30).

  According to the above configuration, the television has the same effect as the information processing apparatus.

  The information processing apparatus and the display apparatus according to each aspect of the present invention may be realized by a computer. In this case, the information processing apparatus is operated by causing the computer to operate as each unit included in the information processing apparatus and the display apparatus. An information processing apparatus that implements a display device on a computer, a control program for the display device, and a computer-readable recording medium that records the information storage device also fall within the scope of the present invention.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  The present invention can be suitably used for an electronic device that includes a display unit, such as a television, a video, and a personal computer, and that can be operated using an operation device.

1, 3 TV (TV)
2 Remote control (operating device)
10, 40 Command determining device (information processing device)
100 IR command receiving unit (command receiving means)
110 Image receiving unit 120 Gesture recognition unit (gesture recognition means)
130 Control command determining unit (control command determining means)
140 Control command transmitter (control command instruction means)
150 Gesture recognition start determination unit 160 Guidance display command unit (guidance display means)
190 Storage unit 20 Display device 200 Light receiving unit 210 IR command transmission unit (command transmission means)
220 Control command receiver (control command receiver)
230 Display control unit (display control means)
240 Display (display unit)
30 cameras

Claims (10)

Command receiving means for receiving a command corresponding to the type of input operation performed on the operating device for the display device;
Gesture recognition means for recognizing a predetermined movement of the operating device from the video of the operating device taken by the camera when the command receiving means receives the command;
When the gesture recognition means recognizes the predetermined movement, a control command for determining a control instruction corresponding to the command is different from a control instruction when the gesture recognition means does not recognize the predetermined movement. A determination means;
An information processing apparatus comprising: control command instruction means for causing the display device to execute the control command. The gesture recognition means identifies the type of the predetermined movement recognized,
The information processing apparatus according to claim 1, wherein the control command determining unit determines the control command according to a combination of the command and the predetermined type of motion.   The gesture recognition means recognizes the predetermined movement when the same command is continuously received for a time equal to or longer than a predetermined first time after the command reception means receives the command. The information processing apparatus according to claim 1, wherein:   4. The gesture recognition unit according to claim 1, wherein the gesture recognition unit recognizes the predetermined movement while the command reception unit continuously receives the same command. 5. Information processing device.   The gesture recognition means, when the same command received by the command reception means is temporarily interrupted, if the interrupted time is less than or equal to a predetermined second time, the gesture recognition means The information processing apparatus according to claim 4, wherein the information processing apparatus is regarded as having been received without interruption.   The guidance display means for displaying on the display device at least one of a character and an image showing the predetermined movement that can be combined with the command when the command receiving means receives the command. The information processing apparatus according to any one of 1 to 5. Command transmitting means for transmitting a command corresponding to an infrared signal received from the operating device at the light receiving unit to the information processing apparatus according to claim 1,
Control command receiving means for receiving the control command from the information processing apparatus as a response to the command;
And a display control means for displaying an image on a display unit in accordance with the control command received by the control command receiving means. The information processing apparatus according to any one of claims 1 to 6,
A display device according to claim 7;
A television comprising the camera.   A control program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 6, wherein the control program causes the computer to function as each of the means.   A computer-readable recording medium on which the control program according to claim 9 is recorded.

Images