JP2009087095A - Electronic device control device, control method and control program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve operability by displaying an operating button for an object to be recognized such as a user's hand at a proper position in an electronic device configured to operate the operating button by holding the hand over the operating button displayed on a display device. <P>SOLUTION: A recognition information generating part 23 recognizes a part of the body of a user 3 photographed by a video camera 2 or an object held by the user 3, and generates recognition object information including the attributes and number of recognition objects, and the positions in a screen, etc. An image photographed by the video camera 2 is converted into a mirror image, and enlarged, reduced or adjusted in position according to the recognition object information while generating an operation image. The display position or range of the operating button is adjusted, and combined images are displayed on a screen of a display device 21. Predetermined selecting operation performed by an operator displayed on the screen, using the operating button, is detected to perform control operation corresponding to the operating button. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a control device, control method, and control program for an electronic device for remotely operating an electronic device having a display device such as a television receiver or a personal computer.

  User interfaces for controlling home appliances such as television receivers at hand have become widespread and have greatly changed the way home appliances are used. At present, this type of operation is still mainstream, but the remote control is based on a mechanism that executes one function with one push. For example, in a television receiver, “power”, “channel”, “volume”, “input switching”. Such a key corresponds to this, and it has been a very convenient remote operation method for a conventional television receiver.

  By the way, recently started data broadcasting requires pressing the “up / down / left / right” and “decision” keys of the remote controller many times in order to select a desired menu screen, which is cumbersome and difficult to use. Further, EPG (Electronic Program Guide) has a similar problem because it selects a desired position from a guide screen arranged in a matrix and presses a key.

Japanese Patent Application Laid-Open No. 2006-091948 (Patent Document 1), which is a prior application by the present applicant, provides both flexibility for various electronic devices and convenience of remote operation, and also provides remote operation devices such as a remote controller. Electronic devices that do not require use at hand have been proposed. The electronic device described in Patent Literature 1 captures the operation of a user's hand with a video camera and identifies the operation content for the operation button displayed on the display device from the operation.
JP 2003-283866 A

  However, in the electronic apparatus described in Patent Document 1 described above, the behavior of the object with respect to the operation button, such as recognizing an object such as a user's hand and holding the hand on the operation button by the user displayed on the display device. However, for example, the display position of the operation button is not appropriate for the recognized position of the user's hand, and the operation button may be difficult to operate.

  The present invention has been made paying attention to this point, and in an electronic apparatus that operates an operation button by holding the hand or the like over the operation button displayed on the display device, the operation button for the recognition target object such as a user's hand is provided. An object of the present invention is to provide a control device, control method, and control program for an electronic device that can make the display position appropriate and improve the operability.

  In order to solve the above-mentioned problems, the invention according to claim 1 is a control device for an electronic apparatus having a display device, and includes a video camera for photographing an operator and the body of the operator photographed by the video camera. Of the object or the object held by the operator, the attribute of the recognized recognition object, the number, the position of the display device on the screen, or the ratio of the area occupied in the screen A recognition information generation unit that generates recognition object information including at least one, and an image captured by the video camera is mirror-image converted, and the recognition object displayed on the screen according to the recognition object information A mirror image adjustment unit for enlarging, reducing, or adjusting the position of an object and an operation image including at least one operation button are generated, and the display position of the operation button is determined according to the recognition object information Is an image signal mixing unit that forms an image by mixing the operation image generation unit that adjusts the range, the image that has passed through the mirror image adjustment unit, and the operation image, and displays the mixed image on the screen. And an image captured by the video camera and displayed on the screen in a state where the image mixed by the image signal mixing unit is displayed on the screen, the operator operates the operation using the recognition object. An electronic apparatus comprising: an operation detection unit that detects a predetermined selection operation for selecting a button; and an appliance control unit that performs a control operation corresponding to the operation button for which the predetermined selection operation is detected by the operation detection unit. It is a control device of the equipment.

  According to such an invention, a mixed image obtained by mixing the operator's image and the operation image including the operation button is displayed, and the operation for the operation button of the operator in the screen is detected to control the electronic device. The size and position of the operator on the screen and the display position and range of the operation buttons are determined according to the attribute, number of objects, position in the measurement area, or the ratio of the area to the predetermined area. Since the adjustment is performed, the operation buttons can be displayed at an appropriate position and range in accordance with the display position of the hand or face of the operator who performs the operation (operation), and the operability can be improved.

  According to a second aspect of the present invention, in the first aspect of the invention, the operation image generation unit is configured such that the attribute of the recognition target included in the recognition target information is an operator's hand, and the recognition target The operation buttons are gathered and arranged around the recognition object displayed on the screen, the attribute of the recognition object is an operator's hand, and the recognition object In the case where the number of objects is two or more, the operation buttons are distributed and arranged around each recognition object displayed on the screen.

  In this case, when the operator's hand is used as the recognition target, the number of the buttons is determined, so that the operation buttons are displayed in a concentrated manner around one hand on the assumption that the operation is performed with one hand. The operation buttons can be distributed and displayed on both hands on the premise of the above operation, and an interface according to the preference of the operator can be provided.

  According to a third aspect of the present invention, in the first aspect of the invention, the operation image generating unit may be displayed on the screen when the attribute of the recognition target included in the recognition target information is an operator's face. The operation buttons are arranged around the displayed recognition object.

  In this case, it is possible to improve the accuracy of the recognition operation by using a face that is more reliable than the hand as the recognition target, and at the periphery of the face (concentrated on either the left, right, top or bottom, distributed on the periphery or both sides) By displaying the operation buttons on the screen, the operator can operate the operation buttons with an operation of a minimum necessary size, and the operability can be improved while improving the accuracy of the operation.

  In this face recognition method, for example, a plurality of faces may be recognized, and operation buttons may be displayed around the recognized faces. In this case, even when multiple people are watching, each person can operate the operation buttons around them, so that each person operates with the minimum necessary operation. be able to.

  The invention according to claim 4 is the invention according to any one of claims 1 to 3, wherein the operation image generation unit performs the operation according to the attribute and the number of recognition objects included in the recognition object information. The display position of the button is switched.

  In this case, by using multiple types of recognition objects such as hands and faces, priority is given to objects that can be recognized more reliably, or arbitrary recognition objects are selected by user operations. The operation environment is appropriately switched by performing the operation of switching the recognition object as appropriate, and switching the operation buttons as appropriate according to the type of the recognized recognition object. And can respond appropriately to user preferences.

  The invention according to claim 5 is the invention according to any one of claims 1 to 4, wherein the operation buttons are grouped according to their attributes, and the operation buttons are displayed in groups on the operation screen. In addition, activation or deactivation is switched in units of groups according to an operator's selection operation, and the display range of the activated group includes an image that has passed through the mirror image adjustment unit, and the operation For the inactivated group mixed with the image for use, at least information on the attribute of the group is displayed in the display range of the group.

  In this case, for example, since the operation buttons are grouped according to the broadcasting system and input type such as terrestrial broadcasting, satellite broadcasting (BS, CS, etc.), input terminals, etc., the user can access the operation buttons desired by the user. Becomes easy. Further, when displaying the grouping, the operable group is activated (activated), and the operator image is mixed there. Therefore, the operator can intuitively grasp the operable buttons. On the other hand, other groups are deactivated (deactivated), but at least the attribute information of the deactivated group is displayed, so the operator can select other options even if the current operation cannot be performed. It is possible to intuitively grasp that there is a group.

  The invention according to claim 6 is the invention according to any one of claims 1 to 5, wherein the operation image generation unit displays information related to an operation in the form of a graphic or a character based on the detection of the selection operation by the operation detection unit. It has the function to produce | generate the guidance icon displayed by this.

  In this case, when an operation on the operation button of the operator is detected, information (guidance and hints) on the next operation (motion) can be displayed, so that the operator can smoothly perform the operation. And operability can be improved.

  The invention according to claim 7 is a method for controlling an electronic apparatus having a video camera for photographing an operator and a display device, wherein the body of the operator photographed by the video camera by a recognition information generation unit. Recognizing a target object that is a part or an object held by an operator, and the attribute of the recognized recognition target object, the number, the position in the screen of the display device, or the ratio of the area occupied in the screen A recognition information generation step for generating recognition object information including at least one, and a mirror image adjustment unit that performs mirror image conversion on an image captured by the video camera and displays the image on the screen according to the recognition object information An operation image including at least one operation button is generated by the mirror image adjustment step for enlarging, reducing, or adjusting the position of the recognized recognition object, and the operation image generator. The operation image generation step for adjusting the display position or range of the operation button according to the recognition object information, the image that has passed through the mirror image adjustment unit, and the operation image are mixed in the image signal mixing unit. Then, an image signal mixing step of forming a mixed image and displaying the mixed image on the screen, and an image mixed by the image signal mixing unit being displayed on the screen and photographed by the video camera An operation detection step in which an operation displayed on the screen detects a predetermined selection operation by which the operator selects the operation button using the recognition target object by an operation detection unit; and the predetermined selection operation by the operation detection unit. And a device control step of performing a control operation corresponding to the operation button detected by the device control unit.

  According to such an invention, a mixed image obtained by mixing the operator's image and the operation image including the operation button is displayed, and the operation for the operation button of the operator in the screen is detected to control the electronic device. The size and position of the operator on the screen and the display position and range of the operation buttons are determined according to the attribute, number of objects, position in the measurement area, or the ratio of the area to the predetermined area. Since the adjustment is performed, the operation buttons can be displayed at an appropriate position and range in accordance with the display position of the hand or face of the operator who performs the operation (operation), and the operability can be improved.

  The invention according to claim 8 is the invention according to claim 7, wherein, in the image generation step, the operation image generation unit is configured such that the attribute of the recognition object included in the recognition object information is an operator's hand. When the number of the recognition objects is one, the operation buttons are gathered and arranged around the recognition objects displayed on the screen, and the attributes of the recognition objects are determined by the operator's hand. And the number of the recognition objects is two or more, the operation buttons are distributed around the recognition objects displayed on the screen.

  In this case, when the operator's hand is used as the recognition target, the number of the buttons is determined, so that the operation buttons are displayed in a concentrated manner around one hand on the assumption that the operation is performed with one hand. The operation buttons can be distributed and displayed on both hands on the premise of the above operation, and an interface according to the preference of the operator can be provided.

  The invention according to claim 9 is the invention according to claim 7, wherein, in the image generation step, the operation image generation unit is configured such that the attribute of the recognition object included in the recognition object information is an operator's face. In addition, the operation buttons are arranged around the recognition object displayed on the screen.

  In this case, it is possible to improve the accuracy of the recognition operation by using a face that is more reliable than the hand as the recognition target, and at the periphery of the face (concentrated on either the left, right, top or bottom, distributed on the periphery or both sides, etc.) By displaying the operation buttons on the screen, the operator can operate the operation buttons with an operation of a minimum necessary size, and the operability can be improved while improving the accuracy of the operation.

  In this face recognition method, for example, a plurality of faces may be recognized, and operation buttons may be displayed around the recognized faces. In this case, even when multiple people are watching, each person can operate the operation buttons around them, so that each person operates with the minimum necessary operation. be able to.

  According to a tenth aspect of the present invention, in the invention according to any of the seventh to ninth aspects, in the image generation step, the operation image generation unit sets the attribute and number of recognition objects included in the recognition object information. In response, the display position of the operation button is switched.

  In this case, by using multiple types of recognition objects such as hands and faces, priority is given to objects that can be recognized more reliably, or arbitrary recognition objects are selected by user operations. The operation environment is appropriately switched by performing the operation of switching the recognition object as appropriate, and switching the operation buttons as appropriate according to the type of the recognized recognition object. And can respond appropriately to user preferences.

  The invention according to claim 11 is the invention according to any one of claims 7 to 10, wherein the operation buttons generated in the operation image generation step are grouped according to their attributes, and the operation image generation In the operation screen generated in step, the operation buttons are displayed in units of groups, and activated or deactivated are switched in units of groups according to an operator's selection operation. In the image signal mixing step The display range of the activated group is mixed with the image that has passed through the mirror image adjustment unit and the operation image, and at least the attribute of the deactivated group is related to the group Information is displayed within the display range of the group.

  In this case, for example, since the operation buttons are grouped according to the broadcasting system and input type such as terrestrial broadcasting, satellite broadcasting (BS, CS, etc.), input terminals, etc., the user can access the operation buttons desired by the user. Becomes easy. Further, when displaying the grouping, the operable group is activated (activated), and the operator image is mixed there. Therefore, the operator can intuitively grasp the operable buttons. On the other hand, other groups are deactivated (deactivated), but at least the attribute information of the deactivated group is displayed, so the operator can select other options even if the current operation cannot be performed. It is possible to intuitively grasp that there is a group.

  The invention according to a twelfth aspect is the invention according to any one of the seventh to eleventh aspects, wherein in the operation detection step, the operation image generation unit relates to an operation based on the detection of the selection operation by the operation detection unit. It has a function of generating a guidance icon for displaying information in graphics or characters.

  In this case, when an operation on the operation button of the operator is detected, information (guidance and hints) on the next operation (motion) can be displayed, so that the operator can smoothly perform the operation. And operability can be improved.

  The invention according to claim 13 is a control program that causes a computer to control an electronic device having a video camera that captures an operator and a display device, and the computer uses the recognition information generation unit to execute the control. Recognizing an object which is a part of the operator's body or an object held by the operator, which is photographed by a video camera, and the attributes and number of the recognized objects to be recognized, positions on the screen of the display device, Or a recognition information generation step for generating recognition target object information including at least one of the proportion of the area in the screen, and a mirror image adjustment unit that converts the image captured by the video camera into a mirror image, and the recognition A mirror image adjustment step for enlarging, reducing, or adjusting the position of the recognition object displayed on the screen according to the object information; An operation image generation step of generating an operation image including at least one operation button by the generation unit, and adjusting a display position or range of the operation button according to the recognition object information; and the mirror image adjustment unit And the operation image are mixed in the image signal mixing unit to form a mixed image, and the mixed image is displayed on the screen and mixed by the image signal mixing unit. In a state in which an image is displayed on the screen, an operation detection unit performs a predetermined selection operation in which an operator, which is captured by the video camera and displayed on the screen, selects the operation button using the recognition target object The device control unit performs an operation detection step detected by the control unit and a control operation corresponding to the operation button for which the predetermined selection operation is detected by the operation detection unit. Characterized in that to execute a vessel control step.

  According to such an invention, the above-described apparatus and method can be implemented using a general-purpose computer or a dedicated computer, and the program can be recorded by recording the program on a computer-readable recording medium. Storage, transportation and installation can be performed easily.

  According to the present invention, in an electronic device that operates an operation button by holding the hand over the operation button displayed on the display device, the display position of the operation button with respect to the recognition target object such as the user's hand is appropriate, Operability can be improved.

(Outline of electronic device control method)
FIG. 1 is an explanatory diagram illustrating a method for controlling an electronic apparatus (television receiver 1) in the present embodiment.

  In the operation method of the present embodiment, the television receiver 1 is provided with the video camera 2, the user 3 is photographed by the video camera 2, the operation of the user 3 is recognized from the image of the video camera 2, and the recognition is performed. The television receiver 1 and related devices are operated in accordance with the performed operation. Here, the operation of the user 3 is specifically a movement of the body (hand, foot, face, etc.) for selecting a desired button from the menu screen displayed on the television receiver 1. In the present embodiment, as will be described later, the most practical method of operating with hand movement is adopted. Further, when the periphery of the television receiver 1 is dark, the same effect as that of the hand movement may be obtained by using a remote operation device (hand control device) having a self-luminous function.

  2 and 3 are explanatory diagrams showing screen configurations of the television receiver 1. As shown in the figure, the screen according to the present embodiment displays the second mixed image M2o generated by mixing the first mixed image M1o on the normal video signal S2. The first mixed image M1o is formed by mixing the operator image S1 captured by the video camera 2 with the operation image Gs which is a user interface. Here, the normal video signal S2 is a video signal from, for example, a terrestrial broadcast or a satellite broadcast, a video deck or a DVD player input from an external input terminal, and an operator image was taken by the video camera 2. It is the user 3 and the scenery behind it.

  In the operation image Gs, operation buttons 40 and 41 for operating a television channel, page transition, and media (broadcast system) switching are arranged. In this screen, the operator holds the hand that is the recognition object Obj over an arbitrary operation button (an arbitrary broadcast station or the like) in the operation buttons 40 and 41 and performs a predetermined operation (for example, bending a finger). The user can perform an operation on a desired operation button by signing the hand to open or close.

(Configuration of control device)
FIG. 4 is a block diagram showing the configuration of the control device of the television receiver 1. The television receiver 1 includes a video camera 2, a reference synchronization generator 11, a timing pulse generator 12, a mirror image converter 13, and a scaler. 14, a graphics generator 15, a first mixer 16, a detection unit 17, a control information determination device (CPU) 18, a pixel number converter 19, a second mixer 20, and a display device 21. The “module” used in the description refers to a functional unit that is configured by hardware such as an apparatus or a device, software having the function, or a combination thereof, and achieves a predetermined operation. .

  The reference synchronization generator 11 is a module that generates a horizontal period pulse and a vertical period pulse that serve as a reference for the television receiver 1 based on an input synchronization signal. Here, the synchronization signal is a horizontal and vertical synchronization signal of the video signal input to the pixel number converter 19. When receiving a television broadcast or when a video signal is input from an external device, the reference synchronization generator 11 generates a pulse synchronized with the synchronization signal of the input signal. The timing pulse generator 12 generates a pulse having an arbitrary phase and width in the horizontal and vertical directions required by each module.

  As shown in FIG. 1, the video camera 2 is a device that is located in front of the television receiver 1 and shoots an image in front of the user 3 or the television receiver 1, and employs a CCD camera, for example. Can do. The output signal of the video camera 2 is a luminance (Y) signal and a color difference (R−Y, B−Y) signal, and is synchronized with the horizontal period pulse and the vertical period pulse output from the reference synchronization generator 11. .

  The mirror image converter 13 and the scaler 14 perform mirror image conversion on the image signal captured by the video camera, and enlarge, reduce, or position of the operator displayed on the screen of the display device according to recognition object information described later. It is a module which comprises the mirror image adjustment part which performs adjustment.

  The mirror image converter 13 is a module for displaying the image of the user 3 (subject image) captured by the video camera 2 on the display device 21 with the left and right reversed like a mirror. Therefore, when displaying characters, the left and right are reversed like a mirror. In this embodiment, mirror image conversion is performed by a method of inverting a horizontal image using a memory. When a cathode ray tube (CRT (Cathode Ray Tube)) is used as the display device 21, the same effect can be obtained by operating the horizontal deflection in reverse. In that case, it is necessary to reverse the graphics and other images to be mixed in advance in the horizontal direction.

  The scaler 14 is an image adjuster that enlarges or reduces the size of a subject image photographed by the video camera 2 and adjusts the display position, as shown in FIGS. This is a module that adjusts the enlargement ratio and reduction ratio in two dimensions under the control of the determiner 18. The scaler 14 can also perform horizontal and vertical phase adjustments without performing enlargement / reduction.

  In this embodiment, as shown in FIG. 4, the scaler 14 is provided at the subsequent stage of the mirror image converter 13 and adjusts the image converted by the mirror image converter 13, but between the video camera 2 and the mirror image converter 13. The scaler 14 may be provided to adjust the image output from the video camera 2.

  The graphics generator 15 is an operation image generation unit that generates a menu screen based on recognition object information for generating a menu screen (operation image) supplied from the control information determination unit 18. Output signal that is combined with or mixed with the video signal in the subsequent stage is the luminance (Y) signal, even if the signal is developed as the primary color signal of the R (red) signal, G (green) signal, and B (blue) signal. The color difference (R−Y, B−Y) signal is assumed. Further, the number of generated graphics planes is not limited, but one plane may be used in the present embodiment.

  In this embodiment, the number of pixels (resolution) of graphics constituting the menu screen is made to match the number of pixels (resolution) of the display device 21. If they do not match, a pixel number converter may be provided in the graphics generator 15 or at the subsequent stage of the graphics generator 15 so that the numbers of pixels match each other.

  In addition, the graphics generator 15 according to the present embodiment includes an operation image control unit 15a, and the operation image control unit 15a includes an operation button for an operation image according to the recognition target object information. This module adjusts the display position or range. As the display switching of the operation buttons, as shown in FIG. 2 and FIG. 9A, the attribute of the recognition object included in the recognition object information is the operator's hand and the number of hands is one. When the number of hands is two or more as shown in FIG. 9B, the operation buttons are gathered and arranged around one hand displayed on the screen. The operation images are distributed and arranged around each of the hands displayed on the screen. In addition, when the attribute of the recognition object included in the recognition object information is the face of the operator, as shown in FIGS. 9C and 9D, one or more faces displayed on the screen Operation buttons may be arranged around the side (both sides and circles). Further, the graphics generator 15 determines the attributes and the number of objects to be recognized, prioritizes objects that can be recognized more reliably, or selects an arbitrary object to be recognized by a user operation. For example, the operation button may be appropriately switched, and the operation button layout may be appropriately switched according to the type and position of the switched recognition object.

  Further, in the operation image generated by the graphics generator 15, the operation buttons are grouped according to their attributes (terrestrial broadcasting, satellite broadcasting (BS, CS, etc.), broadcasting method and input type such as input terminal). It is displayed in units of groups, and activation (activation) or deactivation (deactivation) is switched in units of groups in accordance with an operator's selection operation. Specifically, as shown in FIG. 10, in the operation image, the operation buttons are grouped into each operation window for each group, and the operation window is expressed as an image that looks like a page, The foremost page (operation window) W1 is activated, the operator images are mixed, and the operation buttons can be operated. On the other hand, the other pages W2 to W5 behind the foreground may be only tags that display information about the frame of the page and the attributes of the group.

  Further, the graphics generator 15 displays a guidance icon for displaying information related to the operation as a graphic or a character as shown in FIGS. 11 and 12 (a) and (b) based on the detection of the selection operation by the detection unit 17. It has a function to generate.

The first mixer 16 is an image that mixes the output signal of the scaler 14 (operator image S1) and the output signal of the graphics generator 15 (operation image Gs) by controlling the mixing ratio by the control value α1. This is a signal mixing unit. Specifically, the output signal (first mixed image M1o) is expressed by the following equation (1).
M1o = α1 × S1 + (1−α1) × Gs (1)
The control value α1 is set to a value between “0” and “1”. When the control value α1 is increased, the ratio of the output signal (operator image S1) of the scaler 14 is increased, and the output of the graphics generator 15 is increased. The ratio of the signal (operation image Gs) is reduced. The example of the mixer is not limited to this, but in the present embodiment, the same effect can be obtained as long as two types of input signal information are included. As described above, when the scaler 14 is provided between the video camera 2 and the mirror image converter 13, a signal output from the mirror image converter 13 may be used as an output signal (operator image S1).

  Further, as shown in FIG. 3, the first mixer 16 according to the present embodiment includes the area and shape of the operator image S1 that has passed through the mirror image converter 13 and the scaler 14, and the operation image Gs. The area and the shape of the page W1 which is the foremost and activated are made substantially equal, and further mixed in a state where the coordinate positions thereof are almost matched. Then, the mixed image generated thereby is further mixed with a normal video signal in the second mixer 20 and displayed on the screen of the display device 21.

  In the state where the image mixed by the first mixer 16 and the second mixer 20 is displayed on the screen of the display device 21, the detection unit 17 selects an operation button by the operator displayed on the screen. This is an operation detection unit that detects a predetermined selection operation, and includes a first detector 171, a second detector 172, ... an nth detector 17n. The number of detectors included in these detectors 17 does not specify the number, but varies in accordance with the number of operation buttons (to be described later) representing the control content linked to the menu screen generated by the graphics generator 15. The operation button corresponds to an operation button clicked with a mouse on a personal computer.

  The control information determination unit 18 analyzes the data (detection signal) output from the detection unit 17 and outputs various control signals. The control information determination unit 18 generates a control signal corresponding to the operation button operated based on the input data, and performs a control operation of each part of the own device and an external electronic device. The control information determination unit 18 is a module that performs various processes using software. In the present embodiment, processing by hardware (each functional block) and processing by software (CPU of the control information determination unit 18) are handled as a mixed module, and the module is not particularly limited to the division shown here. Hardware and software may be used in any way.

  In particular, in this embodiment, the control information determination unit 18 recognizes an object photographed by the video camera, and recognizes the attribute, the number of recognized objects, the position in the screen, or the ratio of the area occupied in the screen. A recognition information generation unit 23 that generates recognition target object information including at least one of them is provided.

  Specifically, as shown in FIG. 6, the recognition information generation unit 23 detects a recognition feature from a recognition feature storage unit 231 that stores recognition features such as hand and face shapes and colors, and the like from the image signal. A recognition target detection unit 232, and an attribute determination unit 233, a position / size determination unit 234, and a number determination unit 235 that determine the attribute, position / size, and number based on the detected recognition feature. . The recognition information generation unit 23 recognizes the object based on features such as the shape and color of the hand and face stored in the recognition feature storage unit 231 from the video captured by the video camera. An object is extracted, and data such as attributes are sent to the graphics generator 15 and the scaler 14 as recognition object information.

  The control information determination unit 18 is connected to a voice recognition unit 22 for recognizing voice input from a voice input device such as a microphone. Voice, applause and other acoustic operations can be incorporated.

  The pixel number converter 19 performs pixel number conversion to match the pixel number (resolution) of the input video signal with the pixel number (resolution) of the display device 21. The input video signal is input from the outside of the television receiver 1 such as a video signal by a television signal (including data broadcasting) received by a built-in tuner (not shown) or a video signal from a video deck. It is a video signal. Although not shown here, the horizontal and vertical synchronization signals are extracted from the input video signal, and each unit is synchronized with the horizontal and vertical synchronization signals of the input video signal by the reference synchronization generator 11.

The second mixer 20 has the same function as the first mixer 16. That is, the output signal (first mixed image M1o) of the first mixer 16 and the video signal S2 of the pixel number converter 19 are mixed by controlling the mixing ratio with the control value α2. Specifically, the output signal M2o is expressed by the following equation (2).
M2o = α2 × M1o + (1−α2) × S2 (2)
The control value α2 is set to a value between “0” and “1”. When the control value α2 is increased, the ratio of the output signal M1o of the first mixer 16 is increased, and the output signal of the pixel number converter 19 is increased. The ratio of S2 becomes small. The example of the mixer is not limited to this, but in the present embodiment, the same effect can be obtained as long as two types of input signal information are included.

  The display device 21 is assumed to be a CRT, a liquid crystal display (LCD), a plasma display (PDP), or a projection display, but the display method of the display is not limited. The input signals of the display device 21 are a luminance signal Y and color difference signals RY and BY, and are converted into RGB primary color signals and displayed inside the display device 21.

  Each of the detectors 171 to 17n includes an object extractor 30, a timing gate unit 31, and an object feature data detector 32 as shown in FIG. The object extractor 30 includes a filter according to the feature of the image, and outputs the filtered image signal to the timing gate unit 31.

  The timing gate unit 31 controls the passage of the image signal from the video camera 2 based on the timing pulse. The passing area is in a detection area described later. The object feature data detection unit 32 performs various filter processes on the signal limited to the detection area and extracts an image of the hand of the user 3 captured by the video camera 2.

  The object feature data detection unit 32 includes functional blocks for detecting various features from the extracted image signal of the hand (object) of the user 3 and generating detection signals. In the present embodiment, a histogram detector 33, an average luminance (APL) detector 34, a high frequency generation amount detector 35, a minimum value detector 36, and a maximum value detector 37 are provided. Although there are other requirements for characterizing the image, in this embodiment, detection signals generated from these detectors are used to determine that the hand is a hand by the control information determination unit 18 and to recognize its operation.

  FIG. 5B shows a detailed block diagram of the object extractor 30. In the present embodiment, in order to detect the hand of the user 3, a filter process that focuses on the skin color is performed. That is, the object extractor 30 of this embodiment includes a specific color filter 71, a gradation limiter 72, a time difference filter 75, a combiner 73, and an object gate unit 74.

  The specific color filter 71, the gradation limiter 72, and the time difference filter 75 are filters that specify objects. The specific color filter 71 limits the hue and color density (saturation) of the color signal input as the color difference signal, and the gradation limiter 72 limits the specific gradation range of the luminance signal. The time difference filter 75 extracts a hand region based on a slight difference in luminance and the like to identify a human hand even if the background of the human hand is a skin color. In this embodiment, a human hand is an object for operating an operation button described later. However, by adjusting and optimizing the hue and gradation as parameters, the object extractor 30 also detects an object other than the hand. can do.

  The synthesizer 73 synthesizes the signals input from the specific color filter 71, the gradation limiter 72, and the time difference filter 75 and converts them into region pulses. In the present embodiment, when all of the signal that has passed through the specific color filter 71, the signal that has passed through the gradation limiter 72, and the signal that has passed through the time difference filter 75 are present (AND), the region pulse that becomes high level is detected. Output. If there is a signal that has passed one or two of the specific color filter 71, the gradation limiter 72, and the time difference filter 75, a high-level region pulse may be output.

  If there is a signal that has passed through two of the specific color filter 71, the gradation limiter 72, and the time difference filter 75, a signal that has passed through three is less likely to be erroneously detected if it is set to a high level. If it is made high level, it will become more difficult to detect falsely.

  The region pulse generated by the synthesizer 73 is supplied to the object gate unit 74. The object gate 74 passes the luminance signal and the color difference signal when the region pulse is at the high level. When the area pulse is at a low level (a range outside the area pulse), neither a luminance signal nor a color difference signal is passed, and a signal having a specified value is output. In this embodiment, a black level luminance signal and a saturation zero color difference signal are output.

(Electronic device control method)
The control method of the present invention can be implemented by operating the control device having the above configuration. FIG. 7 is a flowchart showing the operation of the control device according to the present embodiment.

  First, the operator image S1 obtained by the video camera 2 is mirror-image converted by the mirror image converter 13, and the scaler 14 performs enlargement / reduction and position adjustment (S101 to S103). In parallel with this, a normal video signal is received or input (S112).

  And about this operator image S1, the recognition information generation part 23 recognizes the recognition target object Obj, the attribute of the recognized target object Obj, the number, the position in the screen of the said display apparatus, or it occupies in this screen Recognition object information including at least one of the area ratios is generated (S104). When the operator is too small, too large, or in a biased position in the image by this authentication object information, it is determined that the operator image needs to be adjusted, and enlargement / reduction and position adjustment are performed again ( “Y” in S105). For example, it is conceivable that the size of the hand is small depending on the position of the person or the position of the push button does not reach the angle of view of the video camera. In this case, the scaler 14 adjusts “enlargement”, “reduction”, and “phase (position)” of the image so that the person is appropriately sized on the screen. That is, as shown in FIGS. 8A and 8B, when the operator is far away (or close) or at the edge of the screen, the scaler 14 is controlled to “enlarge” (or “reduce”). Then, by adjusting the “phase” so that the person is positioned at the center of the screen, the user is displayed at an appropriate position in an appropriate size as shown in FIG.

  Further, based on the authentication object information, it is determined whether or not the position of the operation button of the operation image needs to be adjusted (S106). If it is determined that the adjustment is necessary ("Y" in S106), the object is determined. The display form of the operation button is determined according to the position and attribute (S107), and the operation image Gs is generated (changed) based on this determination (S108). In the default setting of the apparatus according to the present embodiment, as shown in FIG. 2, the operator image is adjusted so that the user's hand is a recognition target and the recognized hand is positioned at the center of the screen. The operation image is generated / changed so that the operation button is arranged at the center of the screen so as to be aligned with the displayed hand.

  Next, the operation image Gs generated in this way is mixed with the operator image S1 by the first mixer 16, and is mixed with the video signal S2 by the second mixer 20. The mixing process in the first mixer 16 is generated by mixing M1o, which is a mixed image of the operation image Gs and the operator image S1, at a mixing ratio corresponding to the control value α1, The mixing process in the mixer 20 is generated by mixing the mixed image M1o and the video signal S2 at a mixing ratio corresponding to the control value α2 (S109).

  The operation image Gs forms a menu screen for performing control, and operation buttons operated in a square push button type are arranged. One operator image S1 is photographed and mirror-image converted by the user and displayed in the same manner as a mirror.

  The operation buttons in the operator image S1 are associated with detection areas of the detection unit 17 including a plurality of detectors 171 to 17n, and the detectors 171 to 171 are operated by performing operations on the detection areas. 17n detects the operation and determines that the operation is for the corresponding operation button.

  In this way, the user's image is displayed on the display device 21 in the same manner as a mirror, and the control menu screen is also displayed superimposed. Thereby, the user himself / herself can recognize his / her movement and the position of the control menu. The control is performed by moving the hand (body) so that the user looks at the screen and touches the operation button on the screen. In other words, control can be performed by the user operating while looking at the screen so that the hand is positioned on the operation button.

  Specifically, each of the detectors 171 to 17n controls the passage of the image signal from the video camera 2, and the passing area is in a detection area corresponding to each operation button. The signal limited to the detection area is further filtered to extract the user's hand captured by the video camera 2, and the operation is detected (S110).

  The object extractor 30 includes a filter according to the characteristics of the image. In the present embodiment, in order to detect the user's hand, the object extractor 30 performs a filter process that pays particular attention to the skin color. The object extractor 30 further limits the hue (angle) and saturation of the color signal with the specific color filter 71 by the luminance signal level. By limiting the hue and saturation with the specific color filter 71, it can be narrowed down to the human skin color, and it can be adjusted within the limits of human skin hue, saturation, and luminance signal gradations that vary depending on sunburn and race. Detect user's hand. In this embodiment, it is intended for human (user) hands. However, if other hues and gradations are appropriately adjusted, they can be distinguished from others in the same manner in the parameter change category in this embodiment. Can do.

  Then, the movement of the hand in the detection area corresponding to the operation button touched (corresponding) is detected, and it is recognized that the corresponding operation button is pressed (“Y” in S110). In the present embodiment, the operation of bending the finger of the hand held up is repeated twice to recognize that the operation button has been pressed.

  In this embodiment, when the user touches the operation button, the user highlights the touched button by changing the color of the touched button, and displays the guidance icons shown in FIGS. On the other hand, the next operation (operation) is guided. For example, in the example shown in FIG. 12A, when the user's hand touches the operation button, the balloon is displayed, and the guidance icon in the shape of a hand performs the operation of bending the finger. Also, as shown in FIG. 12B, a guidance icon may be displayed directly on an operation button that is activated by touching the user's hand, instead of blowing.

  Further, detailed information related to the function of the corresponding operation button may be displayed in or around the guidance icon. As detailed information, for example, program information such as a program name and program contents can be considered.

  By recognizing this operation, a control signal corresponding to the control content of the corresponding operation button is output from the control information determination unit (CPU) 18 to perform operation control (S111). At this time, the operation buttons on the graphic menu screen change the shape and / or color to indicate that the operation has been recognized on the display. For example, the image may be changed as if the operation button was pressed.

  In addition, the user operation includes switching of the display form of a menu (operation button group) displayed on the screen. For example, when the operation buttons are grouped and displayed as shown in FIG. 10 and switching between active and inactive is performed for each group, an active page ( The group of operation buttons) can be switched sequentially.

(Control program for electronic devices)
The above-described control apparatus and control method for an electronic apparatus according to the present embodiment can be realized by executing a program described in a predetermined language on a computer. That is, by installing this program on a computer such as a user terminal or a Web server or an IC chip and executing the program on the CPU, a control device having the above-described functions can be easily constructed. This program can be distributed through a communication line, for example, and can be transferred as a package application that operates on a stand-alone computer.

  Such a program can be recorded on a recording medium readable by a general-purpose computer. Specifically, it can be recorded on various recording media such as a RAM card in addition to a magnetic recording medium such as a flexible disk or a cassette tape, or an optical disk such as a CD-ROM or DVD-ROM.

  According to the computer-readable recording medium in which the program is recorded, the above-described apparatus and method can be implemented using a general-purpose computer or a dedicated computer, and the program can be stored, transported, and installed. It can be done easily.

(Example of change)
In the above-described embodiment, the operation image developed on the graphics displayed on the display device is operated while looking at its own mirror image superimposed on it. There is. In that case, the mixing ratio of the first mixer 16 or the second mixer 20 is controlled with a timing pulse (horizontal and vertical) for specifying an area on the screen of the detector so that it can be easily seen. For example, in order to make the operation buttons that are control images easy to see, the first mixer 16 may control with the timing pulse so as to reduce the value of the control value α1. By finely controlling the mixer as described above, the relationship between the viewer and the control image (operation button) can be easily understood.

  The operation method of this embodiment is to operate by touching an operation button drawn on symmetrical graphics using a human hand. It is possible to replace all of the conventional remote controls such as “channel”, “volume”, “input switching”, “up”, “down”, “left”, “right”, “decision”.

  The present invention is not limited to the embodiment described above, and various modifications can be made. For example, the mixing ratios of the first mixer 16 and the second mixer 20 shown in FIG. 4 may be effectively combined in the level direction, the space direction, and the time direction. As a result, various combinations of images from the video camera 2 and the generated graphics menu screen and broadcast program video are possible, and various information can be exchanged at the same time. The electronic device to be operated according to the present invention is not limited to a television receiver, but a personal computer, a combination of a television receiver and a video disk recorder, or a combination of a television receiver and a video tape recorder. Any device having a display device may be used.

(Action / Effect)
By using the operation method as described above, it is possible to control a control target without using a control device such as a remote controller. Therefore, troubles related to the remote control such as battery exhaustion and contact failure are eliminated. In addition, when there are a plurality of remote control devices, it has been necessary to search for a remote control device to be controlled. However, in the operation method of the present invention, it is sufficient if there is a human body, and such a problem is solved.

  In addition, since the control content is displayed on the screen, various control devices can be handled. This is important in a networked environment, and controls a network device through a device having a display device, in other words, by transferring a control screen of a device connected to the network to a device having a display device through the network. Since this screen becomes the control screen, any control screen can be supported.

  In particular, in the present embodiment, a mixed image M1o obtained by mixing the operator image S1 and the operation image Gs including the operation button is displayed, and an operation on the operation button of the operator in the screen is detected to detect the operation on the electronic device. When executing the control operation, the size and position of the operator on the screen, the display position of the operation button, etc. according to the attribute, number of objects, the position in the measurement area, or the ratio of the area to the predetermined area Since the range is adjusted, the operation buttons can be displayed at an appropriate position and range in accordance with the display position of the hand or face of the operator who performs the operation (action), and operability can be improved.

  In the present embodiment, the operator image S1 and the activated page W1 in the forefront of the operation image Gs are mixed so that the area, shape, and coordinate position are substantially matched. It is possible to eliminate a sense of incongruity caused by a visual step at the boundary between the activated page W1 in the foreground image Gs and the peripheral image S2.

  Furthermore, in the present embodiment, the operation buttons are grouped according to attributes such as the broadcasting system and broadcasting station, and activation or deactivation can be switched on a group basis so as to turn the page. Access to the operation buttons desired by the user is facilitated, and the operator can intuitively grasp the operation buttons that can be operated. On the other hand, other groups are deactivated (deactivated), and attribute information such as station titles is displayed, so the operator still has groups as other options even if they cannot currently operate It is possible to intuitively grasp what to do.

  Further, in the present embodiment, when an operation on the operation button of the operator is detected, information (guidance or hint) regarding the next operation (operation) is displayed, so that the smooth operation of the operator can be guided. And operability can be improved.

It is explanatory drawing which shows the method of controlling the electronic device (television receiver 1) which concerns on embodiment. It is explanatory drawing which shows the screen structure of the television receiver 1 which concerns on embodiment. It is explanatory drawing which decomposes | disassembles and shows the screen structure of the television receiver 1 which concerns on embodiment. It is a block diagram which shows the structure of the television receiver 1 which concerns on embodiment. (A) is a block diagram which shows the structure of the detectors 171-17n which concern on embodiment, (b) is a block diagram which shows the structure of the object extractor 30. FIG. It is a block diagram which shows the structure of the recognition information generation part 23 which concerns on embodiment. It is a flowchart figure which shows operation | movement of the control apparatus which concerns on embodiment. It is explanatory drawing about adjustment (enlargement / reduction / position) of the operator image which concerns on embodiment. It is explanatory drawing about adjustment of the image for operation which concerns on embodiment. It is explanatory drawing about grouping of the operation button in the image for operation which concerns on embodiment. It is explanatory drawing of the guidance icon in the image for operation which concerns on embodiment. It is explanatory drawing which shows operation | movement of the guidance icon which concerns on embodiment.

Explanation of symbols

Obj ... recognition object α1, α2 ... control value Gs ... operation image M1o ... first mixed image M2o ... second mixed image S1 ... operator image S2 ... normal video signal W1-W5 ... page 1 ... television Receiver 2 ... Video camera 3 ... User 11 ... Reference synchronization generator 12 ... Timing pulse generator 13 ... Mirror image converter 14 ... Scaler 15 ... Graphics generator 15a ... Operation image controller 16 ... First mixer 17 DESCRIPTION OF SYMBOLS ... Detection part 18 ... Control information judgment device 19 ... Pixel number converter 20 ... 2nd mixer 21 ... Display apparatus 22 ... Speech recognition part 23 ... Recognition information generation part 30 ... Object extractor 31 ... Timing gate device 32 ... Object Feature data detector 33 ... Histogram detector 34 ... Average luminance detector 35 ... High frequency generation amount detector 36 ... Minimum value detector 37 ... Maximum value detection 40, 41 ... operation buttons 231 ... recognition feature storage unit 232 ... recognition target detection unit 233 ... attribute discrimination unit 234 ... position / size discrimination unit 235 ... number discrimination unit 71 ... specific color filter 72 ... gradation limiter 73 ... synthesizer 74 ... Object gate device 75 ... Time difference filter 171-1n ... Detector

Claims (13)

A control device for an electronic device having a display device,
A video camera to shoot the operator,
Recognizing a part of the operator's body photographed by the video camera or an object that is an object held by the operator, the attributes, the number of recognized recognition objects, and the position on the screen of the display device Or a recognition information generating unit that generates recognition target object information including at least one of the proportion of the area in the screen;
A mirror image adjustment unit that converts an image captured by the video camera into a mirror image, and performs enlargement, reduction, or position adjustment of the recognition object displayed on the screen according to the recognition object information;
An operation image generating unit that generates an operation image including at least one operation button and adjusts a display position or a range of the operation button according to the recognition target object information;
An image signal mixing unit that mixes the image that has passed through the mirror image adjustment unit and the operation image to form a mixed image, and displays the mixed image on the screen;
In a state where the image mixed by the image signal mixing unit is displayed on the screen, the operator picked up by the video camera and displayed on the screen selects the operation button using the recognition object. An operation detection unit for detecting a predetermined selection operation;
A control device for an electronic device, comprising: a device control unit that performs a control operation corresponding to the operation button for which the predetermined selection operation is detected by the operation detection unit. The operation image generation unit includes:
When the recognition object attribute included in the recognition object information is an operator's hand and the number of the recognition object is one, the operation is performed around the recognition object displayed on the screen. Place the buttons together,
When the attribute of the recognition object is an operator's hand and the number of the recognition objects is two or more, the operation buttons are distributed around each of the recognition objects displayed on the screen. The electronic device control device according to claim 1, wherein the electronic device control device is arranged. The operation image generation unit includes:
2. The operation button is arranged around the recognition object displayed on the screen when the attribute of the recognition object included in the recognition object information is an operator's face. The control apparatus of the electronic device as described in 2. The operation image generating unit switches the display position of the operation button according to the attribute and the number of recognition objects included in the recognition object information. Electronic device control device. The operation buttons are grouped according to their attributes,
In the operation screen, the operation buttons are displayed in units of groups, and activation or deactivation is switched in units of groups according to an operator's selection operation,
In the display range of the activated group, an image that has passed through the mirror image adjustment unit and the operation image are mixed,
5. The electronic device control device according to claim 1, wherein at least information related to the attribute of the deactivated group is displayed within a display range of the group. 6.   6. The operation image generating unit has a function of generating a guidance icon that displays information related to an operation with a graphic or a character based on detection of the selection operation by the operation detection unit. The control apparatus of the electronic device in any one of. A method of controlling an electronic device having a video camera for photographing an operator and a display device,
The recognition information generation unit recognizes a part of the operator's body photographed by the video camera or an object that is an object held by the operator, and the attributes, number, and display of the recognized recognition object A recognition information generating step for generating recognition object information including at least one of a position in the screen of the device or a ratio of the area occupied in the screen;
A mirror image image is obtained by performing mirror image conversion on an image photographed by the video camera by a mirror image adjustment unit, and performing enlargement, reduction, or position adjustment of the recognition object displayed on the screen according to the recognition object information. An adjustment step;
An operation image generation step of generating an operation image including at least one operation button by the operation image generation unit and adjusting a display position or a range of the operation button according to the recognition object information;
An image signal mixing step in which an image that has passed through the mirror image adjustment unit and the operation image are mixed in an image signal mixing unit to form a mixed image, and the mixed image is displayed on the screen;
In a state where the image mixed by the image signal mixing unit is displayed on the screen, the operator picked up by the video camera and displayed on the screen selects the operation button using the recognition object. An operation detection step of detecting a predetermined selection operation by the operation detection unit;
And a device control step of performing, by the device control unit, a control operation corresponding to the operation button for which the predetermined selection operation is detected by the operation detecting unit. In the image generation step, the operation image generation unit includes:
When the attribute of the recognition object included in the recognition object information is an operator's hand and the number of the recognition objects is one, the recognition object displayed on the screen is surrounded around the recognition object. Arrange the operation buttons together,
When the attribute of the recognition object is an operator's hand and the number of the recognition objects is two or more, the operation buttons are distributed around each of the recognition objects displayed on the screen. The method of controlling an electronic device according to claim 7, wherein the electronic device is arranged in a manner. In the image generation step, the operation image generation unit includes:
The operation button is arranged around the recognition target displayed on the screen when the attribute of the recognition target included in the recognition target information is an operator's face. The control method of the electronic device of description. 10. The operation image generation unit in the image generation step switches a display position of the operation button according to an attribute and the number of recognition objects included in the recognition object information. The control method of the electronic device in any one of. The operation buttons generated in the operation image generation step are grouped according to their attributes,
In the operation screen generated in the operation image generation step, the operation buttons are displayed in groups, and activated or deactivated are switched in groups in accordance with an operator's selection operation.
In the image signal mixing step, the display range of the activated group is mixed with the image that has passed through the mirror image adjustment unit and the operation image, and at least about the deactivated group 11. The method for controlling an electronic device according to claim 7, wherein information relating to the attribute of the group is displayed within a display range of the group.   In the operation detection step, the operation image generation unit has a function of generating a guidance icon for displaying information related to an operation with a graphic or a character based on detection of the selection operation by the operation detection unit. The control method of the electronic device in any one of Claims 7 thru | or 11. A control program that uses a computer to control an electronic device having a video camera that captures an operator and a display device,
The recognition information generation unit recognizes a part of the operator's body photographed by the video camera or an object that is an object held by the operator, and the attributes, number, and display of the recognized recognition object A recognition information generating step for generating recognition object information including at least one of a position in the screen of the device or a ratio of the area occupied in the screen;
A mirror image image is obtained by performing mirror image conversion on an image photographed by the video camera by a mirror image adjustment unit, and performing enlargement, reduction, or position adjustment of the recognition object displayed on the screen according to the recognition object information. An adjustment step;
An operation image generation step of generating an operation image including at least one operation button by the operation image generation unit and adjusting a display position or a range of the operation button according to the recognition object information;
An image signal mixing step in which an image that has passed through the mirror image adjustment unit and the operation image are mixed in an image signal mixing unit to form a mixed image, and the mixed image is displayed on the screen;
In a state where the image mixed by the image signal mixing unit is displayed on the screen, the operator picked up by the video camera and displayed on the screen selects the operation button using the recognition object. An operation detection step of detecting a predetermined selection operation by the operation detection unit;
An electronic device control program that causes a device control unit to execute a control operation corresponding to an operation button for which the predetermined selection operation is detected by the operation detection unit.