WO2004084544A1

WO2004084544A1 - Remote control apparatus and method based on video image detection

Info

Publication number: WO2004084544A1
Application number: PCT/CN2004/000210
Authority: WO
Inventors: Xuwen Yu; Wentao Zheng
Original assignee: Matsushita Electric Industrial Co., Ltd.
Priority date: 2003-03-19
Filing date: 2004-03-17
Publication date: 2004-09-30
Also published as: CN1258135C; CN1533180A

Abstract

The present invention provides a remote control apparatus based on video image detection, the apparatus includes: image acquisition device for extracting video image and converting the video image into the video signal; selected position calculation device for detecting identification in the video signal and calculating cursor position pointed by the remote control apparatus; instruction input device for inputting the instruction of entering or exiting the menu option selection state to the remote control apparatus; remote control signal transmission device for receiving the cursor position information outputted by the selected position calculation device and transmitting control signal which points the cursor position information, so that the display device shows the object directed by the remote control apparatus. The present invention still includes a remote control system and a remote control method using the remote apparatus.

Description

Remote control device and method based on video picture detection

Technical field

The present invention relates to a remote control device based on picture detection and a method thereof, and in particular, to a remote control that determines the direction of a remote control device by detecting a marker in an interactive television, thereby selecting a candidate object, and performing a remote control for remote control operation. Device and method. Background technique

The ordinary TV remote control can select TV channels and different functions by pressing the numbers and function buttons provided on the remote control. But these remotes do not have interactive selection. With the continuous expansion of TV functions, analog TVs with only simple viewing functions are developing towards interactive, intelligent digital TVs. In the digital TV screen, in addition to displaying videos, a menu interface can also be provided to implement a variety of data broadcasting, multimedia services, and video on demand (V0D) services. Convenient and simple selection of specific parts such as menus in the screen, and operations such as clicking and selecting are the basis for realizing interaction between digital television and people.

As in the computer field, the tools that have achieved similar functions are mainly mice. However, the movement of the mouse and the movement of the cursor do not correspond directly and naturally. Instead, the movement of the cursor is associated with the up, down, left, and right arrow keys or with the mouse. Corresponding to a plane of movement, the operation is not natural enough.

Electronic devices currently equipped with menus and programming menus usually use function keys on the remote control to call up the menu screen. When the menu screen is called up, the pointer is usually located in the upper left corner of the menu. The user moves the pointer to the option to be selected by using the arrow keys on the remote control, and then presses the OK key to select the option. In the existing remote control, the user needs to repeatedly press the arrow keys on the remote control to move the pointer to the menu option to be selected. When there are many menu options displayed on the screen, the operation is cumbersome and lacks flexibility. In addition, when the menu is relatively complicated, the operation of the remote control in the prior art affects the speed of the interactive operation. Summary of the Invention

An object of the present invention is to provide a remote control system that determines the direction of a remote control device by detecting a marker in a television picture, and moves the remote control device to point at an option to be selected to select a specific option.

Another object of the present invention is to provide a method for detecting a marker in a television picture by using a remote control device to determine Point the remote control device, and move the remote control device to point at the option to be selected to select the remote control device for the specific option.

It is still another object of the present invention to provide a method for determining a specific direction of a remote control device by detecting a marker in a television picture, and moving the remote control device to point at an option to be selected.

According to a first aspect of the present invention, a remote control system based on video image detection is provided, including: a remote control device provided with an image acquisition device, and the image collection device is configured to collect video images so that the remote control device detects A marker in a video image, and calculating a target pointed by the remote control device according to the position information of the marker; and a display device for receiving a control signal sent by the remote control device, and displaying it for the remote control device through a video display interface Video images of detected markers and interactive menu options for selection, and indicate the target to which the remote control is pointing.

According to a second aspect of the present invention, a remote control device based on video image detection is provided, including: an image acquisition device for collecting video images and converting the video images into video signals; and selecting a position calculation device for selecting Measuring a marker in the video signal and calculating a cursor position pointed by the remote control device; an instruction input device for inputting an instruction to the remote control device to enter or exit a menu option selection state; and a remote control signal transmitting device for Receiving the cursor position information output by the selected position calculation device and transmitting a control signal indicating the cursor position information, so that the display device displays a target pointed by the remote control device.

According to a third aspect of the present invention, a remote control method based on video image detection is provided, comprising the steps of: collecting a video image displayed by a display device; detecting whether the captured video image includes a marker for determining a direction of the remote control device; If the video image includes the marker, calculating position information of the marker on a display interface of the display device; calculating the remote control device pointing display based on the calculated position information of the marker Position information of the interface; sending the calculated position information of the remote control device pointing to the display interface to the display device, and displaying the currently selected option of the remote control device on the display interface.

The remote control device and method according to the present invention overcome the shortcomings of the prior art, which require repeatedly pressing a button to select a candidate option and cumbersome operations. BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of the present invention will be made clearer by the following detailed description of the non-limiting embodiments of the present invention with reference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram showing a remote control system according to an embodiment of the present invention;

2 is a block diagram showing a remote control system according to an embodiment of the present invention; 3 is a flowchart showing an operation process of a remote control system according to an embodiment of the present invention; FIG. 4 is a diagram showing an outline of a marker according to an embodiment of the present invention;

FIG. 5 is an example of an image taken by a remote controller of the present invention; and

Fig. 6 shows the outline of a marker according to another embodiment of the present invention. detailed description

The principle of the present invention is described below. First, use the function keys on the remote control device to send a control signal to the TV to make the TV enter the TV interactive operation state. In addition to displaying the interactive menu screen on the screen, the TV automatically displays a special marker on the screen. Secondly, the menu screen is captured by the image collection device installed on the remote control device to detect the marker, and the position pointed by the remote control device is calculated by calculating the position of the marker, that is, a specific part of the screen that should be selected . Finally, a corresponding indication cursor is displayed at the calculated screen position and the cursor is fed back to the operator. The operator can change the selected position or perform the selection operation by referring to the indication cursor.

Embodiments of the present invention will be described below with reference to the drawings.

First, a schematic diagram of a remote control system according to an embodiment of the present invention is described with reference to FIG. 1. The remote control system according to the present invention includes a remote controller and a television. According to the principle of the present invention, the position pointed by the remote controller is determined by detecting the screen displayed on the screen, and the menu option to be selected is selected by changing the direction of the remote controller. Screen detection itself is a more difficult problem, because the pattern on the screen is constantly changing, and the external background light may change at any time. Moreover, the system requires the detection speed to be real-time, which causes a lot of interference to the screen detection. In order to solve these problems, the present invention adopts a method of displaying a marker on a television screen after entering a detection state, so as to implement fast and accurate screen detection.

Specifically, an image collecting device is installed on a television remote control. When the function key on the remote control is pressed to switch the screen to the menu screen, the position of the TV screen is detected in real time through the position of the marker in the image captured by the image acquisition device in the remote control, and the position pointed by the remote control is determined And pointing to achieve the purpose of selecting a specific position in the picture.

FIG. 1 includes a television 1, a remote control 2, and a picture 3 taken by an image collecting device of the remote control. The position of the marker and the position of the remote control are marked on the TV screen. In the figure, the ellipse in the upper right corner of the screen of the television 1 shows a marker, the dots on the screen show the points pointed by the remote control, and the reference numeral 3 indicates a picture taken by the image acquisition device of the remote control 2.

An embodiment of the remote control system of the present invention is described below with reference to FIG. 2. As shown in Figure 2, the remote control system package Includes a remote control section 21 and a television section 22 that emit control signals. The remote control part 21 includes an image collection device 23 that collects images, a selection position calculation device 24 that calculates detection markers and indicates a cursor position, and a remote control signal transmission device 25 that transmits a remote control signal to a television set, for controlling The system enters or exits the command input key 26 of the menu option selection state. The instruction input key 26 includes a menu selection start key and a selection cancel key. The television section 22 includes a video display device 27 and a remote control signal receiver 28, and other devices (not shown) included in the television.

The operation of the remote control system of the present invention is described below. The remote control detects whether the screen selection state is entered. If the selection start key of the remote control is pressed, it sends an instruction to the image collection device 23, which is a camera in this embodiment, and the selection position calculation device 24 to enter the screen menu selection state. . The signal transmitter 25 in the remote control transmits a signal for entering the menu selection state to the signal receiver 28 provided in the television. The remote control system enters the menu selection state. At this time, the menu options available for selection are displayed on the TV screen. As an alternative, currently a digital TV remote control is generally provided with a data key for receiving data broadcasting services other than general video viewing. Therefore, the remote control system can also be set to the menu selection state when the data key is pressed. After entering the menu selection state, the video signal of the marker to be detected is generated inside the TV and the marker is displayed on the screen.

Next, point the remote control at the display screen, and the image gathering device 23 on the remote control, which is a camera in this embodiment, starts to capture the image in the pointed direction, and converts the captured image into a video signal and provides it to the selected position. The computing device 24 performs processing (the processing process will be described below), thereby obtaining the position pointed by the remote control, and determining which menu option the pointing position of the remote control is within, and highlighting the corresponding menu option State, that is, with the cursor pointing at the menu option. Thereafter, the user can simply change the direction of the remote controller, point the remote controller at the option to be selected, and use the instruction input key 26 to make a selection.

In the prior art, a screen position selection device, such as a common menu selection remote control, enters the selection state, and the general cursor or focus falls on a certain menu bar. The corresponding menu bar is selected by pressing the front, back, left, or right keys on the remote control. It is inconvenient and unnatural to operate.

Different from the prior art position selection device, the present invention needs to determine the positional relationship between the remote control and the screen. This is achieved by detecting markers on the screen. After determining the relative position relationship, we can determine the position where the cursor should point. For example, if you point the remote control to the left, the cursor can also be moved to the left on the screen. It feels like the cursor is sent by the remote control. Light spots are the same. Operation is more convenient and natural.

In a preferred embodiment of the present invention, a marker is first detected to determine which points on the image captured by the camera 23 on the remote control are marker points that we need. Followed by the location of the markers so that The detected marker point determines the center point of the marker. Since the relative position of the marker and the screen is determined, knowing the relative positional relationship between the marker and the remote control also knows the relative positional relationship between the screen and the remote control. After the position of the marker is determined, the position of the indicating cursor to be displayed on the screen can be determined through the geometric relationship.

The preferred embodiments of detecting the marker and calculating the pointing position of the remote controller in the present invention are described in detail below. First, detection of a marker will be described. The marker is displayed in the upper right corner of the screen, but the present invention is not limited to this. The marker can be displayed at any position on the screen, such as the lower right or lower left corner. At this time, the remote control points at the TV screen, and the image acquisition device 23, that is, the camera, shoots the marker.

In this embodiment, the marker is determined by detecting the color of the marker. In the captured image, the point detection and median filtering methods of the HSI color model are used to detect landmarks in real time. Among them, H is Hue, S is Saturation, and I is Intensity. Typically, the RGB color model most commonly used, it is considered the color green (Gr _e en), and blue (Blue) mixed three primary colors red (Red), formed by the. The combination of different intensities of each basic color constitutes different colors. For example, if the maximum intensity of each basic color is represented by an 8-bit binary number, then the maximum intensity of each basic color is 255, then red is R = 255, G = B = 0 _; and yellow is R = G = 255, B = 0. However, with the change in the brightness of the light, the three RGB components change simultaneously, which does not reflect an independent description of color.

The HSI color space is based on the human visual system and can clearly show the changes in hue, brightness, and color saturation. Hue and saturation are commonly referred to as hue, and are used to indicate the type and shade of color. Because human vision is much more sensitive to brightness than to color shading, in order to facilitate color processing and identification, the human visual system often uses the HSI color space, which is more in line with human visual characteristics than the RGB color space. In image processing and computer vision, a large number of algorithms can be conveniently used in the HSI color space. They can be processed separately and independently of each other. Therefore, the HSI color space can greatly simplify the workload of image analysis and processing.

HSI color space and RGB color space are different representations of the same physical quantity, so there is a conversion relationship between them.

The marker detection in the present invention is to transform the color representation of the object from the RGB color space to the HSI color space. After the image collection device 23 captures the marker, it processes its color to obtain an RGB color binary map of the marker. Then, the selected position calculation device 24 can obtain the HIS value of the marker by calculating the following conversion formula. The gray value of the marker is obtained by the following formula (1)

I = (R + G + B) / 3 (1) Where R, G, and B represent the values of the three primary colors.

Use the following formula (2) to calculate the saturation of the marker:

S = l- minCR, G,) 〃 (2)

Where min (R, G, B) means take the minimum of R, G, B.

Use the following formula (3) to calculate the chroma-

H = ((π / 2― arctan (/) + {, G> B;, G <B}) / 2TV; (3)

Where F is a variable, and F = (2 * i? -G-;

The H and S components can remain basically unchanged when the light changes, so when the light changes, the H and S components can be detected to detect the marker, eliminating the interference of external light and achieving the purpose of detecting the marker. As an example, the selection position calculation device 24 may be implemented by a microprocessor. FIG. 4 shows an embodiment of a selected marker. The marker is mapped to chroma H and saturation S components with a single color feature (blue), and the calculated H and S component values are compared with The comparison of the threshold values of the chrominance and the saturation in the position calculation device 24 is selected, and the points greater than or equal to the predetermined threshold value are set to 1, the points smaller than the threshold value are set to 0, and the part of the image that meets the predetermined condition is a landmark. Thus, a binary map B of the marker is obtained, thereby obtaining candidate points of the marker. The point set to 1 in B is the marker point. Inevitably, there are some isolated noise points in the binary map B. You can use median filtering to remove these isolated noise points. Through the above processing, the marker displayed on the screen can be determined.

In the above embodiment, a marker with a single color may be confused with the color of the marker and an object of the same color in the image displayed on the screen. Alternatively, the detection marker may have other embodiments. For example, after entering the detection state, the marker can be set off the screen. Although setting the marker off the screen increases the production cost, the processing effect will be better because the interference from the screen image is less.

The point color detection and simple filtering detection methods described above are relatively fast, but other image processing techniques, such as pattern recognition and some more complicated filtering techniques, can also be used. For example, the design of markers can use regular shapes, such as circles, diamonds, etc. In this way, the detection of markers can also assist the method of template matching in pattern recognition. That is, once you know the shape of the markers, you can Interference points that match this shape are removed.

In addition, the detection of markers can also be determined by multiple color combinations, that is, the markers are designed with multiple colors and suitable color distribution shapes to improve the anti-interference ability of detection. Figure 6 shows another embodiment of a marker. The marker contains three colors that are regularly distributed in space. The rules in this example are: The left of the blue dot is red and the right is green. After detecting three colors, only the points that meet the spatial distribution rules of the three colors It is considered a landmark point. This method can improve the stability of the system and effectively overcome the possible interference during the detection process, such as a situation where the hue and saturation are completely the same as the marker on the screen, and so on.

The above description is about converting the landmark image captured by the camera into a binary map B, and the point set to 1 in B is the landmark point. This can determine the position of the center point of the marker in the image captured by the camera.

The following describes the process of selecting the position calculation device 24 to calculate the position of the marker. As a preferred embodiment of the present invention, the following gravity center calculation formulas (4) and (5) are used to obtain the center of gravity position X of the marker as follows:

Among them, B represents a binary map, X (i, j) is the abscissa, Y (i, j) is the ordinate, the lower left corner of the captured image is selected as the coordinate origin, and W and H are the horizontal and vertical resolutions of the camera, respectively.

After the position of the center of gravity of the marker in the captured image is obtained, since the position of the center of gravity can be used as the center position of the marker, and the horizontal and vertical resolutions of the camera are fixed, the position of the indicating cursor can be calculated from this. For example, if the marker is to the right in the captured image, the camera on the remote control can be pointed to the left of the screen, then the indication cursor should appear in the direction pointed by the remote control, that is, the left side of the screen. The relative positions of other markers and indicating cursors can be deduced by analogy. The position of the cursor on the screen can be converted by the position of the marker in the captured image. X _s and Y _s (with the lower left corner of the screen as the origin) _d

The cursor position i _s on the screen can be obtained from the following conversion formulas (6) and (7):

X _s = W _s / 2 + K _x * (W / 2-X _z ) (6)

Y _s = H _s / 2 + K _y * (H / 2-Y _z ) (7)

Where ^; represents the horizontal resolution of the screen, H represents the vertical resolution, _J ^ is a coefficient representing the speed of the cursor movement, and other symbols represent the same as described above.

Taking formula (6) as an example, if the marker is at _z of the image captured by the camera, then the corresponding description indicates that the camera is pointing in the opposite direction, which is _-2 . The speed of the cursor movement is controlled by the coefficient _ ^. Because the origin is taken in the lower left corner, plus the amount of translation compensation, the final formula expression is as shown in (6). (7) The same is true.

In summary, the significance of formulas (6) and (7) is that if the marker appears on one side of the camera, it indicates the light The cursor appears on the corresponding side of the screen, indicating that the speed of the cursor movement is controlled by the coefficient and.

After the position information of the indicator light is obtained through calculation of the above formulas (6) and (7), the cursor position information is sent to the television through the signal transmitting device 25 in the remote controller. The TV shows the currently selected location on the screen.

In the menu selection application, if the position falls within a menu area, the menu bar is selected; press the button to enter this option. In addition, the location information can also be applied to future interactive television, such as continuous area selection.

If you press the selection cancel key on the remote control, it enters the non-selection state, and the corresponding specific mark and position indicator disappear.

The operation flow of the remote control system is described below with reference to FIG. 3. First, in step S31, it is detected whether the remote control system is in an interactive operation state. If the result is negative, it is detected whether a marker is displayed on the screen in step S32, and if a marker is displayed on the screen, the marker is eliminated in step S33 to facilitate viewing the displayed image. If the judgment result of step S32 is negative, that is, no marker is displayed on the screen, the flow returns to step S31 to detect whether the remote control system is in an interactive state. If the result of step S31 is positive, the flow proceeds to step S34, and the remote control device 21 sends a remote control signal to the television to display a marker on the television. Next, the flow proceeds to step S35, and the remote control device 21 determines whether a marker on the image is detected in the images collected by the image acquisition device 23. If no marker is detected, the process returns to step S31. If a marker in the image is detected in step S35, calculation is performed on the center position of the marker and the position of the point pointed by the cursor of the remote control device, and the selected option of the point is fed back to the method by "highlighting" or the like. The operator, so that the operator can change the selected position or perform the selection operation by adjusting the pointing of the remote control with reference to the instruction cursor. After that, the remote control can send a remote control signal and transmit the position information indicating the click of the cursor to subsequent operations.

FIG. 5 shows a specific example of a screen marker captured by the remote control system of the present invention. The left half of the figure shows a screen shot with markers, where a blue marker with the text "PANASONIC" is displayed in the upper right corner of the screen. The right half of the figure shows the image actually taken by the camera of the remote control, which shows the blue marker.

The above describes the remote control system and method for selecting a specific position in a picture based on the position detection of a television picture according to the present invention.

In the present invention, when the television is in an interactive operation state, the television automatically displays a special marker on the screen. The marker is detected by a camera mounted on the remote control, and the position pointed by the remote control is calculated from it Setting, that is, a specific part of the picture that should be selected. Finally, a corresponding indication cursor is displayed at the calculated screen position and fed back to the operator. The operator can change the selection position or perform the selection operation by referring to the pointer. With this technology, it becomes more convenient and natural to select a specific location on the screen.

Although the present invention is given and described with reference to the preferred embodiments of the present invention, those skilled in the art should understand that various changes and modifications can be made to the present invention without departing from the spirit and scope of the present invention.

Claims

1. A remote control system based on video image detection, comprising:

A remote control device provided with an image gathering device, the image acquisition device is used to gather video images, so that the remote control device detects a marker in the video image, and calculates a pointer pointed by the remote control device according to the position information of the marker. Goals; and

Right

The display device is configured to receive a control signal sent by the remote control device, and display a video image of a marker detected by the remote control device and an interactive menu option for selection through a video display interface, and indicate a target pointed by the remote control device.

2. A remote control device based on video image detection, comprising:

An image acquisition device for acquiring a video image and converting the video image into a video signal; selecting a position calculation device for detecting a marker in the video signal and calculating a cursor position pointed by the remote control device;

An instruction input device for inputting an instruction to enter or exit a menu option selection state to the remote control device; and

The remote control signal transmitting device is configured to receive the cursor position information output by the selected position calculation device and transmit a control signal indicating the cursor position information, so that the display device displays a target pointed by the remote control device.

3. The remote control device based on video image detection according to claim 2, wherein the image acquisition device is a camera.

4. The remote control device based on video image detection according to claim 2, wherein the selected position calculation device converts the RGB three-primary color model of the markers collected by the image collection device into hue-saturation-brightness (HSI) color model to detect the marker.

5. The remote control device based on video image detection according to claim 4, wherein the selected position calculation device performs conversion of the RGB three-primary color model into a hue-saturation-intensity (HIS) color model according to the following formula-

I = (R + G + B) / 3

S = l-min (R, G, B) / I

H = ((π / 2-arctan (/ S) + {0, G> Β; π, β <Β}) / 2π;

Where R, G, and B represent the values of the red, green, and blue components of the marker, = (2 *?-G- represents brightness, S represents color saturation, and H represents chromaticity.

The remote control device based on video image detection according to claim 5, wherein after mapping to the chroma H and chroma S components, the calculated H and S component values are set with the chroma and For the threshold comparison of color saturation, points greater than or equal to a predetermined threshold are set to 1, points less than the threshold are set to 0, and an image portion satisfying a predetermined condition is a landmark.

7. The remote control device based on video image detection according to claim 6, wherein after the selected position calculation device detects a marker, the marker image is converted into a binary map B, and the calculation is performed according to the following formula Position of the marker in the captured image X ^ Β Υ _ζ:

Among them, B represents the binary image of the landmark image transformation, X (i, j) is the abscissa, Y (i, j) is the ordinate, the lower left corner of the captured image is selected as the coordinate origin, and W and H are the camera level , Vertical resolution.

The remote control device based on video image detection according to claim 7, wherein the selected position calculation means calculates the indication cursor on the screen according to the following formula according to the position information of the marker in the captured image: χ ^ Π Υ _ζ Position ₃ , Y _s :

_{_{X s = W s / 2 +}} K x - * (W / 2-X:)

Y _s = H _s / 2 + K _y * (H / 2-Y ₂ ) where ^; represents the horizontal resolution of the screen, and H _s represents the vertical resolution, which is a coefficient representing the speed of the cursor movement.

9. The remote control device based on video image detection according to claim 2, wherein the instruction input device activates the image collection device and the selected position when an instruction to enter a menu option selection state is input to the remote control device. Computing device.

10.—A remote control method based on video image detection, including steps:

釆 collection of video images displayed by the display device;

Detecting whether the set of video images contains a marker for determining the direction of the remote control device; if the video image includes the marker, calculating a target of the marker on a display interface of the display device information; Calculating target information of the remote control device pointing to the display interface according to the calculated position information of the marker;

Sending the calculated target information of the remote control device pointing to the display interface to the display device, and displaying the currently selected option of the remote control device on the display interface.

The remote control method based on video image detection according to claim 10, further comprising adjusting the pointing of the remote control device when the remote control device is not pointing at the option to be selected on the display interface, so that the remote control device points The option to choose.

12. The remote control method based on video image detection according to claim 10, wherein the step of detecting the marker comprises converting a RGB three-primary color model of the marker to a hue-saturation-brightness (HSI) color model A step of.

13. The remote control method based on video image detection according to claim 12, further comprising converting the RGB three-primary color model into a chroma H and a chroma S component, and calculating the calculated H and S component values and the selected position A step of setting a threshold value comparison of chromaticity and color saturation in the computing device, where a point greater than or equal to a predetermined threshold value is set to 1, and a point smaller than the threshold value is set to 0.

14. The remote control method based on video image detection according to claim 10, further comprising, after detecting the marker, converting the marker image into a binary map B, and calculating the in-set set of the marker Steps ^ and ^ in the image.

15. The remote control method based on video image detection according to claim 14, further comprising calculating a position X _s of the indication cursor on the screen of the remote control device according to the position information of the marker in the captured image and Y _z , ; ^ Steps.