JP2008181199A - Image display system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To change the size and position of a display image in response to the position of a user, by imaging an image display part by a camera in a remote controller side to detect the position of the user. <P>SOLUTION: This image display system is constituted of a TV 1 and a remote controller 2. The remote controller 2 is provided with a camera 20 capable of imaging an image display part 10 of the TV 1, and a position detecting part 23 for detecting a direction and a relative position of the remote controller 2 by comparing coordinates of a picked-up cursor image 14b in a pick-up image picked up by the camera 20, with coordinates of a displayed cursor image 14a displayed at that time. A communication part 25 is also provided to transmit data thereof. An image generation part of the TV 1 enlarges the display image and changes the display position, based on the direction and the relative position of the remote controller 2. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image display system including a television (TV) or a personal computer (PC) and a remote control device (remote control).

  There has been proposed an image display system for pointing (pointing) an arbitrary position using a remote controller on a display screen of a TV or PC. In such a system, it is desirable to change the size of the display image on the display screen (for example, the size of the menu image) according to the position of the user holding the remote control, or to change the display position. . That is, as the user is farther from the display screen, the display image is enlarged, and when the user moves left and right from the center of the display screen, the display image also moves accordingly.

  Conventionally, the following has been proposed as an image display system with such a device added.

  For example, in Patent Document 1, a camera is installed beside the display screen, and the position and size of the display image are changed based on the position and size of the user imaged by the camera.

  In Patent Document 2, light emitted from a remote controller is detected by two light receiving portions of a TV, a distance between the TV and the remote controller is obtained by a triangle method, and the character size of the display portion is changed according to this distance.

Moreover, in patent document 3, a camera is installed in a tablet and the pointer on a display screen is moved according to the position and movement of the user's nose imaged with this camera.
Japanese Patent Laid-Open No. 08-123612 JP 2004-287168 A Japanese Patent Laid-Open No. 07-093089

  In the image display systems shown in the above Patent Documents 1 and 3, a camera is provided on the display device side to capture a part of the user's face and body. However, in such an apparatus, it is difficult to correctly recognize the user because a part of the user's face and body are not always the same. If an attempt is made to increase the accuracy of image recognition, the image processing time for recognition becomes longer and the responsiveness becomes lower. In addition, when a plurality of users are in the camera field of view, there is a possibility that the user cannot be recognized.

  In the image display system disclosed in Patent Document 2, since it is necessary to accurately detect the light receiving angle at the light receiving unit, there is a problem that the structure of the light receiving unit is complicated or expensive. An object of the present invention is to provide an image display system capable of recognizing a user's position with low cost and high accuracy by capturing an image display unit with a camera on a remote controller and detecting the position of the user. is there.

The image display system of the present invention has the following configuration in order to solve the above problems. That is,
An image display device including an image generation unit that generates an image including a predetermined image, and an image display unit that displays the generated image;
In an image display system comprising: an operation unit that controls display of the image display device; and a remote controller that transmits an operation content operated by the operation unit to the image display device.
The remote control is
A camera capable of capturing an image display unit when the remote control is pointed at the image display device;
The coordinates of the predetermined display image displayed on the image display unit are compared with the coordinates of the predetermined image captured in the captured image captured by the camera, the direction of the remote control with respect to the image display unit, the image display unit, A position detection unit for detecting a remote control relative position representing a distance between the remote controls;
A transmission unit that transmits the remote control direction and the remote control relative position detected by the position detection unit to the image display device;
The image generation unit of the image display device sets a display state of a display image to be displayed on the image display unit based on a remote control direction and a remote control relative position transmitted from the transmission unit.

  In the image display system, the position detection unit detects the orientation of the remote control and the remote control relative position by comparing the coordinates of the predetermined image captured by the camera with the coordinates of the predetermined display image.

  The predetermined image is, for example, a cursor image. An image of a menu screen on which commands, selectable contents, and the like are arranged may be used. The predetermined image can be stored in advance as a reference predetermined image in a memory provided in the remote controller.

  Therefore, if, for example, a reference cursor image is stored as a reference predetermined image in the memory of the remote controller, the position detection unit compares the image of the image display unit captured by the camera with the reference cursor image stored in the memory. . At this time, if the cursor can be found in the image of the image display section captured by the camera, the coordinates of the display cursor image at that time are acquired from the image display device by infrared rays, wireless, or the like. By comparing the coordinates of the acquired display cursor image with the coordinates of the captured cursor image captured by the camera, that is, by calculating a determinant indicating the projection relationship of the image, the orientation of the camera with respect to the image display unit, and the image display The distance between the camera and the camera, that is, the direction of the remote control and the remote control relative position can be detected.

  The display state of the display image displayed on the image display unit is set based on the direction of the remote controller and the relative position of the remote controller.

  In other words, the image generation unit displays the display state of the display image on the display screen of the image display unit so that the display image is displayed at a position closest to the remote control. Set based on.

  Further, the image generation unit is configured to display a display image display state relative to the remote control so that the display image is displayed larger as the distance between the image display unit and the remote control increases. Set based on.

  According to the present invention, an image displayed on the image display unit is captured by a camera provided in the remote controller. Then, the orientation of the remote controller and the relative position of the remote controller are compared by comparing the coordinates of a predetermined captured image (such as a captured cursor image) in the captured image with the coordinates of a predetermined display image (such as a display cursor image) sent from the image display device. Detect position. With this configuration, the direction of the remote controller and the remote controller relative position can be detected with high accuracy. Further, since it is only necessary to provide an inexpensive camera, it can be configured at low cost.

  FIG. 1 is a configuration diagram of an image display system according to an embodiment of the present invention. This image display system includes a TV 1 that is an image display device and a remote controller 2. The image display device can be configured by a device such as a personal computer including an image display unit instead of the TV 1.

  The TV 1 includes an image display unit 10, an image generation unit 11, a communication unit 12 that performs communication with the remote controller 2, and a control unit 13 that performs overall control of the TV 1.

  The image display unit 10 can display a TV image of the channel selected by the remote controller 2 on the display screen 15 or a menu image as shown in the figure. Also, when there is an electronic mail function, a mail arrival guidance character can be displayed, and when there is a TV phone function, the face image of the other party can be displayed.

  When the menu image is displayed, a cursor image 14 as shown is displayed. By moving the cursor image 14 to an arbitrary position by the remote controller 2 and further performing a predetermined key operation, a command instructed by the cursor image 14 can be passed to the control unit 13.

  The remote controller 2 compares a CCD camera (hereinafter referred to as a camera) 20, a memory 21 for storing a reference cursor image, and coordinates of a captured cursor image captured by the camera 20 with coordinates of a display cursor image transmitted from the TV 1. Then, the image processing unit 22 including the position detection unit 23 that detects the direction of the remote control and the remote control relative position (distance between the image display device 1 and the remote control 2), the operation unit 24, the communication unit 25, and the control unit 26 With.

  The operation unit 24 includes a plurality of keys, and transmits a command operated with each key to the TV 1 via the communication unit 25.

  The communication unit 25 transmits / receives data to / from the communication unit 12 of the TV 1 using infrared (IrDA) or radio waves of several GHz band.

  The control unit 26 performs overall control of the remote controller 2.

  The image processing unit 22 includes a position detection unit 23 that performs calculations for detecting the direction of the remote control and the remote control relative position.

  The image display system in FIG. 1 includes a first function that is a normal function of the remote controller 2 and a second function.

  In the first function, an operation content (operation command) operated by the operation unit 24 is transmitted to the TV 1.

  In the second function, the direction of the remote control and the remote control relative position calculated by the position detection unit 23 of the image processing unit 22 are transmitted to the TV 1 by the communication unit 25.

  In the second function, the entire image displayed on the image display surface 15 of the image display unit 10 is captured by the camera 20, and the entire image captured by the camera 20 and the reference cursor image stored in the memory 21 are displayed. In comparison, the position of the cursor image (captured cursor image) in the captured image is extracted. Next, the coordinates of the display cursor image are acquired from the TV 1, and the coordinates of the display cursor image and the coordinates of the imaging cursor image are compared to detect the direction of the remote control and the remote control relative position.

  The TV 1 receives the direction of the remote controller detected by the remote controller 2 and the remote controller relative position by the communication unit 12, and the image generation unit 11 sets the display state of the display image. Setting the display state of the display image means changing the size of characters, changing the display position of the menu image, and the like.

  FIG. 2 shows a display cursor image (FIG. 2A) displayed on the TV 1 and a captured cursor image captured by the camera 20 (FIG. 2B). The memory 21 also stores a reference cursor image similar to the display cursor image (FIG. 2A). However, coordinates are not stored. The coordinates shown in FIG. 2A are transmitted from the TV 1 as will be described later.

  The image processing unit 22 extracts the captured cursor image 14b from the image captured by the camera 20 by a known image recognition method using the reference cursor image. Further, the coordinates of the display cursor image at this time are inquired to the TV 1 and acquired. Here, the coordinates (x1, y1) (x2, y2) (x3, y3) on the display screen 15 representing the feature points of the cursor image are acquired. The coordinates of the tip position of the display cursor image 14a are (x1, y1) here.

  In the image processing unit 22, the coordinates (x1, y1) (x2, y2) (x3, y3) of the three feature points on the display screen 15 and the coordinates of the corresponding feature points on the image taken by the camera 20 are displayed. (X1 ′, y1 ′) (x2 ′, y2 ′) (x3 ′, y3 ′) are compared to calculate the projection relationship. By this calculation, the relationship between the coordinate system of the camera 20 and the coordinate system on the display screen 15 can be specified, and the remote control instruction position on the display screen 15 can be obtained. At the same time, the relative position and orientation of the remote controller 2 with respect to the display screen 15 can be detected.

  When a predetermined operation (for example, an operation for displaying a menu screen) is performed on the operation unit 24, the menu screen and the cursor image 14 are displayed on the display screen 15 of the TV 1. The cursor image at this time is displayed at the center position of the screen 15.

  In a state where the display cursor image 14 a is displayed at the center position of the display screen 15 of the TV 1, the position detection unit 23 of the image processing unit 22 uses the reference cursor image from among the images captured by the camera 20. 14b is extracted. Further, the coordinates of the display cursor image 14a are acquired from the TV1. The coordinates of the imaging cursor image 14b are compared with the coordinates of the display cursor image 14a. Thereby, the direction of the remote controller 2 can be detected.

  Further, the relative position of the remote controller (the distance between the display screen 15 and the remote controller 2) can be detected by comparing the reference cursor image 14a and the captured cursor image 14b.

  Therefore, the direction of the remote controller and the remote controller relative position are transmitted to the TV 1 by the communication unit 25. In the TV 1, the display position of the menu screen or the like is displayed at a position closest to the remote control based on the received direction of the remote control, or the character size is changed based on the remote control relative position. That is, the size of the character is increased as the remote control is further away from the TV 1.

  FIG. 3 is a diagram schematically illustrating a display example of the display screen 15.

  FIGS. 4A and 4B are display examples of a menu screen. FIG. 4A shows a display example when the remote controller 2 is separated (when the distance is long), and FIG. 4B shows a display example when the remote controller 2 is close (when the distance is short).

  The further away the remote controller 2 is from the TV 1, the larger the menu image. In addition, the display characters are large.

  3C and 3D are display examples of mail arrival images. FIG. 4C shows a display example when the remote controller 2 is separated (when the distance is long), and FIG. 4D shows a display example when the remote controller 2 is close (when the distance is short).

  The farther the remote controller 2 is from the TV 1, the larger the characters of the mail arrival image.

  FIGS. 3E and 3F are display examples when the TV phone function is selected. In the videophone function, the face image of the other party is displayed on the display screen 15.

  FIG. 5E shows a display example when the remote control 2 is shifted to the left with respect to the center position of the display screen 15, and FIG. 4F shows the center position of the display screen 15. Shows a display example when the remote control 2 is shifted to the right side of the screen.

  When the remote control 2 is shifted to the right side, the face image is also shifted to the right side. When the remote control 2 is shifted to the left side, the face image is also shifted to the left side.

  In FIGS. 3E and 3F, when the remote controller 2 is moved away from the TV 1, the displayed face image becomes larger. On the other hand, when the remote controller 2 approaches the TV 1, the displayed face image becomes smaller.

  The above control is performed by the image generation unit 11 of the TV 1 based on the direction of the remote control and the remote control relative position.

  In the present embodiment, the display of the cursor image is controlled so that the cursor image 14 follows the position change of the remote controller 2. Since the direction of the remote controller 2 is the same information as the position indicated by the remote controller on the display screen 15, the image generator 11 of the TV 1 displays the cursor image 14 on the display screen 15 based on the information on the direction of the remote controller 2. The display of the cursor image is controlled so as to follow the direction of the remote controller 2. Therefore, when the remote controller 2 is slightly moved or rotated toward the TV 1 side while the cursor image 14 is displayed at the center position of the display screen 15, the cursor image 14 follows the movement. When a specific key is operated on the operation unit 24 of the remote controller 2 with the cursor image 14 moved to a predetermined position, a command at a position designated by the cursor image 14 on the display screen 10 is input to the control unit 13. It becomes possible.

  FIG. 4 shows the indicated position of the remote controller 2 with respect to the display screen 15 of the TV 1. The line of sight L of the camera 20 built in the remote controller 2 is located at the tip of the cursor image 14 on the display screen 15. When the remote controller 2 is rotated in the arrow X direction in this state, the line of sight L is also rotated in the X direction accordingly, and the cursor image 14 displayed on the display screen 15 is moved accordingly.

  In the above embodiment, the cursor image 14 is displayed at the center position of the display screen 15 of the TV 1 when a predetermined operation (such as an operation for displaying a menu screen) is performed on the operation unit 24 of the remote controller 2. Accordingly, the cursor image 14 moves on the display screen 15 following the movement or rotation of the remote controller 2 after a predetermined operation is performed.

  Thus, by displaying the cursor image 14 at the center position of the display screen 15 when a predetermined operation is performed with the operation unit 24 of the remote controller 2, the field of view of the camera 20 can be obtained even when the remote controller 2 is quite close to the TV 1. Since the cursor image 14 always enters the inside, the extraction of the cursor image 14 in the image processing unit 22 is sure.

  When the viewing angle of the camera 20 is very wide, the initial position of the cursor image 14 when a predetermined operation is performed on the operation unit 24 does not need to be the center. For example, the center position of the area showing the menu on the left side of FIG. 1 can be set as the initial position. The initial position may be arbitrary.

  As another embodiment, the screen size and screen shape of the image display unit 10 may be stored in the memory 21 together with the reference cursor image as a reference predetermined image. FIG. 5A shows the screen size and screen shape of the image display unit 10. The memory 21 stores this information and the reference cursor image together. FIG. 5B shows a captured image captured by the camera 20. Thus, by storing the screen size and the screen shape of the image display unit 10 in the memory 21 as the reference predetermined image, as long as the entire image display unit 10 is displayed in the field of view V of the camera 20, the remote control instruction position Can be calculated. Specifically, three feature points are extracted and coordinates are set according to the size and shape of the screen. The image display unit is extracted from the image of the camera 20 using the screen size and screen shape of the image display unit 10 stored in the memory 21. The coordinates of the three points corresponding to the feature points of the extracted image display unit are obtained, and the remote control instruction position is calculated by calculating the projection relationship.

  Further, the size of the cursor image 14 displayed on the display screen 15 can be controlled based on the relative position of the remote controller. That is, it is possible to increase the visibility of the cursor image 14 by the camera 20 by decreasing the size of the cursor image 14 if the distance is short, and increasing the size of the cursor image 14 as the distance increases. The detection accuracy of the remote control indication position can be increased.

  6A and 6B are diagrams illustrating the size of the cursor image 14 with respect to the position of the remote controller 2. FIG. In FIG. 5B, the cursor image 14 is relatively large because the remote control 2 is separated from the TV 1. Note that, as described above, changing the cursor position according to the position of the remote controller 2 also has an effect of improving the visibility of the operator at a long distance or a short distance. The distance obtained as described above can be used not only for controlling the size of the cursor image 14 but also for controlling the volume of the TV 1. That is, when the distance is increased, the volume of the TV 1 is increased, and when the distance is decreased, the volume is decreased accordingly. In addition, various controls are possible depending on the distance between the TV 1 and the remote controller 2.

  Next, the operation of the control unit 26 of the remote controller 2 will be described.

  7 and 8 are flowcharts showing an outline of the operation of the control unit 26.

  In FIG. 7, when there is a key operation on the operation unit 24, it is determined whether or not the key operation is a menu key operation (ST1). If the operation is other than the menu key operation such as a normal channel key operation, a command corresponding to the key operation is transmitted to the TV 1 (ST2).

  If it is a menu key operation in ST1, the camera 20 captures an image displayed on the front, that is, the image display unit 10 of the TV 1 (ST3). Further, based on the captured image and the reference cursor image stored in the memory 21, an operation for calculating the direction of the remote controller 2 and the remote controller relative position is performed (ST4). Then, the direction of the remote control and the remote control relative position obtained in ST4 are transmitted to the TV 1 (ST5).

FIG. 8 shows the operation of ST4.

  First, a camera captured image captured by the camera 20 is acquired (ST10). Pattern matching between this image and the reference cursor image stored in the memory 21 is performed (ST11). The coordinates of the imaging cursor image 14b in the camera captured image are detected by the pattern matching (ST12). At this time, as for the imaging cursor image 14b, as shown in FIG. 2, the coordinates of at least three points are detected (ST12). And the coordinate of the display cursor image 14a at that time is received from TV1. From the coordinates of the display cursor coordinates 14a and the coordinates of the imaging cursor image 14b, the direction of the remote controller 2 and the remote controller relative position are obtained (ST13). As described above, the orientation of the remote control 2 on the display screen 15 can be obtained by calculation if the coordinates are three or more points, and further the remote control relative position (distance to the TV 1 of the remote control 2) is obtained by calculation. Can do. The accuracy of this calculation increases as the number of points (number of coordinate points) used in the calculation increases. However, since the calculation amount increases accordingly, the burden on the control unit 26 becomes heavy.

  In the TV 1, the communication unit 12 receives data on the direction of the remote controller 2 and the relative position of the remote controller. Based on the received data, the image generator 11 sets the display state of the display image on the display screen 15 as shown in FIG. That is, a character and a display image are set larger as the remote control 2 is farther from the TV 1, the left and right positions of the remote control 2 are determined based on the display screen, and the display image is displayed at a position closest to the remote control 2.

  In the present embodiment, the image generation unit 11 also controls the display position of the cursor image so that the cursor image 14 follows the remote controller 2.

  Note that once the orientation and relative position of the remote controller 2 can be detected, the data is stored in the control unit 13 of the TV 1. Therefore, even if the menu image is erased and the cursor image is no longer displayed, the display state of the display image is set based on the stored data. When the menu image is selected again, the direction and relative position of the remote controller 2 are detected again, and the data is replaced with the data stored so far.

  As described above, the memory 21 may store the menu image as a reference menu image (reference predetermined image). In this case, in ST11 of FIG. 8, the camera captured image and the reference menu image stored in the memory 21 are pattern-matched. In ST12, the coordinates of the captured menu image are detected. In ST13, the direction of the remote controller 2 and the remote controller relative position are calculated based on the coordinates of the display menu image (transmitted from the TV 1) and the coordinates of the captured menu image. If the coordinates of the display menu image are fixed, these coordinates may be stored in the memory 21 in advance. In this case, it is not necessary to transmit this information from the TV 1.

  In another embodiment of the present invention, when a menu image is displayed on the TV 1, the menu image may be transmitted to the remote controller 2 and stored in the memory 21 as a reference menu image. In this way, even when there are a plurality of menu images, the reference menu image stored in the memory 21 can be appropriately set corresponding to each menu.

  In still another embodiment of the present invention, the position detection unit 23 included in the image processing unit 22 of FIG. 1 can be provided on the TV 1 side.

  In this embodiment, the remote controller 2 does not require the memory 21 and the position detection unit 23. On the other hand, the communication unit 25 needs to have a function of transmitting an image captured by the camera 20 to the TV 1 at high speed. That is, the communication unit 25 of the remote controller 2 and the communication unit 12 of the TV 1 are required to have a high-bandwidth communication function. The TV 1 compares the captured image received by the communication unit 12 with the display image displayed on the display screen 15 at that time, and detects the direction of the remote control and the remote control relative position. This detection method is the same as in the above embodiment.

  In the above embodiment, the shape of the cursor image 14 is an arrow shape, but a triangular shape, a palm shape, or any other shape can be applied.

Configuration diagram of an image display system according to an embodiment of the present invention Diagram showing the display cursor image and the imaging cursor image Figure showing a screen display example The figure explaining the state which the cursor image 14 follows with respect to rotation of the remote control 2. The figure which shows the screen size of an image display part, a screen shape, and the captured image with the camera 20 The figure explaining the size of the cursor image with respect to the remote control position Flow chart showing schematic operation of remote control 2 Flow chart showing schematic operation of remote control 2

Explanation of symbols

1-TV
2-Remote control 10-Image display unit 14-Cursor image 15-Display screen 20-Camera (CCD)
22-Image processing unit 23-Position detection unit

Claims (6)

An image display device including an image generation unit that generates an image including a cursor image, and an image display unit that displays the generated image;
In an image display system comprising: an operation unit that controls display of the image display device; and a remote controller that transmits an operation content operated by the operation unit to the image display device.
The remote control is
A camera capable of capturing an image display unit when the remote control is pointed at the image display device;
A memory for storing in advance a reference cursor image serving as a reference for the cursor image;
Using the reference cursor image stored in the memory, the imaging cursor image in the captured image captured by the camera is extracted, and the coordinates of the display cursor image displayed on the image display unit and the coordinates of the imaging cursor image And a position detection unit that detects the orientation of the remote control relative to the image display unit and a remote control relative position that represents the distance between the image display unit and the remote control;
A transmission unit that transmits the remote control direction and the remote control relative position detected by the position detection unit to the image display device;
The image generation unit of the image display device includes:
Based on the direction of the remote control and the remote control relative position transmitted from the transmission unit, the display state of the display image is displayed so that the display image is displayed at a position closest to the remote control on the display screen of the image display unit Is set based on the orientation of the remote control transmitted from the transmission unit, and the display state of the display image is set so that the display image is displayed larger as the distance between the image display unit and the remote control increases. An image display system that is set based on a remote control relative position transmitted from a transmission unit. An image display device including an image generation unit that generates an image including a predetermined image, and an image display unit that displays the generated image;
In an image display system comprising: an operation unit that controls display of the image display device; and a remote controller that transmits an operation content operated by the operation unit to the image display device.
The remote control is
A camera capable of capturing an image display unit when the remote control is pointed at the image display device;
The coordinates of the predetermined display image displayed on the image display unit are compared with the coordinates of the predetermined image captured in the captured image captured by the camera, the direction of the remote control with respect to the image display unit, the image display unit, A position detection unit for detecting a remote control relative position representing a distance between the remote controls;
A transmission unit that transmits the remote control direction and the remote control relative position detected by the position detection unit to the image display device;
The image generation unit of the image display device sets a display state of a display image to be displayed on the image display unit based on a remote control direction and a remote control relative position transmitted from the transmission unit. Image display system. The remote control includes a memory that stores in advance a reference predetermined image serving as a reference for the predetermined image,
The image display system according to claim 2, wherein the position detection unit extracts a captured predetermined image in the captured image using a reference predetermined image stored in the memory.   The image display system according to claim 3, wherein the reference predetermined image is a reference cursor image.   The image generation unit is configured to display the display state of the display image based on the direction of the remote control transmitted from the transmission unit so that the display image is displayed at a position closest to the remote control on the display screen of the image display unit. The image display system according to claim 2, wherein the image display system is set.   The image generation unit is configured to display the display state of the display image based on the remote control relative position transmitted from the transmission unit so that the display image is displayed larger as the distance between the image display unit and the remote control increases. The image display system according to claim 2, wherein the image display system is set.

Images