JP4221445B1 - Videophone terminal, videophone terminal control method and program - Google Patents

Abstract

Provided is a user interface device with good operability in which the relationship between a subject photographed by a camera and menu items can be easily grasped on a screen.
A user interface unit 210 according to the present invention displays a camera 113 for photographing a subject, the subject photographed by the camera 113, and displays phone book menu items 504a to 504c on the subject. A sound output unit 219; a position detection unit 213 that detects the position of an object that is a part of the subject photographed by the camera 113; and if the object is located in an area corresponding to the menu item, the telephone directory A menu processing unit 217 for selecting menu items 504a to 504c.
[Selection] Figure 4

Description

  The present invention relates to a user interface device that receives a user's operation provided in a device having a display unit such as an information terminal device such as a computer or a videophone terminal and selects a menu item.

  There has been proposed a user interface device that recognizes an operator's hand and displays a corresponding icon on a monitor to select a menu item (see, for example, Patent Document 1).

  In the user interface device described in Patent Literature 1, an operator's hand is photographed with a camera, and menu items displayed on the monitor are selected with icons generated based on the movement and shape characteristics of the hand. Thereby, the operator does not need an input device such as a keyboard or a mouse, and can easily operate the device.

On the other hand, in the videophone terminal, since the operator operates the input device to call the other party's videophone terminal, a simpler user interface is required for elderly people who are not good at operating the device.
Japanese Patent No. 3777650

  However, in the user interface device described in Patent Document 1, only menu items and icons are displayed on the monitor. As a result, when the icon is not accurately displayed at the position of the hand, it is difficult for the operator to grasp the relationship between the position of his hand and the position of the icon displayed on the monitor, and the operability of the icon is poor. There's a problem.

  Accordingly, an object of the present invention is to provide a user interface device, a videophone terminal, a user interface control method, and a program with good operability, in which the relationship between a subject and menu items can be easily grasped on a screen.

  In order to achieve the above object, a user interface device according to the present invention includes a camera that captures a subject, a display unit that displays the subject captured by the camera, and displays at least one menu item on the subject. A position detection unit that detects a position of an image representing an object that is a part of the subject included in the subject photographed by the camera, and an overlap between the image representing the object and an area corresponding to the menu And a menu processing unit for selecting the menu item accordingly.

  According to this configuration, the user interface device according to the present invention can simultaneously display the menu item and the subject photographed by the camera on the monitor. As a result, the menu item and the subject are overlapped and displayed on the monitor, and the operator using the user interface device according to the present invention can easily grasp the positional relationship between the video and the menu item, and the operability is improved. .

  Furthermore, in the user interface device according to the present invention, when an image representing a predetermined object photographed by the camera is located in an area corresponding to the menu item, the menu item can be selected. As a result, the operator does not need an input device such as a keyboard or a mouse, and can select menu items and easily operate the device.

  The display unit may further display a pointer at a position of an image representing the object.

  According to this configuration, the user interface device according to the present invention can simultaneously display the pointer indicating the position of the image representing the predetermined object, the menu item, and the subject on the monitor. Thus, the operator can easily recognize which part of the image representing the object that is a part of the subject displayed on the monitor corresponds to which menu item is selected.

  The display unit may transmit the subject that overlaps the menu item.

  According to this configuration, in the user interface device according to the present invention, a part of the subject overlapping the menu item is displayed. Thereby, even when the menu item and the image representing the object overlap, the operator can easily recognize the overlap of the menu item and the image representing the object.

  The menu processing unit may select the menu item when an image representing the object and an area corresponding to the menu item overlap each other for a predetermined time.

  According to this configuration, in the user interface device according to the present invention, the menu item can be selected even when the image representing the object does not overlap the menu item. Thereby, a menu item close to an image representing an object can be selected, and operability is improved.

  Further, the position detection unit identifies a shape of the object, and the user interface device further includes a shape determination unit that determines whether or not the shape of the object is a predetermined shape. The menu processing unit may select the menu item when the shape determining unit determines that the shape is a predetermined shape.

  According to this configuration, the operator who uses the user interface device according to the present invention can select the menu item by making the object a predetermined shape in the area corresponding to the menu item. This makes it easier for the operator to obtain a menu item selection feeling.

  The display unit may further display a pointer, and the user interface device may further include a pointer moving unit that controls movement of the pointer in response to movement or shape of an image representing the object. .

  According to this configuration, the operator who uses the user interface device according to the present invention can move the pointer by changing the movement or shape of the object. Thereby, an erroneous operation can be prevented.

  In addition, the pointer moving unit controls the pointer to be displayed at a predetermined position, and after the position of the image representing the object overlaps the position of the pointer, the position of the image in which the pointer represents the object The pointer may be controlled to be displayed at a predetermined position after the menu item is selected.

  According to this configuration, in the user interface device according to the present invention, after the menu processing unit selects the menu item, the position of the pointer is displayed at a predetermined position.

  This prevents the menu item from being selected when the menu item is selected while the pointer remains displayed at the same position after the menu item is selected.

  The menu processing unit may select the menu item according to the time when the pointer overlaps the area corresponding to the menu item or the behavior of the object when the pointer overlaps.

  According to this configuration, in the user interface device according to the present invention, the pointer displayed at the position of the image representing the object is overlapped with the area corresponding to the menu for a predetermined time, or according to the behavior when the pointer is overlapped. The menu can be selected. Thereby, an erroneous operation can be prevented.

  A videophone terminal according to the present invention includes the user interface device, and the menu item relates to an operation of the videophone terminal.

  According to this configuration, the videophone terminal according to the present invention does not require an external input device that receives an instruction from the operator. In addition, the menu item and the subject photographed by the camera can be displayed on the monitor at the same time.

  Thereby, the menu item and the subject are displayed on the monitor so as to overlap each other, and the operator using the videophone terminal according to the present invention can easily grasp the positional relationship between the video and the menu item, and the operability is improved. .

  Further, when a predetermined object photographed by the camera is located in an area corresponding to the menu item, the menu item can be selected.

  As a result, the operator does not need an input device such as a keyboard or a mouse, and can select menu items and easily operate the device. Therefore, an elderly person who is not good at operating the device can easily perform videophone communication.

  Furthermore, since the videophone terminal is originally provided with a camera and a monitor, it is preferable to use the user interface device according to the present invention. Thereby, the cost increase by introduction can be suppressed.

  The videophone terminal may further include a selection prohibiting unit that prohibits selection of the menu item by the menu processing unit while the videophone terminal is in a call with another videophone terminal.

  According to this configuration, in the videophone terminal according to the present invention, the menu item is not selected during communication with another videophone terminal. This prevents a menu item from being erroneously selected by an operator's gesture that naturally occurs during conversation during communication.

  The display unit may further display a pointer at a position of an image representing the object, and the selection prohibition unit may further prohibit display of the pointer by the display unit during the call.

  According to this configuration, the videophone terminal according to the present invention does not display a pointer during communication with other videophone terminals. This prevents the pointer from moving in the screen due to an operator's gesture that naturally occurs during conversation during communication, and deterioration of the visibility of the screen.

  Further, the videophone terminal further includes an image change determination unit that determines whether or not the subject photographed by the camera during the call has a change in a predetermined time, and the image change determination unit When it is determined that there is no change, an end executing unit that ends the call may be provided.

  According to this configuration, in the videophone terminal according to the present invention, if the subject does not change within a predetermined time during the videophone communication, the videophone communication is terminated. Thus, the operator can end the videophone communication without performing a special operation.

  In addition, the videophone terminal further includes a voice acquisition unit that acquires voice, and whether or not the voice acquired by the voice acquisition unit during the call has a predetermined volume or more at a predetermined time. A voice determination unit for determining whether or not the voice determination unit determines that the volume is not equal to or higher than a predetermined volume.

  According to this configuration, in the videophone terminal according to the present invention, when the voice of the operator is acquired during the videophone communication and the voice does not have a predetermined volume within a predetermined time, the videophone communication Exit. Thus, the operator can end the videophone communication without performing a special operation.

  The videophone terminal further includes an end image determination unit that determines whether or not a predetermined end image is included in the subject photographed by the camera during the call, and the end image determination unit And an end executing unit that ends the call.

  According to this configuration, in the videophone terminal according to the present invention, when the operator displays a predetermined end image within the shooting range of the camera during the videophone communication, the videophone communication is terminated. Thus, the operator can end the videophone communication without using an external input device.

  The present invention can be realized not only as such a user interface device and a videophone terminal, but also as a control method and program for controlling the user interface device and the videophone terminal. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM and a transmission medium such as the Internet.

  As described above, the present invention provides a user interface device and a videophone terminal with good operability that enable the user to easily grasp the relationship between the menu item and the icon displayed at the position of the subject photographed by the camera and the operator's hand. A user interface control method and program can be provided.

  Hereinafter, embodiments of the videophone terminal according to the present invention will be described in detail with reference to the drawings.

(Embodiment 1 of the present invention)
The videophone terminal according to Embodiment 1 of the present invention displays a subject photographed by the camera of the videophone terminal as a self-portrait on a monitor. The videophone terminal displays a menu item on the self-portrait and a pointer at the position of the hand included in the self-portrait. The video phone terminal selects the menu item when a predetermined time pointer is stopped in the area corresponding to the menu item. Thereby, it becomes easy to grasp the positional relationship between the menu item displayed on the self-portrait, the pointer, and the hand of the operator, and the operability is improved.

  FIG. 1 is a diagram showing a system configuration of the videophone terminal 110 according to Embodiment 1 of the present invention.

  A videophone system 100 shown in FIG. 1 includes a plurality of videophone terminals 110A and 110B, a management server 130, and general terminals 140A and 140B. In addition, when not distinguishing the plurality of videophone terminals 110A and 110B in particular, it is denoted as the videophone terminal 110, and when not distinguishing the plurality of general terminals 140A and 140B in particular, it is denoted as the general terminal 140.

  The video phone terminal 110, the general terminal 140, and the management server 130 are connected via a network 120 such as the Internet. In FIG. 1, two videophone terminals 110 and two general terminals 140 are connected via the network 120, but each of the videophone terminal 110 and the general terminal 140 connected to the network is connected to each other. The number may be any number.

  In the videophone system 100, videophone communication for transmitting and receiving video and audio is performed between the videophone terminal 110 and the general terminal 140.

  The general terminal 140 is a videophone terminal operated by a general user. The general terminal 140 is, for example, a personal computer.

  The management server 130 is a computer or the like that controls videophone communication between the videophone terminal 110A and the videophone terminal 110B. Specifically, the management server 130 allocates and manages the IP addresses of the videophone terminals 110 and the general terminals 140 included in the videophone system 100. In addition, the management server 130 controls videophone communication connections in response to requests from the videophone terminals 110 and the general terminals 140.

  The video phone terminal 110 is a video phone terminal operated by a computer weak person such as an elderly person. Specifically, a set top box (STB) 111, a monitor 112, a camera 113, a speaker 114, and a microphone 115 are provided.

  The STB 111 is a computer having various communication I / Fs that perform videophone communication with other videophone terminals 110 and general terminals 140 via the network 120 under the control of the management server 130. The STB 111 is also connected to the monitor 112, camera 113, speaker 114, and microphone 115, and controls the monitor 112, camera 113, speaker 114, and microphone 115.

  The monitor 112 is connected to the STB 111, and is captured by the camera 113 and a partner image transmitted from another videophone terminal 110 and a general terminal 140 (hereinafter referred to as a partner terminal) that are in videophone communication with the videophone terminal 110. The self-portrait that is the subject to be displayed is displayed. The monitor 112 displays a pointer at the position of the hand in the self-portrait. The monitor 112 is, for example, a television used in a general home.

  The camera 113 is connected to the STB 111 and photographs the state of the operator who is operating the videophone terminal 110.

  The speaker 114 outputs sound transmitted from the partner terminal during videophone communication. The speaker 114 emits a sound when a menu item is selected when videophone communication is not being performed (hereinafter referred to as non-communication). Note that the speaker 114 may be, for example, a speaker built in a television.

  Microphone 115 acquires the voice to be transmitted to the partner terminal during videophone communication.

  Next, a detailed configuration of the videophone terminal 110 will be described.

  FIG. 2 is a block diagram showing the configuration of the STB 111 provided in the videophone terminal 110.

  The STB 111 includes a user interface unit 210, a voice acquisition unit 221, a transmission / reception unit 223, and a communication control unit 225.

  The STB 111 includes a processor such as a CPU and a storage unit such as a RAM, a hard disk, or a nonvolatile memory. The functions of the processing units (user interface unit 210, voice acquisition unit 221, transmission / reception unit 223, and communication control unit 225) illustrated in FIG. 2 are realized by the processor executing a program stored in the storage unit. . In addition, the holding and storing functions by the processing units illustrated in FIG. 2 are realized using the storage unit. Note that some or all of the processing units illustrated in FIG. 2 may be realized by a dedicated circuit included in the STB 111.

  The user interface unit 210 detects a hand from image data that is a subject photographed by the camera 113 and displays a pointer to the hand position of the self-portrait. When the pointer is located in the area corresponding to the menu item, the user interface unit 210 selects the menu item. The user interface unit 210 includes an image acquisition unit 211, a position detection unit 213, a pointer movement unit 215, a menu processing unit 217, and a display sound output unit 219.

  The image acquisition unit 211 acquires image data that is a subject photographed by the camera 113.

  The position detection unit 213 is a processing unit that extracts hand image data from the image data and obtains hand coordinates.

  The pointer moving unit 215 outputs pointer data for displaying the pointer at the coordinates output by the position detecting unit 213 to the display sound output unit 219.

  The menu processing unit 217 selects the menu item when the pointer is located in an area corresponding to the menu item. The menu processing unit 217 outputs menu item display data, which is display information of menu items displayed on the monitor 112, to the display sound output unit 219. A specific configuration of the menu processing unit 217 will be described later.

  The audio acquisition unit 221 acquires audio data that is a sound acquired by the microphone 115.

  The transmission / reception unit 223 transmits and receives control signals such as image data, audio data, and a call signal to and from the partner terminal via the network 120 and the management server 130.

  The communication control unit 225 controls videophone communication between the videophone terminal 110 and the partner terminal. Specifically, the communication control unit 225 transmits the image data acquired by the image acquisition unit 211 and the audio data acquired by the audio acquisition unit 221 to the partner terminal via the transmission / reception unit 223 during videophone communication. Further, the communication control unit 225 outputs the audio data and image data received from the counterpart terminal during the videophone communication to the display sound output unit 219.

  The display sound output unit 219 displays menu items, pointers, a self-portrait and a partner image on the monitor 112 during videophone communication, and outputs audio data received from the partner terminal to the speaker 114. The self-portrait is image data acquired by the image acquisition unit 211. The partner image is image data received from the partner terminal. Further, at the time of non-communication, the display sound output unit 219 displays the menu item, the pointer, and the image data own image acquired by the image acquisition unit 211 on the monitor 112.

  Next, a detailed configuration of the menu processing unit 217 will be described.

  FIG. 3 is a block diagram illustrating a configuration of the menu processing unit 217.

  The menu processing unit 217 selects the menu item when the pointer is located in an area corresponding to the menu item. The menu processing unit 217 includes a menu storage unit 301, a mutual coordinate determination unit 303, a selection determination unit 305, a menu execution unit 307, and a menu display unit 309.

  The menu storage unit 301 stores menu item information corresponding to each menu item. The menu item information includes icon image data, a first area and a second area of the icon image, and an IP address. The icon image data is image data corresponding to each menu item displayed on the monitor 112. The area where the icon image is displayed on the self-portrait is the first area. In the self-portrait, the area obtained by dividing the screen by a set of points at which the distance to the first area of adjacent menu items is equal is the second area. Details of the first region and the second region will be described later.

  The mutual coordinate determination unit 303 determines whether or not the pointer is positioned in the second area of the menu item. The selection determination unit 305 determines whether a menu item has been selected.

  The menu execution unit 307 reads the menu item information from the menu storage unit 301 and outputs the first area and the icon image to the menu display unit 309. In addition, when the selection determination unit 305 outputs information indicating a menu item, the menu execution unit 307 executes processing corresponding to the menu item.

  The menu display unit 309 switches display of menu items according to information output from the mutual coordinate determination unit 303 and the menu execution unit 307.

  From the above, the menu processing unit 217 highlights the menu item closest to the pointer displayed on the monitor 112, and selects the menu item when the pointer stops on the same menu item for a predetermined time. To do.

  Next, an image displayed on the monitor 112 in Embodiment 1 of the present invention will be described.

  FIG. 4 is a diagram showing a display example of the monitor 112 during non-communication using the videophone terminal 110.

  As shown in FIG. 4, the display sound output unit 219 displays a self-portrait 501, a pointer 503, and telephone directory menu items 504 a to 504 c on the monitor 112. A pointer 503 and telephone directory menu items 504 a to 504 c are displayed on the self-portrait 501. The pointer 503 moves following the hand image in the self-portrait 501. In FIG. 4, “eldest son”, “eldest daughter”, and “manager” are displayed as call partners registered in advance in the telephone directory menu items 504a to 504c.

  Specifically, the position detection unit 213 extracts hand image data in the image data output from the image acquisition unit 211 using, for example, color information, calculates the center of gravity of the extracted hand image data, and calculates the self-image. Convert to pointer coordinates which is the coordinate system of. Note that the object detected by the position detection unit 213 may be a predetermined object (for example, a card or a pen) without being a hand, or may be a part of the body other than the hand. The pointer moving unit 215 outputs pointer data for displaying a pointer icon at the pointer coordinates to the menu processing unit 217 and the display sound output unit 219. The icon of the pointer 503 is displayed in front of the self image 501 and the menu item, and does not transmit the self image 501 and the menu item overlapping the pointer 503.

  Note that the icon of the pointer 503 may display only a frame and the inside may be transparent. The icon of the pointer 503 may be wholly or partially translucent.

  The phone book menu items 504a to 504c display a call partner registered in advance, and the phone book menu items 504a to 504c closest to the pointer 503 are selected. That is, when the coordinates of the pointer 503 are located in the second area of the phone book menu items 504a to 504c, the phone book menu items 504a to 504c are selected.

  The menu processing unit 217 performs processing of the phone book menu items 504a to 504c. In addition, menu items other than the phone book menu items 504a to 504c (communication start confirmation menu item, end menu item, end confirmation menu item, and incoming call menu item) are also processed in the same manner as the phone book menu items 504a to 504c. Performed by the unit 217.

  The menu processing unit 217 selects the menu item when the pointer is positioned in the second area of the menu item. The menu processing unit 217 includes a menu storage unit 301, a mutual coordinate determination unit 303, a selection determination unit 305, a menu execution unit 307, and a menu display unit 309.

  The menu storage unit 301 stores menu item information corresponding to each menu item. The menu item information includes icon image data, a first area and a second area of the icon image, and an IP address. The icon image data is image data corresponding to each menu item displayed on the self-image.

  The mutual coordinate determination unit 303 determines whether or not the hand is located in an area corresponding to the menu item. Specifically, the mutual coordinate determination unit 303 determines whether or not the pointer coordinates output from the position detection unit 213 are located in the second area of the menu item stored in the menu storage unit. When the pointer coordinates are located in the second area of the menu item, the mutual coordinate determination unit 303 outputs information indicating the menu item to the selection determination unit 305 and the menu display unit 309.

  For example, in FIG. 4, since the position of the hand (that is, the pointer coordinate) is located in the second area of the phone book menu item 504b indicating "eldest daughter" among the phone book menu items 504a to 504c, the mutual coordinate determination unit 303 outputs the information of the telephone directory menu item 504 b indicating “elder daughter” to the selection determination unit 305 and the menu display unit 309.

  The selection determination unit 305 determines whether a menu item has been selected. Specifically, when the pointer coordinates output from the position detection unit 213 do not move for a predetermined time and information indicating a menu item is sent from the mutual coordinate determination unit 303, the selection determination unit 305 receives the menu. It is determined that the item has been selected, and information indicating the menu item is output to the menu execution unit 307.

  For example, in FIG. 4, when the position of the operator's hand stops for a predetermined time, the selection determining unit 305 determines that the operator has selected the phone book menu item 504 b indicating “elder daughter”, and “ Information indicating that the telephone directory menu item 504 b indicating “the eldest daughter” has been selected is output to the menu execution unit 307.

  The menu execution unit 307 reads the menu item information from the menu storage unit 301 and outputs the first area and the icon image to the menu display unit 309. In addition, when the selection determination unit 305 outputs information indicating that a menu item has been selected, the menu execution unit 307 executes processing according to the menu item.

  For example, in FIG. 4, when the position of the operator's hand stops for a predetermined time, the menu execution unit 307 executes a process according to the phone book menu item 504b.

  The menu display unit 309 holds image data of menu items to be output to the display sound output unit 219. Specifically, menu item display data indicating an icon image corresponding to the menu item output by the menu execution unit 307 is output to the display sound output unit 219.

  When the mutual coordinate determination unit 303 determines that the pointer coordinates are located in the second area, the menu display unit 309 switches the menu item display data of the menu item to highlight display and outputs it. Specifically, the highlight display of menu items is repeated display of enlargement display, color switching, blinking, and display size enlargement / reduction. Note that two or more highlight displays may be performed at the same time among repeated enlargement display, color switching, blinking, and display size enlargement / reduction.

  For example, in FIG. 4, the phone book menu item 504 b indicating “elder daughter” output from the mutual coordinate determination unit 303 is highlighted with a thick frame.

  From the above, the menu processing unit 217 highlights the menu item closest to the pointer 503, and executes the menu item when the pointer 503 stops on the same menu item for a predetermined time.

  In each menu item described below (communication start confirmation menu item, end menu item, end confirmation menu item, and incoming call menu item), the processing in the menu processing unit 217 is the same as that of the telephone directory menu items 504a to 504c.

  In addition, here, in order to prevent an operator's erroneous operation, the videophone communication is started by performing a two-step operation, and the operator selects the other party in the telephone directory menu items 504a to 504c. After that, by selecting a communication start menu item described later, videophone communication with the partner terminal is started.

  FIG. 5 is a diagram showing a display example of the self-portrait 501 when confirming the start of videophone communication.

  As shown in FIG. 5, a pointer 503 and communication start confirmation menu items 505a to 505b are displayed on the self-portrait 501.

  The communication start confirmation menu items 505a to 505b are displayed when the operator selects the other party in the phone book menu items 504a to 504c in FIG. The communication start confirmation menu items 505a to 505b include “Yes” and “No”. When the operator selects “Yes”, the videophone terminal 110 calls the other party selected in the phone book menu items 504a to 504c. Start. If the operator selects “No”, the videophone communication is not started, and the image of FIG.

  4 and 5, the size of the self-portrait 501 displayed on the monitor 112 is arbitrary.

  FIG. 6 is a diagram illustrating a display example of the monitor 112 during videophone communication using the videophone terminal 110.

  As shown in FIG. 6, the display sound output unit 219 displays a self-image 501, a pointer 503, an end menu item 506, and a partner image 511 on the monitor 112 during videophone communication. The display sound output unit 219 displays the image received from the counterpart terminal on the counterpart image 511 via the transmission / reception unit 223 and the communication control unit 225. A pointer 503 and an end menu item 506 are displayed on the self-portrait 501. Here, the processing of the end menu item 506 in the menu processing unit 217 is the same as that of the telephone directory menu items 504a to 504c.

  The end menu item 506 is selected when the operator ends the videophone communication.

  Here, similarly to the start of the videophone communication, in order to prevent an erroneous operation, the videophone communication is terminated by performing a two-step operation. The operator ends the videophone communication by selecting an end menu item 506 and an end confirmation menu item to be described later.

  FIG. 7 is a diagram showing a display example of the self-portrait 501 when confirming the end of the videophone communication.

  As shown in FIG. 7, a pointer 503 and end confirmation menu items 507 a to 507 b are displayed on the self-portrait 501.

  The end confirmation menu items 507a to 507b are menu items displayed when the operator selects the end menu item 506 in FIG. The end confirmation menu items 507a to 507b include “Yes” and “No”. When the operator selects “Yes”, the videophone terminal 110 disconnects the videophone communication, closes the screen of the partner image 511 displayed on the monitor 112, and in the same way as in FIG. Display. When the operator selects “No”, the videophone terminal 110 continues the videophone communication and displays the pointer 503 and the end menu item 506 on the self-portrait 501 as in FIG.

  The video displayed on the partner image 511 is the same as that shown in FIG.

  FIG. 8 is a diagram illustrating a display example of the self-portrait 501 when there is a call from another terminal during non-communication.

  As shown in FIG. 8, an incoming call menu item 508 is displayed on the self-portrait 501. In FIG. 8, the pointer 503 is not displayed because the hand image is not included in the self-portrait 501, but when it is included, the pointer 503 is displayed at the position of the hand.

  The incoming call menu item 508 is a menu item that informs that there is a call from another terminal. The operator selects the incoming call menu item 508 to permit the start of the videophone communication.

  Here, in order to prevent an erroneous operation by the operator, the start of the videophone communication is realized by performing a two-step operation. Therefore, the operator selects the incoming call menu item 508 and then selects the communication start confirmation menu items 505a and 505b, thereby starting videophone communication with the partner terminal.

  The image displayed on the monitor 112 in Embodiment 1 of the present invention has been described above.

  In addition, the menu display unit 309 displays menu items (specifically, communication start confirmation menu items 505a to 505b and end confirmation menu items 507a to 507b) displayed immediately before each in order to prevent erroneous operations. It is displayed at a different position from the menu item.

  FIG. 9 shows menu items (phonebook menu item 504b, communication start confirmation menu items 505a to 505b, end menu item 506, end confirmation menu items 507a to 507b, and incoming call menu item 508 displayed on the self-portrait 501 described so far. It is a figure which shows the display of () typically. 9 indicates that the self-portrait 501 in the background of the menu item is transmitted.

  FIG. 9A is a diagram illustrating menu items in which only characters and frames are displayed.

  FIG. 9B is a diagram illustrating menu items in which characters are displayed in a frame.

  FIG. 9C is a diagram showing menu items in which only characters are displayed.

  The display of the menu item is not limited to the above, and the menu item name may be displayed beside the frame so as to be transparent through the self-portrait 501.

  FIG. 10 is a diagram showing a first area 521 and a second area 522 of menu items stored in the menu storage unit 301.

  A region where the icon image is displayed on the self-portrait is a first region 521. In the self-portrait, a region obtained by dividing the screen by a set of points at which the distance to the first region 521 of adjacent menu items is equal is a second region 522.

  Specifically, the second area is an area obtained by dividing the screen by a set of points at which the distances to the adjacent first areas are equal. Thereby, the menu processing unit 217 determines that the pointer 503 corresponds to the menu item closest to the position where the pointer 503 exists, and highlights the menu item.

  Note that the second area may be a part of the area obtained by dividing the screen by a set of points at which the distance to the first area of adjacent menu items is equal. In addition, the second region may be a region having a predetermined width border on the periphery of the first region.

  FIG. 11 is a flowchart showing a flow of operations when the videophone terminal 110 is not in communication.

  First, the position detection unit 213 of the videophone terminal 110 determines whether or not there is a hand in the image output by the image acquisition unit 211 (S101). If the position detection unit 213 determines that there is no hand in the image (No in S101), the position detection unit 213 continues to detect the hand.

  If the position detection unit 213 determines that the image has a hand (Yes in S101), the hand coordinates are calculated (S102).

  The pointer moving unit 215 outputs pointer data including the coordinate data calculated by the position detecting unit 213 to the display sound output unit 219. The display sound output unit 219 displays the pointer at the coordinates indicated by the pointer data (S103).

  Next, the menu processing unit 217 determines whether or not the pointer is located in the second area of the menu item stored in the menu storage unit 301 (S104).

  If the menu processing unit 217 determines that the pointer is located in the second area of the menu item (Yes in S104), the menu processing unit 217 determines whether the pointer has stopped for a certain period of time (S105).

  When the menu processing unit 217 determines that the pointer has stopped for a certain period of time (Yes in S105), the menu item is selected (S106).

  As described above, according to the first embodiment of the present invention, a menu item and a pointer having a portion that transmits the self-portrait 501 are displayed on the self-portrait 501 that is a subject photographed by the camera 113. An input device such as a keyboard is not required, and the videophone terminal 110 with good operability can be provided.

  Note that a volume control indicating the volume during a call may be displayed on the self-portrait 501 so that the operator can adjust the volume.

  In addition, monitor control indicating the brightness of the screen may be displayed on the self-image 501 so that the operator can adjust the brightness.

  In addition, during communication, the operator may disconnect the volume, brightness, and videophone communication using a device (for example, a remote control or a button) connected to the STB 111.

  When the pointer is located in the second area of the menu item, the display sound output unit 219 may display the pointer in the first area of the menu item.

  In addition, when a call is received from another terminal, the display sound output unit 219 outputs an incoming call menu item 508 to the monitor 112 and also outputs a ring tone that notifies the speaker 114 that there is a call from another terminal. Good.

(Embodiment 2 of the present invention)
Next, a second embodiment of the present invention will be described.

  The videophone terminal according to Embodiment 2 of the present invention detects the operator's hand, moves the pointer to the coordinates according to the operator's hand, and the operator takes a predetermined hand shape. The menu item whose pointer is located in the second area is selected.

  The following description focuses on the differences from the videophone terminal 110 according to Embodiment 1 of the present invention.

  FIG. 12 is a block diagram showing a configuration of user interface unit 610 of the videophone terminal according to Embodiment 2 of the present invention. Here, the same blocks as those in FIG. Compared with the user interface unit 210 according to the first embodiment of the present invention, the user interface unit 610 newly includes a shape storage unit 611 and a shape determination unit 613.

  The user interface unit 610 moves the pointer to the coordinates corresponding to the hand of the operator of the videophone terminal, and when the operator takes a predetermined hand shape, the menu item in which the pointer is located in the second area is displayed. select.

  The shape storage unit 611 stores a predetermined hand shape meaning selection.

  The shape determination unit 613 determines whether the image data of the operator's hand extracted by the position detection unit 213 is a predetermined hand shape stored in the shape storage unit 611. When the shape determination unit 613 determines that the hand image data has a predetermined hand shape, the shape determination unit 613 outputs a signal indicating that the operator has performed a selection operation to the menu processing unit 217. When the selection determination unit 305 of the menu processing unit 217 receives a signal indicating that a selection operation has been performed, if information indicating a menu item is output from the mutual coordinate determination unit 303, information indicating the menu item is displayed. The data is output to the menu execution unit 307.

  Thereby, the user interface unit 610 can select the menu item when the operator makes a predetermined hand shape in the second area of the menu item. For example, in FIG. 4, the operator can select the telephone directory menu item 504 b indicating “eldest daughter” by changing the shape of the hand from an open state to a gripped state.

  FIG. 13 is a diagram illustrating a hand shape representing the selection operation of the operator stored in the shape storage unit 611. 13A shows a state where a hand is held, FIG. 13B shows a state where two fingers are raised, and FIG. 13C shows a state where one finger is raised.

  The user interface unit 610 selects the menu item when the hand shape is as shown in FIG.

  As described above, in the second embodiment of the present invention, when the operator's hand is detected and the pointer 503 is displayed at the position of the operator's hand, and the operator has a predetermined hand shape, the pointer Selects a menu item located in the second region. This makes it easier for the operator to obtain a menu item selection feeling.

  When the operator's hand overlaps the pointer, if the hand has a predetermined shape (for example, grasping), the pointer moves following the position of the hand, and then the hand has a predetermined shape. If it is no longer (for example, opened), the pointer may stop at that position.

  FIG. 14 is a diagram illustrating a state in which the pointer 711 moves when the operator holds his / her hand over the pointer 711.

  The pointer 711 can be moved when the operator puts his / her hand on the hand when the hand has a predetermined shape. The phone book menu items 712a to 712c have the same functions as the phone book menu items 504a to 504c according to the first embodiment of the present invention, but have different icons.

  FIG. 14A is a diagram showing that the pointer 711 is stationary because no hand is positioned on the pointer 711. At this time, the pointer moving unit 215 outputs pointer data of predetermined coordinates to the display sound output unit 219.

  FIG. 14B is a diagram illustrating a state where the operator holds the hand and moves the pointer 711.

  When the operator grips the hand, the position detection unit 213 outputs image data indicating the shape of the hand gripped. The shape determination unit 613 determines whether or not the image data obtained by grasping the hand is stored in the shape storage unit 611. If the image data is stored, a signal indicating that the operator has performed a selection operation is displayed on the menu. The data is output to the processing unit 217 and the pointer moving unit 215.

  The pointer moving unit 215 determines whether the hand coordinates output from the position detection unit 213 match the coordinates of the pointer 711. When the pointer moving unit 215 receives the signal indicating that the selection operation has been performed when it is determined that they match, the pointer moving unit 215 sets the coordinates of the hand as the coordinates of the pointer 711 so that the pointer 711 is displayed at the position of the hand.

  FIG. 14C is a diagram illustrating a state in which the operator opens his hand on the phone book menu item 712b indicating “elder daughter”, and the pointer 711 is displayed on the basket of the phone book menu item 712b.

  When the operator opens his hand, the shape determination unit 613 stops a signal indicating that the selection operation has been performed. The pointer moving unit 215 stops the pointer 711 when the signal indicating that the shape determining unit 613 has performed the selection operation is stopped. As a result, the coordinates of the pointer 711 are the positions where the operator has opened their hands.

  At this time, the menu processing unit 217 determines whether or not the position of the hand when the signal indicating the selection operation is stopped is located in the second area of the menu item, and if so, executes the menu item. .

  Further, after the menu item is selected, the pointer moving unit 215 sets the coordinates of the pointer 711 to a predetermined position.

  As described above, in the videophone terminal according to Embodiment 2 of the present invention, the operator moves and stops the pointer 711 by changing the position and shape of the hand, so that an erroneous operation can be prevented and an execution feeling can be improved. .

  Note that the shape of the hand indicating the selection operation may be either one of FIG. 13 (b) or FIG. 13 (c). The method of indicating the selection operation is not limited to the shape of the hand, and may be an operation of pushing the hand even if the hand is shaken or the hand is tilted. Further, a predetermined card or pen may be illustrated. Further, a part or the whole of the body other than the hand, for example, the arm may be shaken, the body may be twisted, or the head may be shaken.

  The shape of the icons of the phone book menu items 712a to 712c is not limited to the basket, and may be the icon shape shown in the first embodiment of the present invention. Further, after the menu item is selected, the pointer 711 may not be displayed at a predetermined position but may be displayed at a position when the menu item is selected.

  The pointer 711 is displayed at the position of the hand as in the first embodiment of the present invention, and has a second area corresponding to the position indicating the selection operation when the hand has a shape indicating the selection operation. A menu item may be executed.

  The user interface unit according to Embodiments 1 and 2 of the present invention can be applied not only to a videophone terminal but also to a desktop personal computer, a notebook personal computer, and a mobile phone. Furthermore, the user interface unit according to Embodiments 1 and 2 of the present invention can also be applied to a device that operates by receiving an operation by an operator.

(Embodiment 3 of the present invention)
Next, a third embodiment of the present invention will be described.

  The videophone terminal according to Embodiment 3 of the present invention ends the videophone communication when a predetermined end operation is performed during the videophone communication.

  The following description focuses on the differences from the videophone terminal 110 according to Embodiment 1 of the present invention.

  FIG. 15 is a block diagram showing a configuration of STB 811 provided in videophone terminal 810 according to Embodiment 3 of the present invention. Here, the same blocks as those in FIG.

  The STB 811 newly includes an end determination unit 812, an end execution unit 813, and a selection prohibition unit 815, as compared with the STB 111 of FIG.

  The end determination unit 812 determines whether to end the videophone communication based on the image acquired by the image acquisition unit 211 or the audio acquired by the audio acquisition unit 221 during the videophone communication. The detailed configuration of the end determination unit 812 will be described later.

  The end execution unit 813 requests the communication control unit 225 to end the videophone communication based on the control signal indicating whether to end the videophone communication output from the end determination unit 812.

  The selection prohibition unit 815 permits or prohibits the menu item selection in the menu processing unit 217 in accordance with the signal output from the communication control unit 225. Specifically, selection of menu items by the menu processing unit 217 during videophone communication is prohibited, and selection of menu items by the menu processing unit 217 is permitted during non-communication.

  The selection prohibition unit 815 may prohibit at least one of the movement of the pointer of the pointer moving unit 215 and the display of the pointer of the display sound output unit 219 in addition to the prohibition of selection of the menu item during videophone communication. Good.

  FIG. 16 is a block diagram illustrating a configuration of the end determination unit 812.

  The end determination unit 812 determines the end of the videophone communication by a predetermined call end operation during the videophone communication. The end determination unit 812 includes a past image storage unit 401, an image change determination unit 403, an end image storage unit 405, an end image determination unit 407, a past sound storage unit 409, and a sound determination unit 411.

  The past image storage unit 401 stores images output by the image change determination unit 403 at regular time intervals. Note that one or more images may be stored in the past image storage unit 401.

  The image change determination unit 403 includes a timer, compares the image data acquired by the image acquisition unit 211 and the image stored in the past image storage unit 401 at regular intervals, and determines whether there is a difference between the images. Determine. Specifically, the determination of whether or not there is a difference between images is realized by providing a threshold value. If the image change determination unit 403 determines that there is a difference, the image change determination unit 403 stores the image data acquired from the image acquisition unit 211 in the past image storage unit 401. If the image change determination unit 403 determines that there is no difference, the image change determination unit 403 outputs an end determination to the end execution unit 813.

  Note that the image comparison determination unit 403 may compare the images acquired from the image acquisition unit 211 with a plurality of images stored in the past image storage unit 401.

  The end image storage unit 405 stores a predetermined end image. Note that there may be a plurality of end images.

  The end image determination unit 407 determines whether or not the image data acquired by the image acquisition unit 211 includes the end image stored in the end image storage unit 405. Outputs termination judgment.

  The past voice storage unit 409 stores voice data output by the voice acquisition unit 221 at regular intervals. Note that the number of audio data stored in the past audio storage unit 409 may be one or plural.

  The voice determination unit 411 includes a timer, and compares whether or not the voice data output from the voice acquisition unit 221 exceeds a predetermined volume at regular intervals. When the sound determination unit 411 determines that the sound data exceeds a predetermined volume, the sound determination unit 411 stores the sound data acquired from the sound acquisition unit 221 in the past sound storage unit 409. If the sound determination unit 411 determines that the sound data does not exceed a predetermined volume, the sound determination unit 411 outputs an end determination to the end execution unit 813.

  Note that the voice determination unit 411 determines whether there is a difference between the voice data, whether the voice data stored in the past voice storage unit 409 and the voice data output from the voice acquisition unit 221 are different. You may judge by. At this time, the audio determination unit 411 may compare the audio data acquired from the audio acquisition unit 221 with a plurality of audio data stored in the past audio storage unit 409.

  Also, the end determination unit 812 is output when the end determination is output by both the image change determination unit 403 and the sound determination unit 411 or when the end determination is output by both the end image determination unit 407 and the sound determination unit 411. The end determination may be output to the end execution unit 813.

  FIG. 17 is a diagram showing a self-portrait 501 when the operator uses the end image 801 to end the videophone communication during the videophone communication. The operator ends the videophone communication by showing a predetermined end image 801 on the camera 113. Note that the end image 801 may be any image. Further, similarly to the first embodiment of the present invention, end confirmation menu items 507a to 507b may be displayed thereafter.

  FIG. 18 is a flowchart showing the flow of the menu item selection prohibiting operation of the videophone terminal 810.

  The communication control unit 225 of the videophone terminal 810 outputs to the selection prohibition unit 815 whether communication is in progress (S201). If the communication control unit 225 determines that videophone communication is in progress (Yes in S201), the selection prohibition unit 815 prohibits selection of menu items in the menu processing unit 217 (S202). When the communication control unit 225 determines that the videophone communication is not being performed (No in S201), the selection prohibition unit 815 permits the menu item selection in the menu processing unit 217 (S203).

  As described above, in Embodiment 3 of the present invention, videophone terminal 810 ends the videophone communication when a predetermined end operation is performed during the videophone communication.

  Videophone terminal 810 has a first mode for executing selection prohibition unit 815 and a second mode for prohibition, and the operator may arbitrarily select any mode.

  The user interface device, the videophone terminal, the user interface control method, and the program according to the present invention have been described based on the embodiments. However, the present invention is not limited to these embodiments. Unless it deviates from the meaning of this invention, the form which made the various deformation | transformation which those skilled in the art think about in each embodiment, and was constructed | assembled combining the component in a different embodiment is also contained in the form of this invention.

  For example, in the first embodiment, the position detection unit 213 extracts a moving part in the image data by taking a difference at regular intervals of the image data output from the image acquisition unit 211. Next, the position detection unit 213 may use the uppermost point of the moving part as the pointer coordinates.

  FIG. 19 is a diagram illustrating an example of a self-portrait in which a pointer 503 is displayed at the highest point of the movement of the subject. Here, since the operator fixes a part other than the hand and shakes only the hand, the pointer 503 is displayed at the uppermost point of the hand. In FIG. 19, in order to make it easy to see the positional relationship between the pointer 503 and the operator, the display of each menu item is omitted.

  The present invention can be applied to a user interface device, and in particular, can be applied to a user interface device that is easy to use for elderly people who are not good at operating devices.

It is a figure which shows the structure of the video telephone system which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of STB with which the videophone terminal which concerns on Embodiment 1 of this invention is provided. It is a block diagram which shows the structure of the menu process part of the video telephone terminal which concerns on Embodiment 1 of this invention. It is a figure which shows the example of a screen displayed at the time of the non-communication in the videophone terminal which concerns on Embodiment 1 of this invention. FIG. 5 is a diagram showing a display example of a self-portrait when confirming the start of videophone communication in the videophone terminal according to Embodiment 1 of the present invention. It is a figure which shows the example of a screen displayed at the time of videophone communication in the videophone terminal which concerns on Embodiment 1 of this invention. FIG. 6 is a diagram showing a display example of a self-portrait when confirming the end of videophone communication in the videophone terminal according to Embodiment 1 of the present invention. FIG. 7 is a diagram showing a display example of a self-portrait when a call is received from another terminal during non-communication in the videophone terminal according to Embodiment 1 of the present invention. It is a figure which shows typically the display which shows the menu item of the video telephone terminal which concerns on Embodiment 1 of this invention. In the videophone terminal concerning Embodiment 1 of this invention, it is a figure which shows the 1st area | region and 2nd area | region of the menu item stored in the menu memory | storage part. 5 is a flowchart showing a flow of operation during non-communication in the videophone terminal according to Embodiment 1 of the present invention. It is a block diagram which shows the structure of the user interface part of the video telephone terminal which concerns on Embodiment 2 of this invention. In the videophone terminal concerning Embodiment 2 of this invention, it is a figure which shows the shape of the hand showing the selection operation | movement of the operator stored in the shape memory | storage part. In a videophone terminal concerning Embodiment 2 of the present invention, it is a figure showing signs that a pointer moves when an operator holds a hand on the pointer. It is a block diagram which shows the structure of STB with which the videophone terminal which concerns on Embodiment 3 of this invention is provided. It is a block diagram which shows the structure of the completion | finish determination part of the video telephone terminal which concerns on Embodiment 3 of this invention. In the videophone terminal according to Embodiment 3 of the present invention, it is a diagram showing a self-portrait when an operator uses the end image to end the videophone communication during videophone communication. 10 is a flowchart showing a flow of menu item selection prohibition operation in the videophone terminal according to Embodiment 3 of the present invention. In the videophone terminal concerning Embodiment 1 of this invention, it is a figure which shows the example of a display of a self-portrait in which a pointer is displayed on the highest point of a subject's movement.

Explanation of symbols

100 Videophone system 110, 810 Videophone terminal 111, 811 Set-top box (STB)
DESCRIPTION OF SYMBOLS 112 Monitor 113 Camera 114 Speaker 115 Microphone 120 Network 130 Management server 140 General terminal 210,610 User interface part 211 Image acquisition part 213 Position detection part 215 Pointer movement part 217 Menu processing part 219 Display sound output part 221 Audio | voice acquisition part 223 Transmission / reception part 225 Communication control unit 301 Menu storage unit 303 Mutual coordinate determination unit 305 Selection determination unit 307 Menu execution unit 309 Menu display unit 401 Past image storage unit 403 Image change determination unit 405 End image storage unit 407 End image determination unit 409 Past voice storage unit 411 Voice determination unit 501 Self-portrait 503, 711 Pointer 504a to 504c, 712a to 712c Phonebook menu item 505a to 505b Communication start confirmation menu item 506 End New items 507a~507b end confirmation menu item 508 called menu item 511 partner image 521 first region 522 second region 611 shape storage unit 613 shape determination unit 801 ends the image 812 end determining unit 813 ends execution unit 815 selects prohibiting unit

Claims (13)

A video phone terminal,
A camera to shoot the subject,
A display unit that displays the subject photographed by the camera and displays at least one menu item related to the operation of the videophone terminal on the subject;
A position detection unit that detects a position of an image representing an object that is part of the subject included in the subject photographed by the camera;
A menu processing unit that selects the menu item according to an overlap between an image representing the object and an area corresponding to the menu;
A selection prohibiting unit that prohibits selection of the menu item by the menu processing unit during a call with another videophone terminal;
The display unit further displays a pointer at a position of an image representing the object. The videophone terminal according to claim 1, wherein the selection prohibition unit further prohibits display of the pointer by the display unit during the call. The videophone terminal further includes:
An image change determination unit that determines whether or not the subject photographed by the camera during the call has a change in a predetermined time;
The videophone terminal according to claim 1, further comprising: an end executing unit that ends the call when the image change determining unit determines that there is no change. The videophone terminal further includes:
An audio acquisition unit for acquiring audio;
A voice determination unit that determines whether or not the voice acquired by the voice acquisition unit during the call has a predetermined volume or more at a predetermined time;
The videophone terminal according to claim 1, further comprising: an end executing unit that terminates a call when the sound determination unit determines that the volume is not higher than a predetermined volume. The videophone terminal further includes:
An end image determination unit that determines whether or not a predetermined end image is included in the subject captured by the camera during the call;
The videophone terminal according to claim 1, further comprising: an end executing unit that ends the call when it is determined by the end image determining unit. The videophone terminal according to claim 1, wherein the display unit transmits the subject that overlaps the menu item. The videophone terminal according to claim 1, wherein the menu processing unit selects the menu item when an image representing the object and an area corresponding to the menu item overlap each other for a predetermined time. The position detection unit identifies the shape of the object,
The videophone terminal further includes:
A shape determination unit for determining whether or not the shape of the object is a predetermined shape;
The videophone terminal according to claim 1, wherein the menu processing unit selects the menu item when it is determined by the shape determination unit that the predetermined shape is predetermined. The videophone terminal further includes:
The videophone terminal according to claim 1, further comprising: a pointer moving unit that controls movement of the pointer in response to movement or shape of an image representing the object. The pointer moving unit controls to display the pointer at a predetermined position, and after the position of the image representing the object overlaps the position of the pointer, the pointer follows the position of the image representing the object. The videophone terminal according to claim 9, wherein control is performed so that the pointer is displayed at a predetermined position after the menu item is selected. The videophone terminal according to claim 10, wherein the menu processing unit selects the menu item according to a time when the pointer overlaps an area corresponding to the menu item or a behavior of the object when the pointer overlaps. A videophone terminal control method for controlling a videophone terminal,
A shooting step of shooting the subject with the camera;
A display step of displaying the subject photographed by the camera and displaying at least one menu item relating to the operation of the videophone terminal on the subject;
A position detecting step for detecting a position of an image representing an object that is a part of the subject included in the subject photographed by the camera;
A menu processing step of selecting the menu item in accordance with an overlap between an image representing the object and an area corresponding to the menu;
A selection prohibiting step for prohibiting selection of the menu item by the menu processing unit during a call with another videophone terminal,
In the display step, a pointer is further displayed at a position of an image representing the object. A program for controlling a videophone terminal,
A shooting step of shooting the subject with the camera;
A display step of displaying the subject photographed by the camera and displaying at least one menu item relating to the operation of the videophone terminal on the subject;
A position detecting step for detecting a position of an image representing an object that is a part of the subject included in the subject photographed by the camera;
A menu processing step of selecting the menu item in accordance with an overlap between an image representing the object and an area corresponding to the menu;
During a call with another videophone terminal, causing the computer to execute a selection prohibiting step for prohibiting selection of the menu item by the menu processing unit,
In the display step, a pointer is further displayed at a position of an image representing the object.

Images