WO2007116662A1

WO2007116662A1 - Electronic device and method for operating same

Info

Publication number: WO2007116662A1
Application number: PCT/JP2007/056081
Authority: WO
Inventors: Hiroyuki Ishihara; Hideaki Watanabe
Original assignee: Pioneer Corporation
Priority date: 2006-03-27
Filing date: 2007-03-23
Publication date: 2007-10-18

Abstract

Provided is an electronic device which can suitably detect existence and movement of a viewer by using imaging results obtained by a camera. An electronic device (10) is provided with a plurality of imaging sections (1, 2) for converting the image of an object (20) in front of the display screen of a display section (5) into an image signal; an image processing section (6) for signal-processing the image signal; and a display control section (7) for controlling an image to be displayed on the display section (5) based on the signal processing results of the image processing section (6). The image processing section (6) specifies the position of the object (20), based on, for instance, a difference between the image signals outputted from the imaging sections (1, 2).

Description

Specification

Electronic device and operation method thereof

Technical field

The present invention relates to an electronic device that controls the operation of a display device that includes a display unit having a display screen, and an operation method thereof.

Background art

A human interface system that detects and recognizes hand gestures and body actions using an image sensor has been proposed (for example, Patent Document 1).

[0003] In addition, electronic devices have been proposed in which the gesture, hand gesture, facial expression, and line of sight of a viewer in front of the display device are read by a camera and the display device is controlled based on the camera (for example, Patent Documents 2 and 3).

[0004] Further, a display device that detects the position of the viewer in front of the display device using a plurality of sensors and controls the image and volume according to the detected position of the viewer has also been proposed (for example, Patent Document 4).

Patent Document 1: JP-A-10-31551

Patent Document 2: Japanese Translation of Special Publication 2004—527954

Patent Document 3: Japanese Patent Application Laid-Open No. 2004-213486

Patent Document 4: Japanese Patent Laid-Open No. 2005-49656

Disclosure of the invention

Problems to be solved by the invention

[0005] However, in each of the above patent documents, it is not disclosed how to read the viewer's gesture, hand gesture, facial expression, and line of sight with a camera. You can see that it is not easy to read!

[0006] For example, it is difficult to read accurately only by switching the TV channel according to the gesture of the viewer unless the gesture is large. When you get close to the camera, you don't need to make a lot of gestures, but on the other hand, viewers have difficulty operating the TV.

[0007] For this reason, the same control as a remote controller currently used in a television or the like is possible. And is easy to operate like a remote controller.

[0008] In addition, a system for recognizing a viewer using a sensor can perform only relatively basic control such as turning on and off the display device and adjusting the volume according to the distance from the viewer. .

[0009] The above-described problem is given as an example of the problem of the present invention.

Means for solving the problem

[0010] In order to solve the above-described problem, the present invention provides, as a first aspect, an electronic device that controls the operation of a display device including a display unit having a display screen, the object being in front of the display screen An image processing unit that performs signal processing on the image signal, and controls the operation of the display unit based on a signal processing result of the image processing unit. An electronic device is provided.

[0011] Furthermore, the present invention provides, as a second aspect, an electronic device operation method for controlling an operation of a display device including a display unit having a display screen, at a predetermined position in front of the display screen. A first step of identifying an object by detecting a predetermined shape, a predetermined movement, or a predetermined movement of the predetermined shape; and a plurality of menus in response to the first step; A second step of displaying on the display screen a cursor at a position corresponding to one of the plurality of menus, a third step of moving the cursor according to the movement of the object, And a fourth step of executing a menu corresponding to the position of the cursor according to the movement of the object.

[0012] Further, according to a third aspect of the present invention, there is provided a program for causing a computer to execute an operation method of an electronic device that controls an operation of a display device including a display unit having a display screen. Is a first process for identifying an object by detecting a predetermined shape, a predetermined movement, or a predetermined movement of the predetermined shape at a predetermined position in front of the display screen. A second process of displaying a plurality of menus and a cursor at a position corresponding to any one of the plurality of menus on the display screen in response to the first step; A third process for moving the cursor according to the movement of the object, and a menu corresponding to the position of the cursor according to the movement of the object. And a fourth process for executing a program.

Furthermore, the present invention provides, as a fourth aspect, a display device including a display unit having a display screen and the electronic device described above.

Brief Description of Drawings

FIG. 1 is a block diagram showing a configuration of an electronic device according to Embodiment 1 of the present invention.

FIG. 2 is a block diagram of a display control unit that is one component of the electronic device according to the first embodiment of the invention.

FIG. 3 is a flowchart showing a flow of an example of an operation for determining whether or not an object to be imaged by a camera is a viewer.

FIG. 4 is a flowchart showing an example of an operation for determining whether or not an object to be imaged by a camera is a viewer.

FIG. 5 is a flowchart showing a flow of another example of an operation for determining whether or not an object to be imaged by a camera is a viewer.

FIG. 6 is a schematic front view showing a positional relationship between a camera and a display device including a display unit having a display screen.

FIG. 7 is a diagram showing the XZ plane of FIG.

FIG. 8 is a block diagram showing a configuration of an electronic apparatus according to Embodiment 2 of the present invention.

FIG. 9 is a block diagram showing a configuration of an electronic apparatus according to Embodiment 3 of the present invention.

FIG. 10 is a flowchart of an example of a method for discriminating viewer gestures, hand gestures, and other gesture commands by the image processing unit.

FIG. 11 is a block diagram showing a configuration of an electronic device according to Embodiment 4 of the present invention.

FIG. 12 is a block diagram illustrating a structure of a display device including the electronic device according to the first embodiment of the invention.

FIG. 13 is a block diagram showing a structure of a display device including an electronic device according to Embodiment 2 of the present invention.

FIG. 14 is a block diagram showing a structure of a display device including an electronic device according to Embodiment 3 of the present invention.

FIG. 15 is a block diagram showing the structure of a display device including an electronic device according to Example 4 of the invention. It is.

Explanation of symbols

10 Electronic device according to Embodiment 1 of the present invention

1 Camera (imaging part)

2 Camera (imaging part)

3 Camera (imaging part)

4 Display screen

5 Display section

6 Image processing section

7 Display controller

71 Central processing unit

72 First memory

73 Second memory

74 Human interface

75 Output interface

8 Viewing history generator

9 Viewing history storage

21 Display device

11 Notification light emitting part

12 Light emission controller

13 Audio generator

14 Voice control unit

15 Main power input section

20 Object to be imaged

30 Electronic device according to Embodiment 2 of the present invention

40 Electronic device according to Embodiment 3 of the present invention

50 Electronic device according to Embodiment 4 of the present invention

100 display devices 200 display devices

300 display

400 display device

BEST MODE FOR CARRYING OUT THE INVENTION

In one embodiment of the electronic device according to the present invention, it is preferable to control the operation of the display unit according to the position or shape of the object specified by the image processing unit.

In one embodiment of the electronic device according to the present invention, the image processing unit may specify a position of the object based on a difference between the image signals output from the imaging unit. preferable.

[0018] One embodiment of the electronic device according to the present invention may further include a display control unit that controls an image to be displayed on the display unit based on a signal processing result of the image processing unit. preferable.

In one embodiment of the electronic device according to the present invention, the display control unit displays on the display unit according to the position or shape of the object specified by the image processing unit. Preferred to control the image.

[0020] In one embodiment of the electronic device according to the present invention, the display control unit may be configured such that the object is positioned at a predetermined position or the object has a predetermined shape. The image displayed on the display unit is preferably controlled according to the movement of the object.

In one embodiment of the electronic device according to the present invention, the display control unit displays the above-mentioned display when the object is in a predetermined position or when the object has a predetermined shape. Preferably, a plurality of menus and a cursor at a position corresponding to any one of the plurality of menus are displayed on the screen, and the cursor is moved according to the movement of the object.

In one embodiment of the electronic device according to the present invention, the display control unit selects a menu corresponding to the position of the cursor when the movement of the object is a predetermined movement. It is preferable to display the image corresponding to the menu on the display unit. In one embodiment of the electronic device according to the present invention, when the display control unit loses sight of the object, information indicating that the object has been lost is displayed on the display screen. It is preferable to display.

In one embodiment of the electronic device according to the present invention, a viewing history generation unit that generates a past viewing history in the display device, and a viewing history that stores the viewing history generated by the viewing history generation unit. A storage unit, and the display control unit is stored in the viewing history storage unit according to the position or shape of the object specified by the image processing unit and the date and time. It is preferable that the viewing history is searched and a program or a list thereof is displayed on the display unit with a high possibility of being viewed.

In an embodiment of the electronic device according to the present invention, the notification operation of the notification operation unit is controlled under the control of a notification operation unit that performs a notification operation for a viewer and the image processing unit. A notification operation control unit for controlling the notification operation control unit when the display control unit detects a specific movement of the object specified by the image processing unit. It is preferable to output the notification operation control unit so as to operate the unit.

In one embodiment of the operation method of the electronic device according to the present invention, when the position of the object is lost, a fifth notification is given to the operator that the first step is to be executed again. It is preferable to further include the process.

In one embodiment of the operation method of the electronic device according to the present invention, the position of the object is determined based on a difference between image signals output by the at least two imaging units imaging the object. It is preferable to further include a sixth step of determining whether or not the vehicle is in a predetermined position.

[0028] In one embodiment of the method for operating an electronic device according to the present invention, it is preferable to turn on the main power of the display device in the second step.

[0029] In one embodiment of the method for operating an electronic device according to the present invention, a seventh step of generating and storing a past viewing history in the display device, and a position or shape of the object and Eighth step of searching the viewing history according to the date and time, and a program or a list thereof is displayed on the display unit with a high possibility of being viewed according to a search result of the viewing history. It is preferable to further include a ninth step.

Example 1

FIG. 1 is a block diagram showing a configuration of an electronic device 10 according to Embodiment 1 of the present invention.

Hereinafter, Example 1 will be described with reference to FIG.

As shown in FIG. 1, the electronic device 10 according to the first embodiment of the present invention is one of the components constituting the display device (for example, the display device 21 shown in FIG. 6), and performs display operation. Generated by two cameras (imaging units) 1 and 2 and cameras 1 and 2 that capture an image of the imaging object 20 located in front of the display screen of the display unit 5 to perform and convert the captured image into an image signal An image processing unit 6 that performs signal processing on the image signal, and a display control unit 7 that controls an image to be displayed on the display unit 5 based on the signal processing result received from the image processing unit 6.

[0033] Examples of the display device having the display unit 5 include a general television device and a personal computer. The electronic device 10 according to the first embodiment of the present invention is used by being connected to the display unit 5 of such a display device.

The cameras 1 and 2 are, for example, CCD cameras.

[0035] The image processing unit 6 processes the image signals input from the cameras 1 and 2, so that the imaging object 20 positioned in front of the display unit 5 is viewed by the viewer (the image displayed on the display unit 5 and It is determined whether or not the user is willing to view audio output from a speaker (not shown) provided in the display device.

[0036] Further, the display control unit 7 displays the automatically selected program on the display unit 5 when the image processing unit 6 determines that there is a viewer in front of the display unit 5. Take control.

FIG. 2 is a block diagram of the display control unit 7.

[0038] The display control unit 7 includes a central processing unit (CPU) 71, a first memory 72, a second memory 73, and an input interface for inputting various commands and data to the central processing unit 71. 74, an output interface 75 for outputting the result of the processing executed by the central processing unit 71, and a bus 76 for connecting the CPU 71 and other components.

[0039] Each of the first and second memories 72 and 73 is a read-only memory (ROM), a random access memory (RAM), a semiconductor memory device such as an IC memory card, a flexible disk, etc. Storage medium, hard disk, or optical magnetic disk, etc. There will be power. In the present embodiment, the first memory 72 also has ROM power, and the second memory 73 consists of RAM.

The first memory 72 stores various control programs and other fixed data to be executed by the central processing unit 71. The second memory 73 stores various data and parameters and provides an operating area for the central processing unit 71, that is, temporarily necessary for the central processing unit 71 to execute the program. The data that is supposed to be stored. The central processing unit 71 reads the program from the first memory 72 and executes the program. That is, the central processing unit 71 operates according to a program stored in the first memory 72.

Here, as a method for specifying that the viewer is present in front of the display screen included in the display unit 5 of the display device and presenting the program selection to the viewer, the following method can be considered.

(1) Distinguishing between people and non-people, whether they are moving

As a method for discriminating the viewer, the following discrimination is made based on the premise that (A) there is a slight movement for a person and (B) the viewer does not move forward force on the display screen.

[0042] FIG. 3 shows an operation of determining whether or not the object 20 to be imaged by the camera 1 (or the camera 2 or both of the cameras 1 and 2; the same applies hereinafter) is a viewer by the image processing unit 6. It is a flowchart which shows the flow of.

[0043] Hereinafter, an operation of the image processing unit 6 determining whether or not the imaging target 20 of the camera 1 is a viewer will be described with reference to FIG.

[0044] First, the image processing unit 6 detects the movement of the imaging target 20, and distinguishes between the moving object and the non-moving object. That is, the image processing unit 6 determines whether or not the imaging target 20 has a force (Step Sl).

[0045] When there is no movement of the imaging target 20, the image processing unit 6 repeatedly determines whether or not the imaging target 20 has a movement (NO in step S1).

[0046] When the imaging target 20 has a motion (YES in step S1), the image processing unit 6 determines whether or not the imaging target 20 has moved (step S2).

[0047] When the imaging target 20 is moved (YES in step S2), the image processing unit 6 determines again whether or not the imaging target 20 has a motion (step Sl). [0048] If the imaging object 20 has moved (NO in step S2), the image processing unit 6 determines that the imaging object 20 is a viewer (step S4).

That is, the image processing unit 6 determines that the imaging target 20 is a person if the imaging target 20 has a motion (YES in step S1). Further, if the movement is smaller than the predetermined size, it is determined that the person who is the imaging target 20 has not moved (NO in step S2), that is, the imaging target 20 is the viewer ( Step S4).

[0050] Here, in the determination of whether or not the imaging target 20 has a force (step S1), specifically, for example, the image processing unit 6 performs predetermined processing at a predetermined speed within a predetermined time. Judgment is based on whether there is movement over the length. If there is a movement of a predetermined length or more at a predetermined speed within a predetermined time, the image processing unit 6 determines that there is a movement of the imaging target 20 (YES in step S1).

[0051] Further, in determining whether or not the imaging target 20 has a force (step S1), the image processing unit 6 detects, for example, the movement of the imaging target 20 a predetermined number of times or more within a predetermined time. Judge whether it is power or not. When the movement of the imaging object 20 is detected a predetermined number of times or more within a predetermined time, the image processing unit 6 determines that the imaging object 20 has a movement (YES in step S1).

[0052] Further, in determining whether or not the imaging object 20 is moving (step S2), the image processing unit 6 determines whether or not the movement is less than a predetermined magnitude. When the movement of the imaging target 20 is less than the predetermined magnitude (NO in step S2), the image processing unit 6 determines that the imaging target 20 is a viewer (step S4).

FIG. 4 is a flowchart showing a specific example in which the image processing unit 6 determines whether or not the imaging target 20 by the camera 1 is a viewer.

[0054] The camera 1 is assumed to be a CCD camera.

[0055] First, the camera 1 determines whether or not the imaging object 20 is capable of detecting a motion of 5 dots or more and less than 100 dots in 5 seconds (step S1-1).

[0056] If movement of less than 5 dots or more than 100 dots is detected in 5 seconds (step S1

-NO of 1), the image processing unit 6 repeatedly determines whether or not the imaging object 20 is capable of detecting a motion of 5 dots or more and less than 100 dots in 5 seconds (step S1-1). [0057] If a motion of 5 dots or more and less than 100 dots is detected in 5 seconds (in step S 1-1)

YES), the image processing unit 6 determines whether or not the movement is detected more than 6 times per minute (step S1-2).

[0058] When the motion is less than 5 times (NO in step S1-2), the image processing unit 6 determines whether or not a motion of 5 dots or more and less than 100 dots is detected in the imaging target 20 for 5 seconds. This determination is repeated (step S1-1).

[0059] If detected six times or more (YES in step S1-2), there is movement, and the image processing unit 6 determines that the imaging target 20 is a person (YES in step S1 in FIG. 3).

[0060] If the force also moves less than 100 dots in 5 seconds, the image processing unit 6 has not moved the imaging object 20 (NO in step S2 in FIG. 3), that is, the imaging target. The object 20 is determined to be a viewer of the display device (step S4).

[0061] In the flowchart shown in FIG. 3, the check of "force with movement" (step S1) and "whether it is moving" (step S2) is executed only once by one camera 1. Using at least two cameras 1 and 2, based on the image signals generated by the cameras 1 and 2, whether or not the imaging target 20 is moving is determined (step S1), and By determining whether or not the imaging object 20 is moving (step S2), it is possible to improve the accuracy of determining whether or not the imaging object 20 is a viewer (step S4). it can.

(2) Identification of viewer position

FIG. 5 is a flowchart showing an operation for improving detection accuracy further than the operation shown in FIG.

In the example shown in FIG. 5, based on the difference between the image signals generated by the two cameras 1 and 2, the operation of (1) above, that is, the process shown in FIG. The position of the imaging object 20 determined to be a person is specified, and it is determined whether or not the specified position is within a preset viewing range (step S3).

[0063] In the case where the position of the imaging target 20 is not within the preset viewing range (NO in step S3), the image processing unit 6 determines whether the imaging target 20 has a motion) Perform again (step Sl).

[0064] If the position of the imaging object 20 is within the preset viewing range (Y in step S3) ES), the image processing unit 6 determines that the imaging target 20 is a viewer who intends to view the display device (step S4).

FIG. 6 is a schematic front view showing a positional relationship between the display device 21 including the display unit 5 having the display screen 4 and the cameras 1 and 2.

As shown in FIG. 6, it is assumed that cameras 1 and 2 are provided on the left and right of the upper end of the display device 21, respectively.

[0067] The standard viewing range is, for example, a distance from the display screen 4 included in the display unit 5 of the display device 21 (a length in the z-axis direction from the display screen 4), orthogonal to the z-axis, and Assuming that it is defined by the horizontal (X-axis) position and vertical (y-axis) position on the plane parallel to display screen 4 (xy plane), the distance from display screen 4 is displayed. The range is 2 to 3 times the diagonal length of screen 4, and the position in the plane perpendicular to z-axis and parallel to display screen 4 is generated by translating display screen 4 in the z-axis direction. It shall be within the space.

[0068] The two cameras 1 and 2 are adjusted in advance so that the viewing range can be imaged, and the positions of the two cameras 1 and 2 are imaged by the two cameras 1 and 2. It is set so that there is a difference between images.

Subsequently, referring to FIG. 7, a method for specifying the position of the viewer by the image processing unit 6 will be described.

FIG. 7 shows the XZ plane of FIG. 6, and cameras 1 and 2 are installed on the X axis of FIG. Camera 1,

Specifically, 2 is a force that assumes a CCD two-dimensional camera, for example.

[0071] As shown in FIG. 7, the camera 1 includes (n + 1) sensor elements AO, Al,..., Ap on the X axis.

, ... An is arranged. Similarly, camera 2 has (n + 1) sensor elements BO, Bl,

······················ Bn

An object on the straight line a 0 is imaged on the sensor element AO. Similarly, the sensor elements Ap and An are imaged on the object α on the straight line αρ, 0; 11, and the sensor elements 80, 8 ₈ , 811 have the straight line | 8 0, β §

The object on j8 η is imaged.

[0073] That is, the camera 1 captures an object within a range between the straight line α θ and the straight line an, Camera 2 captures an object in the range between straight lines | 80 and | 8n.

[0074] At this time, the position of the object (considered as a point here) imaged by the sensor element Ap of the camera 1 and imaged by the sensor element Bg of the camera 2 is the position where the straight line ap and | 8 g in FIG. You can see that.

As described above, the image processing unit 6 can specify the position of the viewer on the XZ plane.

[0076] In order to specify the position in the height direction of the viewer, it is necessary to specify the position of the viewer on the YZ plane.

In this case, as shown in FIG. 6, the cameras 1 and 2 are on the left and right of the upper end of the display device 21, and the camera 3 different from the cameras 1 and 2 is on the lower portion of the display device 21 (below the camera 1). Deploy.

[0078] Based on the image signals generated by the cameras 1 and 3 in FIG. 7, the image processing unit 6 determines the position of the viewer in the YZ plane in the same manner as specifying the position in the XZ plane. Can be identified.

[0079] Note that the three-dimensional position of the object to be imaged 20, that is, the viewer, can be specified even when three cameras are not used or two cameras are used. In this case, for example, like the cameras 2 and 3 shown in FIG. 6, two cameras located on the diagonal line of the display screen 4, that is, two cameras that are separated from each other in the X direction and the Y direction, respectively. Use the camera.

According to the first embodiment as described above, the operation of the display device 21 including the display unit 5 having the display screen 4 can be controlled based on the signal processing result of the image processing unit 6.

Example 2

FIG. 8 is a block diagram showing a configuration of the electronic device 30 according to the second embodiment of the present invention.

As described below, in the electronic device 30 according to the second embodiment of the present invention, the viewer's viewing program is predicted based on the viewing history of the past display device 21.

As shown in FIG. 8, the electronic device 30 according to the second embodiment of the present invention includes a past viewing history on the display device 21 in addition to the configuration of the electronic device 10 according to the first embodiment shown in FIG. And a viewing history storage unit 9 for storing and holding the viewing history generated by the viewing history generation unit 8. Note that the central processing unit 71 and the second memory 73 shown in FIG. 2 functionally configure the viewing history generation unit 8 and the viewing history storage unit 9 in this embodiment.

[0085] Every time a program is viewed on the display device 21, the viewing history generation unit 8 counts the number of viewers, the position of each viewer, the day of the week, the time zone, and other times. Viewing history data in which the parameters related to the program viewed are associated with each other. The generated viewing history data is stored in the viewing history storage unit 9.

[0086] Thereby, the viewing history storage unit 9 constructs a database that accumulates viewing history data as the viewing history data is stored.

[0087] When the image processing unit 6 determines that the viewer is willing to view, the display control unit 7 determines the number of viewers who are willing to view at that time, the position of each viewer, Based on the conditions such as the day of the week and the time zone, the past viewing history on the display device 21 (the program that matches the conditions) is searched from the viewing history storage unit 9, and is also currently being broadcast or started within a predetermined time EPG (Electronic Program Guide) is searched for the program to be viewed, and the program that is most likely to be viewed by the viewer is displayed on the display unit 5.

[0088] Alternatively, the display control unit 7 displays on the display unit 5 a program list that is most likely to be watched and is currently being broadcast or started within a predetermined time in the order of programs. It is also possible to allow the viewer to select a program to watch by himself / herself. The selection of the program is performed by using, for example, an operation unit (not shown) of the display device 21 or a remote controller.

[0089] When viewing a program displayed on the display device 21 at home, the position for viewing by each member of the family may be determined.

Therefore, the image processing unit 6 can estimate which viewer is based on the position of the viewer and output the estimation result to the display control unit 7.

[0091] The display control unit 7 that has received the estimation result from the image processing unit 6 displays the program normally viewed by the viewer who views at a specific position, that is, from the current day of the week and time zone according to the position. It is possible to search the past viewing history in the device 21 in the viewing history storage unit 9.

[0092] If the position of the viewer is specified as in the first embodiment, the cameras 1 and 2 (or the camera The direction and focus of 1, 2, 3) are adjusted to each viewer's face, each viewer's face is imaged, the captured image is signal processed, and compared with the pre-stored viewer's face image. Thus, the viewer can be specified.

According to the second embodiment as described above, the viewer's viewing program based on the past viewing history of the display device 21 can be predicted based on the signal processing result of the image processing unit 6.

Example 3

FIG. 9 is a block diagram showing a configuration of the electronic device 40 according to the third embodiment of the present invention.

As shown in FIG. 9, in addition to the configuration of the electronic device 10 according to the first embodiment shown in FIG. 1, the electronic device 40 according to the third embodiment of the present invention provides, for example, a predetermined notification to the viewer. The notification light emitting unit 11 for performing the operation by light emission, the light emission control unit 12 for controlling the light emission operation of the notification light emitting unit 11 under the control of the image processing unit 6, and the predetermined notification operation for the viewer by voice And a sound control unit 14 for controlling the sound generation operation of the sound generation unit 13 under the control of the image processing unit 6.

[0096] The notification light emitting unit 11 includes, for example, a lamp, an LED, and the like, and is disposed around the display screen 4 of the display unit 5, for example.

[0097] The sound generator 13 is composed of a speaker. The speaker that constitutes the sound generation unit 13 is provided as one component of the display device 21, for example, and may be shared by a speaker that emits sound during viewing or may be provided separately from the speaker. good.

[0098] When the image processing unit 6 determines the gesture command, gesture, or other gesture command of the viewer, the image processing unit 6 transmits a control signal to one or both of the light emission control unit 12 and the voice control unit 14, and the voice generation unit 13 Alternatively, the notification light emitting unit 11 is activated.

FIG. 10 is a flowchart of an example of a method for discriminating viewer gestures, hand gestures, and other gesture commands by the image processing unit 6. Hereinafter, a method for discriminating the viewer's gesture command by the image processing unit 6 will be described with reference to FIG.

(1) Preparation

As preparation, a range of a three-dimensional space in which a gesture command is provided is set in advance, and the image processing unit 6 holds information indicating the range.

[0100] For example, the distance from the display screen 4 included in the display unit 5 of the display device 21 (from the display screen 4 The length in the z-axis direction) is in the range of 2 to 3 times the diagonal length of the display screen 4, and the horizontal direction on the plane (xy plane) that is perpendicular to the z-axis and parallel to the display screen 4 The position in the X axis and the position in the vertical direction (in the y axis) are within the space created by translating the display screen 4 in the z axis direction.

[0101] The range of the three-dimensional space can be adjusted in advance by the viewer through operation. This adjustment corresponds to, for example, the focus adjustment of the CCD camera.

In the flowchart shown in FIG. 10, the imaging object 20 is determined as a viewer in the same manner as Steps S1 to S4 shown in FIG.

(2) Object identification command and command menu

The image processing unit 6 needs to specify the object 20 that provides the gesture command. The power at which the approximate position of the viewer is known by the processing in steps S1 to S4. Whether there is the target object 20 in the viewer, and if there are multiple viewers, which viewer is the target object 20. It is necessary to specify whether it is.

That is, it is necessary to provide the electronic device 40 with an object identification command (gesture command) indicating that the viewer is the object 20.

[0104] The object specifying command needs to be an operation that does not correspond to the normal operation of the viewer. This is because if a normal operation is defined as an object specifying command, it cannot be distinguished whether the normal operation is performed as an object specifying command.

[0105] As the object specifying command, for example, a motion at a predetermined speed or higher or a repetitive motion such as a reciprocating motion or a circular motion can be considered. In other words, moving the hand at a speed faster than a predetermined speed, gestures and repeated movements of repeatedly shaking the hand left and right, up and down or diagonally, and circular movements such as drawing a circle by hand are conceivable. It can be set as a condition for the object specifying command that these repeated motions and circular motions are performed at a predetermined speed or more.

[0106] When the object 20 performs some operation, the image processing unit 6 determines whether the operation is a gesture one command (step S5).

[0107] When the image processing unit 6 determines that the motion of the object 20 is a gesture command (YES in step S5), the image processing unit 6 outputs a notification command signal to the light emission control unit 12 and the sound control unit 14. . When the notification command signal from the image processing unit 6 is received, the light emission control unit 12 causes the notification light emitting unit 11 to emit light, while the sound control unit 14 causes the sound generation unit 13 to generate a notification sound.

[0109] Power! Alternatively, the image processing unit 6 can output a command to the display control unit 7 and prompt the display unit 5 to display a notification message or the like.

Next, the image processing unit 6 outputs a command to the display control unit 7 and causes the display unit 5 to display a command menu (details will be described later).

[0111] Note that even when an image is displayed on the display unit 5, it overlaps the image displayed on the display unit 5 or a part of the image displayed on the display unit 5. Erase and display the command menu. This indicates to the viewer that the object 20 has been identified, and prompts the viewer for the next command.

(3) Cursor movement and command selection

The command menu displayed on the display unit 5 includes a channel selection menu.

[0112] Specifically, for example, a menu listing terrestrial television, BS television, CS television, and the like is displayed. The command selection is completed by moving the cursor to the desired menu position and selecting that menu. In other words, menu selection is done in the same way as a PC mouse.

[0113] The cursor is moved by repetitive movements of the hand or other part of the viewer's body. The direction is the direction of movement of the hand.

[0114] However, for example, when the hand is reciprocated left and right, whether to move the cursor to the right or left is indicated by the difference in speed when moving to the right or left. For example, the cursor can be moved in the direction of moving the hand slowly. Or conversely, the cursor can be moved in the direction of moving the hand quickly.

[0115] When the cursor is moved in the direction of moving the hand slowly, the image processing unit 6 moves the cursor in that direction in conjunction with the movement of the hand.

That is, the image processing unit 6 detects the type of hand movement and outputs the detection result to the display control unit 7. The display control unit 7 that has received the detection result from the image processing unit 6 moves the cursor according to the type of hand movement. When the cursor is over the target command, the viewer stops the movement of the hand, so that the image processing unit 6 moves the cursor. Stop at the point where it stopped.

[0117] Next, the viewer performs a selection command gesture.

[0118] The gesture of the selection command can also be the movement force of the hand or other part of the viewer's body, but in this case, the movement may be different from the movement for moving the cursor. is necessary.

[0119] When the movement of the cursor is determined by the slow movement of the reciprocating movement of the hand or other body part of the viewer, for example, the hand or other body part of the viewer is moved faster than a predetermined speed. This is done by making the reciprocating motion constitute the gesture of the selection command when it is reciprocated quickly.

[0120] In addition, the selection gesture may be a rotational motion of a part of the viewer's body such as the hand or the like, that is, a motion that draws a circle in a plane parallel to the display screen 4.

[0121] When detecting that the movement of the viewer's hand is the movement of the selection command, the image processing unit 6 transmits a detection signal to the display control unit 7.

[0122] Upon receiving the detection signal from the image processing unit 6, the display control unit 7 causes the display unit 5 to display an image (such as the selected program) according to the selected function (step S6).

[0123] Thereafter, by moving the cursor and selecting the cursor in the same manner as described above, the display device 21 can be continuously controlled by the gesture command via the electronic device 40.

(4) Termination command

When ending the gesture command, the viewer instructs the end command.

[0124] The end command is indicated by a movement of the hand or other part of the viewer's body other than the movement for instructing the cursor movement command, the cursor selection command, and the target object specifying command.

[0125] For example, a gesture for reciprocating a hand or the like faster than a predetermined speed can be used as the end command. If this gesture is the same as the gesture that instructs the cursor selection command, the end command and the cursor selection command can be distinguished by the direction in which the gesture reciprocates.

[0126] In addition, a rotational motion of a part of the body of the viewer or other hand can be used as an end command. If this rotational movement is the same as the cursor selection command, depending on the direction of rotation, The end command and the cursor selection command can be distinguished.

[0127] The image processing unit 6 constantly monitors whether or not the viewer is moving to instruct the end command! / (Step S7).

[0128] When recognizing the end command (YES in step S7), the image processing unit 6 transmits an end command recognition signal to the display control unit 7. Upon receiving the end command recognition signal from the image processing unit 6, the display control unit 7 deletes the command menu display from the display unit 5 and puts the display on the display unit 5 into the designated state.

(5) Other

It is possible that the image processing unit 6 loses sight of the viewer's gesture position, for example, the position of the viewer's hand, by giving a gesture command! For example, when the viewer gives an instruction to the image processing unit 6 by a hand gesture command, if the viewer suddenly drops his hand, the image processing unit 6 positions the viewer's hand. You may lose sight.

[0129] In such a case, the image processing unit 6 transmits a signal indicating that the object 20 cannot be specified to the display control unit 7, the audio control unit 14, or the light emission control unit 12 for display. The control unit 7, the audio control unit 14, or the light emission control unit 12 notifies the viewer to that effect by the image of the display unit 5, the light emission of the notification light emitting unit 11 or the sound generation from the sound generation unit 13.

[0130] The viewer who has received the notification from the display unit 5, the notification light emitting unit 11 or the sound generation unit 13 must again enable the image processing unit 6 to specify the target object 20 using the target object specifying command. Can do.

[0131] Further, if the object 20 cannot be identified and the force cannot be recognized again after a predetermined time (for example, 5 minutes), the image processing is performed. The unit 6 considers that the end command has been issued, and transmits a signal indicating that to the display control unit 7. The display control unit 7 that has received this signal ends the command menu.

[0132] The electronic device 40 according to the third embodiment is configured to include the notification light emitting unit 11 and the sound generation unit 13 as a notification operation unit that performs a notification operation to the viewer. Depending on the situation, it may be configured to have only one of the notification light emitting unit 11 and the sound generation unit 13. Example 4

FIG. 11 is a block diagram showing a configuration of an electronic device 50 according to Embodiment 4 of the present invention.

As shown in FIG. 11, the electronic device 50 according to the fourth embodiment of the present invention is under the control of the image processing unit 6 in addition to the configuration of the electronic device 10 according to the first embodiment shown in FIG. A main power supply unit 15 for turning on the main power of the display device 21 is provided.

[0135] For example, when the image processing unit 6 specifies the object 20 by the viewer's object specifying command (gesture command) and the main power of the display device 21 is off, the image processing unit 6 outputs a command to the main power input unit 15 to turn on the main power of the display device 21

[0136] As described above, not only when the image processing unit 6 specifies the object 20 by the viewer's object specifying command (gesture command), but the image processing unit 6 turns on the main power at an appropriate time. A command can be output to the unit 15 to turn on the main power supply of the display device 21.

Example 5

The electronic device according to the fifth embodiment of the present invention has the same configuration as the electronic device 40 according to the third embodiment, and instead of the gesture command described in the third embodiment, the viewer's facial expression. The only difference is that it identifies the line of sight.

Accordingly, the operation of the electronic device according to the fifth embodiment of the present invention is to detect a gesture command.

The operation is the same as that of the flowchart shown in FIG. 10 except that instead of (step S5), the viewer's facial expression and line of sight are specified.

[0139] Conditions for specifying the viewer's facial expression and line of sight include the following points.

(1) Preparation

Setting the range of the three-dimensional space in which the viewer exists is exactly the same as using the gesture command (Example 3).

(2) Object identification command and command menu

The image processing unit 6 needs to specify the object 20 that provides the object specifying command.

[0140] As the object specifying command, an operation in which blinking of a predetermined period is continuously performed a predetermined number of times or more, that is, an operation of repeatedly closing and opening eyes is considered. Similarly, repeat a specified number of repetitions of opening and closing the mouth at a predetermined cycle. It is a command.

[0141] However, there is a problem with the resolution of cameras 1 and 2 in order to make these operations as object specifying commands. This is because higher resolution is required compared to gesture commands.

[0142] Furthermore, after “cursor movement and command selection” described below, higher resolution is required for cameras 1 and 2. For this reason, if an object 20 is identified, the cameras 1 and 2 need to be focused on the object 20 and zoomed up. It is desirable to reconfirm the object identification command again after zooming up.

[0143] When the object 20 can be identified, the image processing unit 6 sends a signal indicating that the object 20 has been identified to the display control unit 7, the light emission control unit 12, or the audio control unit 14. The information is transmitted to the viewer via the notification light emitting unit 11 and the sound generation unit 13.

[0144] Here, for example, when the main power-on unit 15 is provided as in the electronic device 50 according to the fourth embodiment, and the display device 21 is turned off, the image processing unit When 6 identifies the object 20, it sends a signal to the display control unit 7 to operate the main power supply unit 15 to turn on the main power to the display device 21 and to indicate that the object 20 has been identified. The signal shown is transmitted to the display control unit 7 and the command menu is displayed on the display unit 5.

Note that if the display device 21 is already turned on, the command menu is displayed so as to overlap the image currently displayed on the display unit 5 or to erase a part of the image.

[0146] This indicates to the viewer that the object 20 has been identified, and prompts the viewer for the next command.

(3) Cursor movement and command selection

Similar to gesture commands, the command menu includes a channel selection menu, so the viewer needs to move the cursor and select the command.

[0147] For example, the cursor can be moved by turning the line of sight in the direction in which the viewer moves the cursor. That is, the viewer's white-eye and black-eye portions are detected by the image processing unit 6, and the viewer's line of sight is shifted vertically and horizontally depending on which direction the black-eye portion is biased. You can move the cursor in that direction.

[0148] However, there is one problem with this method. Viewers turn their gaze from side to side or up and down If this is done, the display screen 4 of the display device 21 cannot be seen well, which makes it difficult for the viewer to determine the movement of the cursor.

[0149] Therefore, for example, the cursor can be moved by performing the following two-stage operation.

[0150] For example, assume that the cursor is moved to the right.

[0151] In this case, first, the viewer temporarily moves the line of sight to the right and returns the line of sight to the center again. Second, if the cursor still does not move enough, turn your line of sight to the right again.

[0152] There is also a method of moving the cursor in the winking direction by closing or opening one eye. This method has the drawback that the cursor cannot be moved only by moving left and right, but the image processing unit 6 can determine the direction of the line of sight much more easily.

[0153] As an operation for executing command selection, an operation of closing and opening both eyes can be considered. Or, it is possible to select an operation that repeats the operation of opening and closing the mouth at a predetermined cycle.

(4) Termination command

As in the case of ending the gesture command, when ending the command for detecting the viewer's line of sight or the command for reading the viewer's facial expression, the viewer presents the end command to the image processing unit 6.

[0154] The end command is indicated by a movement other than the cursor movement command, the cursor selection command, and the target object specifying command.

[0155] For example, the cursor movement is a command that detects the viewer's line of sight, such as a gesture that causes a hand or other part of the viewer's body to reciprocate faster than a predetermined speed. Can be combined as a gesture command. When the end command is the same as the cursor selection command, the end command is distinguished from the cursor selection command according to the direction in which the viewer's hand or other part of the viewer's body is reciprocated.

[0156] In addition, the rotation command of the viewer's hand or other part of the viewer's body can be used as the end command. If the end command is the same as the cursor selection command, Depending on the direction in which the other part of the viewer's body rotates, the end command can be distinguished from the cursor selection command.

[0157] When the image processing unit 6 detects the viewer's end command, it sends a command to the display control unit 7 to delete the command menu and place the image on the display unit 5 in the designated state.

[0158] In addition, when the command is not provided from the viewer for a certain period of time, the image processing unit 6 can also be regarded as having provided the end command when the certain period of time has elapsed. (5) Other

As in the case of the gesture command, when the image processing unit 6 cannot recognize the viewer's command for a certain period of time, the image processing unit 6 informs the viewer that the object 20 cannot be identified. As described above, if the viewer's command cannot be recognized for a certain period of time, it is considered that the end command has been presented and the command input screen is ended. You can also. To return to the command input screen again, the viewer needs to present an object specifying command.

[0159] Note that, in each of the above embodiments, the electronic devices 10, 30, 40, 50 applied to the display device 21 have been described, so that the position of the object 20 is within a predetermined range. In this case, the example in which the object 20 is determined to be a “viewer” has been described. However, the electronic devices 10, 30, 40, and 50 according to the above embodiments are applied to devices other than the display device. In this case, the “viewer” can be replaced with the “operator” of the device. That is, it is interpreted that there is an intention to operate the device, and the image processing unit 6 notifies the display control unit 7 to that effect, and the display control unit 7 displays an operation screen on the display unit 5. Thereafter, as in the above embodiments, for example, the camera 1 and 2 captures the gesture command, and the image processing unit 6 performs image processing to interpret the gesture command, and displays the result as the display control unit 7. Give feedback.

[0160] In the electronic devices 10, 30, 40, 50 according to the first to fifth embodiments described above, the number of cameras is 2 (see Fig. 1) or 3 (see Fig. 6). The number of cameras is 2. Alternatively, any number greater than 4 can be selected, not limited to 3. In addition, the positions where these cameras are arranged can be arbitrarily determined as necessary.

[0161] The electronic devices 10, 30, 40, and 50 according to the above-described first to fifth embodiments are provided with the display control unit 7. The display control unit 7 can be omitted if necessary. In that case, for example, the display control unit that the display device 21 originally has can be used as the display control unit 7 in the above embodiment.

The electronic devices 10, 30, 40, and 50 according to the above-described first to fifth embodiments can be attached to the display device 21 as an accessory. Alternatively, the electronic devices 10, 30, 40, and 50 according to the first to fifth embodiments described above can be incorporated in the display device 21 in advance.

FIG. 12 is a block diagram showing the structure of the display device 100 including the electronic device 10 according to Example 1 described above.

As shown in FIG. 12, display device 100 includes cameras 1 and 2, display unit 5, image processing unit 6, and display control unit 7.

FIG. 13 is a block diagram illustrating a structure of a display device 200 including the electronic device 30 according to the second embodiment.

As shown in FIG. 13, display device 200 includes cameras 1 and 2, display unit 5, image processing unit 6, display control unit 7, viewing history generation unit 8, and viewing history storage unit. 9 and.

FIG. 14 is a block diagram illustrating a structure of the display device 300 including the electronic device 40 according to the third embodiment or the electronic device according to the fifth embodiment.

As shown in FIG. 14, display device 300 includes cameras 1 and 2, display unit 5, image processing unit 6, display control unit 7, notification light emitting unit 11, and light emission control unit 12. And voice generator 13 and voice controller

14 and.

FIG. 15 is a block diagram illustrating a structure of a display device 400 including the electronic device 50 according to the fourth embodiment.

As shown in FIG. 15, display device 400 includes cameras 1 and 2, display unit 5, image processing unit 6, display control unit 7, and main power supply unit 15. .

Claims

The scope of the claims

[1] An electronic device that controls the operation of a display device including a display unit having a display screen, wherein at least two imaging units that convert an image of an object in front of the display screen into an image signal;

An image processing unit that performs signal processing on the image signal,

An electronic apparatus that controls an operation of the display unit based on a signal processing result of the image processing unit.

2. The electronic device according to claim 1, wherein the operation of the display unit is controlled according to the position or shape of the object specified by the image processing unit.

[3] The electronic device according to [1] or [2], wherein the image processing unit specifies the position of the object based on a difference between the image signals output from the imaging unit.

[4] The display control unit according to any one of claims 1 to 3, further comprising a display control unit that controls an image displayed on the display unit based on a signal processing result of the image processing unit. Electronic devices.

5. The electronic device according to claim 4, wherein the display control unit controls an image to be displayed on the display unit according to the position or shape of the object specified by the image processing unit. apparatus.

[6] The display control unit responds to the movement of the target object when the target object is located at a predetermined position or the target object has a predetermined shape. The electronic device according to claim 4 or 5, wherein the electronic device is controlled at the same time.

[7] The display control unit may select one of a plurality of menus and a plurality of menus on the display screen when the object is located at a predetermined position or when the object has a predetermined shape. 7. The electronic device according to claim 4, wherein a cursor at a position corresponding to one menu is displayed, and the cursor is moved according to the movement of the object.

[8] When the movement of the object is a predetermined movement, the display control unit determines that a menu corresponding to the cursor position is selected, and displays an image corresponding to the menu on the display unit. 8. The electronic device according to claim 7, wherein the electronic device is displayed.

[9] The display control unit according to any one of claims 4 to 8, wherein the display control unit displays, on the display screen, information indicating that the object is lost when the object is lost. An electronic device according to 1.

[10] A viewing history generation unit that generates a past viewing history in the display device;

A viewing history storage unit that stores the viewing history generated by the viewing history generation unit,

The display control unit searches the viewing history stored in the viewing history storage unit according to the position or shape of the object specified by the image processing unit and the date and time, and may be viewed The electronic device according to claim 4, wherein a program having a high price or a list thereof is displayed on the display unit.

[11] A notification operation unit that performs a notification operation to a viewer;

A notification operation control unit that controls a notification operation of the notification operation unit under the control of the image processing unit, and

The display control unit outputs an output to the notification operation control unit so that the notification operation control unit operates the notification operation unit when detecting a specific movement of the object specified by the image processing unit. The electronic device according to claim 4, wherein the electronic device emits light.

[12] An operation method of an electronic device for controlling an operation of a display device including a display unit having a display screen,

A first step of identifying an object by detecting a predetermined shape, a predetermined movement, or a predetermined movement of the predetermined shape at a predetermined position in front of the display screen; and the first step A second step of displaying a plurality of menus and a cursor at a position corresponding to any one of the plurality of menus on the display screen in response to the movement of the object. A third step to move the cursor,

A fourth step of executing a menu corresponding to the position of the cursor according to the movement of the object;

A method of operating an electronic device, comprising:

[13] When the position of the object is lost, the operator indicates that the first step is to be executed again. 13. The method for operating an electronic device according to claim 12, further comprising a fifth step of notifying the electronic device.

[14] A sixth step of determining whether or not the position of the object is at a predetermined position based on a difference between image signals output by imaging and outputting the object by at least two imaging units. 14. The method for operating an electronic device according to claim 12, wherein the electronic device is operated as described above.

15. The method of operating an electronic device according to claim 12, wherein a main power supply of the display device is turned on during the second step.

[16] a seventh step of generating and storing a past viewing history in the display device;

An eighth step of searching the viewing history according to the position or shape of the object and the date and time;

According to a search result of the viewing history, a ninth step of displaying a program or a list thereof having a high possibility of being viewed on the display unit;

16. The method for operating an electronic device according to claim 12, further comprising:

[17] A program for causing a computer to execute an operation method of an electronic device for controlling an operation of a display device including a display unit having a display screen,

The processing performed by the program is as follows:

A first process for identifying an object by detecting a predetermined shape, a predetermined movement, or a predetermined movement of the predetermined shape at a predetermined position in front of the display screen; and In response to the second process, a second process for displaying on the display screen a plurality of menus and a cursor at a position corresponding to any one of the plurality of menus, and depending on the movement of the object, A third process to move the cursor,

A fourth process for executing a menu corresponding to the position of the cursor according to the movement of the object;

The program characterized by including.

[18] a display unit having a display screen;

An electronic device according to any one of claims 1 to 11,

A display device comprising: