JP6478454B2 - Display device including imaging unit and control method thereof - Google Patents

Description

  The present invention relates to a display device including an imaging device integrally with a relatively thin display device such as an organic EL (electroluminescence), a liquid crystal, or a plasma method. More specifically, the present invention relates to a display device having an imaging function suitable for real-time bidirectional communication such as a video conference.

  In recent years, communication networks have been developed, enabling real-time two-way communication such as video conferencing. The video conference includes an imaging unit that converts a subject image into an electrical image signal, and a display unit that converts the electrical image signal into an optical signal for display. The user of the video conference talks while looking at the other party's face image displayed on the display screen, and at the same time, the user's face is photographed by the imaging means, converted into an electrical signal, and sent to the other party.

  In a conventional two-way communication device such as a video conference, the image pickup means is arranged in the peripheral portion of the display device or installed as an independent image pickup device separate from the display device. For this reason, photographing of the observer who is looking at the display screen is performed from an oblique direction. The other person's face on the display screen has a different line of sight, which can be awkward even if they talk to each other. There is a problem that it causes a sense of incongruity compared to the case of talking face to face with the other party. Further, in the form of separately manufacturing the imaging device and the display device, there is a problem that the cost increases and it is necessary to handle them as separate devices, so that the construction of the system becomes complicated.

  In order to solve these problems, Patent Document 1 discloses a display device with an imaging device. The apparatus includes a display unit including a plurality of display pixels and a light shielding member disposed between the display pixels of the plurality of display pixels, and an imaging unit disposed on the back of the display unit. The light shielding member is partially provided with an opening for guiding light to the imaging means, and the imaging means is provided with an imaging means at a position corresponding to the opening. It is possible to take an image while viewing the display means, and to match the line of sight with the opponent displayed on the display means. As a result, the communication partner can be recognized as if the user is looking at him / herself and talking to the other party who is talking in the video conference or the like. When the lines of sight of the users are matched, display control is performed to superimpose the other party's face image at the position where the camera is disposed so that the users gaze at the location of the camera.

  FIG. 8 is an explanatory diagram of an image display method of a conventional apparatus. On the back surface of the display panel 2 of the display device 1 with the imaging device, a camera 10 is disposed in the center of the display screen. FIG. 8A illustrates a display image, and the face image of the user α is displayed on the display panel 2 so as to overlap the position of the camera 10. As the background of the face image of the user α, for example, a window frame γ is reflected. FIG. 8B shows a case where the user α has moved from the state shown in FIG. The display device with an imaging device 1 performs display control so that the face image of the user α and the position of the camera 10 are superimposed. As a result, as shown in FIG. 8C, the face image of the user α is displayed at the center of the screen.

JP 2005-176151 A JP 2000-48184 A

In the conventional apparatus, when the user on the other side moves the position of the face, the position of the face image is fixed, for example, at the center of the screen where the camera is disposed, and only the background moves. Therefore, the display image appears as if so-called panning is taken, and the user may feel uncomfortable.
An object of the present invention is to provide a display device with an imaging device that can obtain an effect of matching the line of sight of users without a sense of incongruity even when a user on the other side of the conversation moves the face position.

In order to solve the above-described problems, an apparatus according to the present invention is a display device that includes a display unit and an imaging unit disposed on a back surface of the display unit, and includes a face of a subject in an image signal and an image related to the image signal. and output means for outputting the position information of the input and the imaging signal by the imaging unit obtains the information on the position of the face of the subject obtained by the input means, the time variation of the position of the face comprising calculating means for determining the display position of the face image, and a display control means for performing control to display the facial image at the display position of the face image in which the calculating means is determined in accordance with. The computing means acquires information on the observation distance of the display unit and information on the position of the observer based on the imaging signal from the imaging unit, and information on the observation distance and information on the position of the observer. Based on the position of the face of the subject, when determining the display position of the face image by determining the amount of time change of the angle between the display position of the face image and the imaging unit based on the position of the observer based on As the time change amount of the position increases, the time change amount of the angle between the display position of the face image and the imaging unit increases, and the time change of the angle between the display position of the face image and the imaging unit The display position of the image is calculated so that the amount is equal to or less than the threshold value . The threshold is an angle at which the line of sight can be recognized as matching.

  According to the present invention, it is possible to obtain an effect of matching the lines of sight of users without a sense of incongruity even when a user on the other side of the conversation moves the face position.

It is the front view (A) and top view (B) of the display apparatus with an imaging device which concern on embodiment of this invention. It is a block diagram which shows the structural example of the display apparatus with an imaging device. It is explanatory drawing which shows the example of a display of a display apparatus with an imaging device. It is a flowchart explaining operation | movement of the display apparatus with an imaging device. It is a graph explaining the example of control of a display position. It is a graph explaining face tracking processing. It is the front view (A) and top view (B) of the display apparatus with an imaging device which concern on deformation | transformation embodiment of this invention. It is explanatory drawing of the display method by a prior art.

Embodiments of the present invention will be described below with reference to the accompanying drawings. The display device according to the present embodiment includes a display unit and an imaging unit arranged on the back surface of the display unit. With reference to FIG. 1 to FIG. 7, an example in which a plurality of users conduct a video conference using the display device with an imaging device according to the present embodiment (hereinafter simply referred to as a display device) will be described. In the present embodiment, the user β uses the video conference in the first space (hereinafter referred to as space 1), and the user α uses the video conference in a second space (hereinafter referred to as space 2) different from the space 1. An example is shown.

  FIG. 1A is a front view of a display device 101 used in a video conference system in the space 1 or the like. The user α in the space 2 is displayed on the display screen of the display device 101. FIG. 1B is a top view of the display device 101 and shows a positional relationship with the user β who observes the display device 101 in the space 1. Hereinafter, the direction perpendicular to the display surface of the display device 101 is defined as the z direction (the back side is the positive direction). The horizontal direction orthogonal to the z direction is defined as the x direction (right side is the positive direction), and the vertical direction is defined as the y direction (upper direction is the positive direction). In FIG. 1B, the distance between the user β and the display surface is Zβ, and the amount of deviation in the x direction of the user β with reference to the photographing optical axis is Xβ.

  The display panel 102 is display means that can transmit light. On the back surface of the display panel 102 (in the + z direction in the figure), a camera 11 is arranged as an imaging unit for photographing a user who observes the display panel 102. The camera 11 includes a lens group that constitutes an imaging optical system, an imaging element, and an imaging element drive circuit. The camera 11 is arranged slightly above the center of the screen of the display panel 102 (in the + y direction in the figure), and photographs the user β through a part of the display pixel portion of the display panel 102. The display panel 102 is composed of an organic EL, and is configured so that light is transmitted through the photographing optical path of the camera 11. Further, around the display panel 102, microphones 121, 122, 123 and 124 as voice detecting means for detecting a user's voice and speakers 131 and 132 for transmitting the voice detected in the space 2 are arranged. In addition, since it is the same structure also about the display apparatus used in the other party's space 2, description is abbreviate | omitted.

  Next, the control operation of the display device 101 will be described with reference to FIGS. By using the display device 101 for a video conference in different spaces, it is possible to obtain an effect of matching the lines of sight of users and to display an image with no sense of incongruity in movement. It is assumed that the user β participates in the video conference in the space 1 and the user α participates in the video conference in the space 2.

  FIG. 2 is a block diagram illustrating a configuration example of the display device 101. The image sensor 110 receives light from a subject formed through the imaging optical system and photoelectrically converts it. The image sensor drive circuit 51 drives the image sensor 110 in accordance with a control command from the system control circuit 50. The display processing circuit 52 is a display control unit of the display panel 102, and the display panel 102 displays an image according to the display image signal. The microphone 120 (see FIG. 1A: 121 to 124) outputs a detection signal to the sound processing circuit 53. The speaker 130 (see FIG. 1A: 131 and 132) is connected to the drive circuit 54 and outputs sound.

  A system control circuit 50 that controls each part of the display device 101 includes a CPU (Central Processing Unit), reads out and executes a control program from a memory, and performs an image sensor driving circuit 51, a display processing circuit 52, an audio processing circuit 53, and a driving The circuit 54 and the like are controlled. The operation switch (SW) 40 is an operation member used by the user when starting the video conference system, and outputs an operation signal to the system control circuit 50. The remote control circuit 55 is used by the user when operating the display device 101, and outputs an operation signal to the system control circuit 50. The transmission / reception circuit 56 is signal input / output means for transmitting and receiving image signals and audio signals to and from the counterpart display device. That is, the transmission / reception circuit 56 performs an input process of an image signal from the counterpart display device and an output process of outputting the image pickup signal of the camera 11 from the counterpart display device. The arithmetic circuit 57 acquires image information taken by the camera 11 and performs arithmetic processing such as user face recognition. The storage circuit 58 stores various information used in the system, a control program, and the like. The transmission / reception circuit 56, the arithmetic circuit 57, and the storage circuit 58 are connected to the system control circuit 50.

FIG. 4 is a flowchart for explaining the operation of the display device 101.
When the video conference user β operates the operation switch 40 of the display device 101 in the space 1, the video conference system is activated (S100). The system control circuit 50 sets a communication state with the display device in the space 2 through the transmission / reception circuit 56 (S101). When the communication state with the counterpart display device is set, the system control circuit 50 checks the system information of the counterpart display device (S102). The system information is information such as the number of cameras included in the display device, focal length information and position information of the cameras, and the form of signals to be transmitted. The information on the form of the transmitted signal is, for example, information on the frame period of the transmitted image signal.

  Further, the system control circuit 50 initializes the user information of the space 1 (S103). The user information in the present embodiment is information such as the number of users, the positions of the eyes of speakers who are having a conversation, and the positions of eyes of users other than the speakers, and is stored in the storage circuit 58. . Next, the camera 11 captures the user β in the space 1 (S104). At this time, the system control circuit 50 issues a shooting instruction to the image sensor 110 via the image sensor drive circuit 51. At the same time, the audio processing circuit 53 processes the audio signal detected by the microphones 121 to 124 into a predetermined signal form that can be transmitted to the counterpart display device.

  When the space 11 is photographed by the camera 11, the system control circuit 50 transmits an image signal, an audio signal, and a user information signal to the counterpart display device via the transmission / reception circuit 56 (S105). That is, the transmission processing of the user information of the space 1 stored in the storage circuit 58 is executed together with the image signal captured by the camera 11 and the audio signal processed by the audio processing circuit 53. Image signals captured by the camera 11 are sequentially transmitted in time series.

  Next, the arithmetic circuit 57 performs face recognition processing of the user β from the image of the space 1 photographed by the camera 11 (S106). At this time, the arithmetic circuit 57 detects the position of the eye in the face area (S107), and calculates the displacement amount Ixβ of the center position of the user β with respect to the photographing optical axis of the camera 11. The system control circuit 50 stores the calculation result as user information in the storage circuit 58 (S108). A method for recognizing a user's face from an image is disclosed in Patent Document 2 by the present applicant. In this face area extraction process, a skin color area is detected, and after the creation process for creating a projection distribution of the skin color area, a search process for searching for a parabola from the created projection distribution is performed. A face area candidate is extracted from the searched parabola position, and it is determined whether the face area candidate is a face area by knowledge processing.

In addition, the arithmetic circuit 57 calculates a distance Zβ to the position where the user β is observing the display device 101 from the interval between the eyes of the face image of the captured user β. Since the distance between adult eyes is approximately 65 mm, the observation distance Zβ can be calculated based on the imaging magnification of the camera 11.
Next, the system control circuit 50 receives the image signal, the audio signal, and the user information transmitted from the other party display device of the video conference via the transmission / reception circuit 56 (S109). The user information of the space 2 includes the face position Ixα of the user α. This position information is stored in the storage circuit 58 in consideration of the screen size of the display panel 102 of the display device 101.

The arithmetic circuit 57 sets the display position Dxα on the display panel 102 for the face image of the user α in the space 2 in the received image according to the observation position of the user β in the space 1 (S110). The process for setting the display position of the received image will be described later with reference to FIG.
The system control circuit 50 displays an image of the user α in the space 2 on the screen of the display panel 102 via the display processing circuit 52 (S120). FIG. 3 illustrates an image of the user α in the space 2 displayed on the display panel 102. The display image is displayed with a smooth movement of the face and background as if the camera were panned small. Since the user β in the space 1 has a line of sight with the user α in the space 2, the communication is improved and the user is not uncomfortable with the display change.

  When the image of the user α in the space 2 is displayed on the display panel 102, the sound processing circuit 53 selects a speaker who is talking in the space 1 from the sound signals detected by the microphones 121, 122, 123 and 124. It is detected (S121). As a method for detecting a speaker from a plurality of audio signals, for example, there is a delay sum method. When the position of the speaker who is talking is detected, the position data of the user's eyes stored in the storage circuit 58 is updated as the position data of the eyes of the speaker (S122). Then, the system control circuit 50 confirms the state of the operation switch 40 and determines the end of communication (S123). When the video conference system is in use, the process returns to S105 to continue transmission / reception of a photographed image or the like.

  On the other hand, when the system control circuit 50 determines that the operation switch 40 is turned off by the user (Yes in S123), the transmission / reception with the display device on the other end is terminated (S124). Further, the system control circuit 50 ends the user's photographing by the camera 11 and ends the image display on the display panel 102 (S125).

Next, referring to the flowchart of FIG. 4B, the predefined process of S110 of FIG. 4A, that is, the face of the user α of the image sent from the counterpart display device is displayed on the display panel 102. Processing for setting the display position Dxα (see FIG. 1A) will be described.
First, the system control circuit 50 recognizes the center position of the face from the position of the eyes of the user α using the user information in the space 2, and determines the distance Ixα between the center position of the face of the user α and the center of the display screen. Confirm (S111). FIG. 1A illustrates a state in which an image sent from the counterpart display device is displayed on the screen of the display device 101 as it is. FIG. 1A shows an example in which the distance Ixα between the center position of the face of the user α and the center of the display screen matches the display position Dxα of the face of the user α on the display panel 102.

The arithmetic circuit 57 determines the characteristics of the received image signal from the position of the eyes of the user α in the space 2 and the system information of the display device in the space 2 that has already been confirmed (S112). That is, the arithmetic circuit 57 changes the time position of the face of the user α based on the center position information Ixα of the face of the user α stored in the storage circuit 58 and the frame period Δt of the received image. The quantity ΔIxα is calculated. The position of the face of the user α in the space 2 in the previous received image is Ixα0, and the position of the face of the user α in the space 2 in the current received image is Ixα1. The time change amount ΔIxα indicating the change amount of the position with respect to time is
ΔIxα = (Ixα1-Ixα0) / Δt (1)

  The system control circuit 50 confirms the position information Zβ of the user β in the space 1 stored in the storage circuit 58 (S113). The system control circuit 50 checks the time change amount ΔIxα calculated by the arithmetic circuit 57 using the equation (1) (S114). When the position of the face of the user α in the space 2 has changed, the arithmetic circuit 57 calculates the display position Dxα of the face of the user α in the space 2 on the screen of the display panel 102 (S115). If the position of the face of the user α in the space 2 has not changed in S114, the process proceeds to S116.

In this embodiment, even when the position of the face of the other user moves, the user's line of sight is matched and control is performed so that the face and background move smoothly. Do. The arithmetic circuit 57 determines the display position Dxα of the user α according to the time change amount ΔIxα in the equation (1). In the top view of FIG. 1B, the case where the image of the user α sent from the partner display device is displayed as it is on the screen of the display device 101 is illustrated, and the gaze position of the user β in the space 1 and the camera 11 shows the line-of-sight angle θβ formed with the angle 11. Since the user β in the space 1 gazes at the display position Dxα of the face of the user α in the space 2, the line-of-sight deflection angle θβ is
tan (θβ) ≈Dxα / Zβ The relational expression (2) is satisfied. tan (θβ) represents a tangent function of the line-of-sight angle θβ.

Further, the angle θβ formed between the gaze position of the user β and the camera 11 is controlled so as to be within an angle limit θβth where the line of sight can be recognized as matching. In the present embodiment, the limit angle θβth is set to 2 [°].
Here, the display position on the display panel 102 of the face of the user α in the previous received image is Dxα0, and the angle formed by the display position Dxα0 and the camera 11 with respect to the position of the observer is θβ0. Furthermore, the display position of the face of the user α in the current received image on the display panel 102 is Dxα1, and the angle between the display position Dxα1 and the camera 11 with respect to the position of the observer is θβ1. When the amount of time change is Δθβ, the line-of-sight deflection angle is
θβ1 = θβ0 + Δθβ <θβth The expression (3) is satisfied.

FIG. 5 is a diagram for explaining the control of the display position of the face image. The horizontal axis indicates the time change ΔIxα (unit: mm) of the position of the face of the user α taken in the space 2, and the vertical axis is the usage displayed on the display device 101 for the user β in the space 1. A control amount Δθβ (unit: degree) of an angle between the face position of the person α and the camera 11 is shown.
FIG. 5A shows a control example in the case where the angle θβ0 formed by the previous position αα0 of the user α and the camera 11 is zero. That is, the control example when the face image of the user α in the space 2 is located on the optical axis of the camera 11 in the previous received image is shown.
In the present embodiment, when the time change amount ΔIxα of the face position of the user α is equal to or less than a predetermined threshold (for example, a range of control amount Δθβ ≦ 1 [°]), the time change amount ΔIxα is linearly converted. Thus, the control amount Δθβ is set. When the time change amount ΔIxα is larger than the predetermined threshold value, the received image is not displayed as it is, and the image display is performed after correcting the display position so as not to change greatly. The angle control amount Δθβ at this time is set so as to be within the limit value of the angle at which the line of sight can be recognized as matching (θβth or less).

As described above, the face of the user α displayed on the screen of the display device 101 visually recognized by the user β in the space 1 with respect to the time change amount ΔIxα of the position of the user α photographed in the space 2. And the control amount Δθβ of the angle formed by the position of the camera 11 is determined. The arithmetic circuit 57 determines the angle control amount Δθβ to be equal to or less than a predetermined value so that the display change amount does not become too large. When the angle control amount Δθβ is determined, the arithmetic circuit 57 calculates the face display position Dxα1 on the display panel 102 of the user α in the space 2 using the above equations (2) and (3) (S115). The face display position is determined (S116). Thereafter, the process returns to the main routine of FIG. 4A (S117).
FIG. 5A shows an example of control when the face image of the user α in the space 2 is positioned on the optical axis of the camera 11 in the previous received image. FIG. 5B shows a control example when the face image of the user α is not located on the optical axis of the camera 11 in the previous received image. The case where the angle θβ0 formed by the display position DXα0 of the face of the user α and the camera 11 is, for example, 1 [°] is illustrated.

  In FIG. 5B, when the angle θβ0 between the display position DXα0 of the face of the user α and the camera 11 is equal to or larger than a predetermined value, the image is displayed after correcting so that the display position does not change greatly. Even when the amount of time change ΔIxα of the face position is small, the display position is corrected without displaying the received image as it is. The angle control amount Δθβ at this time is set so as to be within the limit value of the angle at which the line of sight can be recognized as matching (θβth or less).

In the present embodiment, an example is shown in which the arithmetic circuit 57 acquires the face position Ixα of the user α in the space 2 from the received image and determines the display position Dxα of the user α face on the display panel 102. . On the other hand, by predicting the movement of the face in advance, the image can be displayed smoothly without delay. FIG. 6 is an explanatory diagram of face tracking, where the horizontal axis indicates time and the vertical axis indicates an angle corresponding to the movement of the face image. The arithmetic circuit 57 monitors temporal changes in the position of the face (solid line area in the figure) and extrapolates the future face position to predict the movement of the face image (dotted line area in the figure). The display position of the face image is determined. Thereby, an image can be displayed smoothly without delay.
In the present embodiment, when the other user who performs the conversation moves the position of the face, only the background does not move in an attempt to fix the position of the face image to the center of the screen where the imaging unit is arranged. Thus, the effect of matching the lines of sight of the users can be obtained without a sense of incongruity.

  In the present embodiment, the example in which the observation distance Zβ of the user β who observes the display device 101 is calculated based on the interval between the eyes of the user β has been described. However, the viewing distance is generally determined by the size of the display panel 102, and therefore the viewing distance Zβ according to the size of the display panel 102 may be automatically set.

[Modified Embodiment]
Next, a modified embodiment of the present invention will be described.
FIG. 7A is a front view of the display device 201 used in the video conference system in the space 2. A user β in the space 1 is displayed on the display screen of the display device 201. FIG. 7B is a top view of the display device 201. The positional relationship with the user α who observes the display device 201 in the space 2 is shown. The display device 201 has the same configuration as the display device 101 used in the space 1.
As shown in FIG. 7A, a non-display area 203 is set in a part of the image of the space 1 displayed on the screen of the display panel 202. As shown in FIG. 7B, when the user α gazes at the display panel 202 at a position Xα away from the photographing optical axis of the camera 211, a non-display area 203 is set on the display screen of the display panel 202. . The purpose of setting the non-display area 203 is to guide the observation position closer to the photographing optical axis of the camera 211 with respect to the observer. The process for determining whether or not to set the non-display area 203 on the display panel 202 is performed as follows.

The system control circuit 50 photographs the user α with the camera 211 provided on the back surface of the display panel 202. Next, the arithmetic circuit 57 executes a face recognition process for the user α in the space 2 from the image captured by the camera 211. At this time, the arithmetic circuit 57 detects the position of the eye in the face area of the user α, and calculates the displacement amount Xα of the center position of the user α with respect to the photographing optical axis of the camera 211. The displacement amount Xα of the center position of the face of the user α relative to the imaging optical axis of the camera 211 in real space is converted from the imaging magnification of the camera 211. When the displacement amount Xα is equal to or larger than a predetermined value (determination reference value), the arithmetic circuit 57 sets the non-display area 203 of the display panel 202 in order to guide the user α closer to the photographing optical axis of the camera 211. The system control circuit 50 displays the image of the user β in the space 1 together with the non-display area 203 on the screen of the display panel 202 via the display processing circuit 52. The user α who has seen the display screen in FIG. 7A moves to an observation position where a screen without the non-display area 203 is displayed.
According to the modified embodiment, the relative position between the face image of the user and the imaging unit is detected, and the user is guided closer to the photographing optical axis of the camera 211, so that the lines of sight of the users are matched. Play.

11, 211 Camera 101, 201 Display device 102, 202 Display panel 121-124 Microphone 131-132 Speaker

Claims (4)

A display device comprising a display unit and an imaging unit disposed on the back of the display unit,
Input / output means for inputting information of the position of the face of the subject in the image signal and the image related to the image signal and outputting the imaging signal by the imaging unit;
Acquires information on the position of the face of the subject obtained by the input means, and calculating means for determining the display position of the face image according to the time variation of the position of the face,
Display control means for performing control to display the face image at the display position of the face image determined by the calculation means,
The computing means is
Based on the imaging signal by the imaging unit, obtain information on the observation distance of the display unit and information on the position of the observer ,
Based on the information on the observation distance and the information on the position of the observer, the time change amount of the angle between the display position of the face image and the imaging unit based on the position of the observer is determined. When determining the display position of the face image,
As the time change amount of the face position of the subject increases, the time change amount of the angle between the display position of the face image and the imaging unit increases, and the display position of the face image and the imaging unit The display position of the image is calculated so that the time change amount of the angle is equal to or less than a threshold value ,
The display device according to claim 1, wherein the threshold is an angle at which the line of sight can be recognized as matching . The display device according to claim 1, wherein the calculation unit determines a display position of the face image by predicting a temporal change amount of the face position. The calculation means detects a relative position between the position of the face image captured by the imaging unit and the imaging unit, determines a non-display area of the image on the display screen of the display unit,
Said display control means display device according to claim 1 or 2, characterized in that the control for displaying the facial image and the computing means decides the non-display area on the display unit. A control method executed in a display device including a display unit and an imaging unit disposed on the back of the display unit,
An input / output step for inputting an image signal and information on the position of the face of the subject in the image related to the image signal by the input / output means, and outputting an imaging signal by the imaging unit;
A calculating step of determining the display position of the face image obtained information on the position of the face of the subject acquired computing means is, according to the time variation of the position of the face by said input means,
A display control step in which display control means performs control to display the face image at the display position of the face image determined by the calculation means, and
In the calculation step, the calculation means includes:
Based on the imaging signal by the imaging unit, obtain information on the observation distance of the display unit and information on the position of the observer ,
Based on the information on the observation distance and the information on the position of the observer, the time change amount of the angle between the display position of the face image and the imaging unit based on the position of the observer is determined. When determining the display position of the face image,
As the time change amount of the face position of the subject increases, the time change amount of the angle between the display position of the face image and the imaging unit increases, and the display position of the face image and the imaging unit The display position of the image is calculated so that the time change amount of the angle is equal to or less than a threshold value ,
The method for controlling a display device, wherein the threshold value is an angle at which the line of sight can be recognized as matching .

Images