JP2015201669A - Display control device of on-vehicle television telephone and display control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display control device of an on-vehicle television telephone and display control method capable of enabling a caller to make a voice telephone call while obtaining the expression of a communication party during the travel of a vehicle.SOLUTION: A display control device includes: an image reproduction section 101 for background-reproducing the image of a television telephone transmitted from a call device of a communication party; an expression change detection section 102 for detecting the occurrence of a change satisfying a fixed condition in the expression of the communication party from the background-reproduced image; an image extraction section 103 for extracting a frame image with the expression change detected therein from the image when the expression change of the communication party is detected; and a display control section 104 for allowing a display 12 to display the extracted frame image as a still image. Whenever the expression of the communication party is changed, a frame image with the expression appearing thereon is displayed by the display 12.

Description

  The present invention relates to a display control device and a display control method for an on-vehicle videophone.

  Conventionally, the use of a videophone system using an IP (Internet Protocol) telephone or the like has become widespread as one of communication means. In the videophone system, each of both parties can make a voice call with the other party while viewing the video sent from the other party in real time. Therefore, both callers can make a voice call while grasping each other's facial expressions.

  As a technology related to a videophone, for example, in Patent Document 1 below, when there is a change in the facial expression of a caller photographed by a camera, facial parts corresponding to the changed part (for example, eyes, eyebrows, A technique for calling a three-dimensional image of a nose, a mouth, and an ear from a memory and generating and transmitting a motion vector of the facial part is disclosed. According to this technology, it is said that privacy-protected image transfer can be realized with a small data transfer amount.

  In the future, videophone systems using in-vehicle communication devices are expected to become popular. However, if the videophone video sent from the other party's call device is displayed on the in-vehicle display while the vehicle is running, the driver may stare at the display, which is a safety issue. is there. For this reason, in a conventional in-vehicle communication device, a videophone video is displayed only when the vehicle is stopped, and the videophone video display disappears when the vehicle starts running.

  For example, in the videophone device described in Patent Document 2, the image of the other party is displayed on the display while the videophone device is moving (while the vehicle is running) so that the image of the videophone does not hinder driving. Instead, only a voice call is made, and when the videophone device is stopped, an image of the other party is displayed on the display and a voice call can be made. In particular, in the videophone device described in Patent Document 2 below, communication of an image that is not displayed while the videophone device is moving is effectively wasted. By notifying the telephone device, the frame rate of the image sent from the other party's videophone device is reduced.

JP 2002-176632 A JP 2006-333152 A

  Video calls are superior to simple voice calls in that you can know the expression of the other party from the image. However, the videophone device described in Patent Document 2 employs a configuration in which the image of the other party is not displayed on the display while the videophone device is moving. For this reason, the videophone device described in Patent Document 2 has a problem that the caller cannot know the expression of the other party during the movement of the videophone device.

  The present invention has been made to solve such a problem, and in a videophone system using a call device mounted on a vehicle, the caller recognizes the facial expression of the other party while the vehicle is running. The purpose is to be able to make a call.

  In order to solve the above-described problems, in the present invention, a videophone video sent from the other party's call device is played back in the background, and certain conditions in the other party's facial expression are determined from the background played video. Detect changes that meet. A frame image in which a change that satisfies a certain condition is detected is extracted from the video, and the extracted frame image is displayed on the display as a still image.

  According to the present invention configured as described above, when the expression of the other party is changed, a frame image (that is, a still image) at the time when the change is detected is sent from the calling apparatus of the other party. It is extracted from the video (i.e. video) and displayed on the display. For this reason, even if it is a vehicle-mounted apparatus in which the display of the moving image during the driving | running | working of a vehicle is prohibited, a part of moving image can be displayed as a still image. Thereby, even when the vehicle is traveling, the caller can make a voice call while grasping the facial expression of the other party by checking the still image displayed on the display from time to time.

It is a block diagram which shows an example of a structure of the vehicle equipment which concerns on this embodiment. It is a flowchart which shows an example of the process by the vehicle equipment which concerns on this embodiment. It is a figure which shows an example of the display control by the vehicle equipment which concerns on this embodiment. It is a block diagram which shows the modification of the structure of the vehicle equipment which concerns on this embodiment. It is a flowchart which shows the modification of the process by the vehicle equipment which concerns on this embodiment.

[Example of configuration of in-vehicle device 10]
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an example of the configuration of the in-vehicle device 10 according to the present embodiment. FIG. 1 illustrates functions and devices related to the display control process of the present embodiment among the functions and devices included in the in-vehicle device 10.

  As shown in FIG. 1, the in-vehicle device 10 of this embodiment includes a speaker 11 and a display 12. The in-vehicle device 10 is connected to a microphone 30, a camera 40, and a vehicle speed sensor 50 mounted on the vehicle. The in-vehicle device 10 has functions as the display control device 100 and the call control unit 110. The display control apparatus 100 includes, as its functional configuration, a video reproduction unit 101, a facial expression change detection unit 102, an image extraction unit 103, a display control unit 104, a measurement unit 105, an elapsed time determination unit 106, a running state determination unit 107, and a notification sound. A generation unit 108 and a video type determination unit 109 are provided.

  Each of the functional blocks can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each functional block is actually configured by including a CPU, RAM, ROM, etc. of a computer, and a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory operates. Is realized.

  The call control unit 110 controls videophone calls by the in-vehicle device 10. For example, the call control unit 110 establishes a call connection for performing a videophone call with the other party's call device via the communication network 20 (various telephone lines, the Internet, etc.). In addition, the call control unit 110 generates a video call video by superimposing the voice input by the microphone 30 on the video captured by the camera 40, and the call video is transmitted to the other party via the communication network 20. Send to device. In addition, the call control unit 110 receives a video-phone call video transmitted from the other party's call device via the communication network 20. The call video received by the call control unit 110 is supplied to the video playback unit 101.

  The video playback unit 101 performs background playback of the call video received by the call control unit 110. The background reproduction here means that the audio superimposed on the video is output from the speaker 11 without displaying the video (moving image) constituting the call video on the display 12. The video playback unit 101 can also play video other than videophone video (for example, TV video, DVD (Digital Versatile Disk) video, Internet video, etc.).

  The facial expression change detection unit 102 detects a change that satisfies a certain condition in the facial expression of the other party from the video reproduced in the background by the video reproduction unit 101. For example, the facial expression change detection unit 102 extracts a frame image from which a facial expression change is to be detected from a video every multiple frames (for example, every 10 frames) or every predetermined time (for example, every second), and extracts this time. A change that satisfies a certain condition is detected based on the difference between the frame image thus extracted and the previously extracted frame image. The change that satisfies a certain condition is, for example, a case where the amount of change in a specific part (for example, mouth corner, eyes, eyebrows, etc.) of the other party's face exceeds a predetermined threshold. It should be noted that various known methods can be applied to the detection of the change in facial expression, and detailed description thereof is omitted here.

  When the facial expression change detection unit 102 detects a change that satisfies a certain condition of the other party's facial expression, the image extraction unit 103 uses the video reproduction unit 101 to convert a frame image in which the change that satisfies the certain condition is detected. Extract from the ground playback video.

  The display control unit 104 displays on the display 12 in accordance with the type of video reproduced by the video reproduction unit 101 (determination result by the video type determination unit 109) and the running state of the vehicle (determination result by the traveling state determination unit 107). To control.

  For example, when a videophone video is being played and the vehicle is running, the display control unit 104 causes the display 12 to display the frame image extracted by the image extraction unit 103 as a still image. On the other hand, when the videophone video is being played back and the vehicle is stopped, the display control unit 104 causes the display 12 to display the played videophone video (moving image).

  Further, when a video other than a videophone video (for example, a TV video, a DVD video, an Internet video, etc.) is being reproduced, and the vehicle is running, the display control unit 104 displays a predetermined warning message (for example, “The image cannot be displayed because the vehicle is running”) is displayed on the display 12. On the other hand, when a video other than a videophone video is being played and the vehicle is stopped, the display control unit 104 causes the display 12 to display the played video (moving image).

  When the frame image extracted by the image extraction unit 103 is displayed on the display 12, the measurement unit 105 measures an elapsed time after the frame image is displayed on the display 12. The elapsed time determination unit 106 determines whether or not the elapsed time measured by the measurement unit 105 has reached a predetermined time.

  The display control unit 104 does not display a new frame image on the display 12 until the elapsed time determination unit 106 determines that “the elapsed time measured by the measurement unit 105 has reached a predetermined time”. Thereby, in the vehicle equipment 10, the frequency of switching the frame image displayed on the display 12 is suppressed. For this reason, it is preferable to set a reasonably long time (for example, 5 seconds) so that the frame image displayed on the display 12 is not frequently switched as the predetermined time.

  The traveling state determination unit 107 determines whether or not the vehicle on which the in-vehicle device 10 is mounted is traveling. For example, the traveling state determination unit 107 determines that the vehicle is traveling when the number of vehicle speed pulses from the vehicle speed sensor 50 installed in the vehicle is 1 or more. On the other hand, when the number of vehicle speed pulses from the vehicle speed sensor 50 is 0, the traveling state determination unit 107 determines that the vehicle is not traveling (that is, stopped). The determination result by the traveling state determination unit 107 is used for display control by the display control unit 104. The traveling state determination unit 107 may determine whether or not the vehicle is traveling based on a brake signal, a GPS signal, or the like.

  The video type determination unit 109 determines whether the video reproduced by the video reproduction unit 101 is a videophone video. For example, when the application that instructs the video playback unit 101 to play a video is a video phone application, the video type determination unit 109 is a video phone video. Is determined. The determination result by the video type determination unit 109 is used for display control by the display control unit 104.

  The notification sound generation unit 108 generates a notification sound from the speaker 11 when the frame image extracted by the image extraction unit 103 is displayed on the display 12 under the control of the display control unit 104.

[Example of processing by in-vehicle device 10]
FIG. 2 is a flowchart showing an example of processing by the in-vehicle device 10 according to the present embodiment. The processing shown in FIG. 2 is started when, for example, a video playback instruction is given to the video playback unit 101 from a videophone application or other applications.

  First, the video playback unit 101 plays back a video in the background (step S202). As a result, the audio superimposed on the video is output from the speaker 11, but the video itself is not displayed on the display 12 at this time. Here, not only videophone video but also other video (for example, TV video, DVD video, Internet video, etc.) can be played in the background.

  Next, the traveling state determination unit 107 determines whether or not the vehicle is traveling (step S204). Here, when it is determined that the vehicle is not running (step S204: No), the display control unit 104 displays the video (moving image) reproduced in the background in step 202 on the display 12 (step S206). And the vehicle equipment 10 advances a process to step S228.

  On the other hand, when it is determined that the vehicle is traveling (step S204: Yes), the video type determination unit 109 determines whether the video reproduced in the background in step S202 is a videophone video ( Step S208). Here, when it is determined that the video reproduced in the background is not a videophone video (step S208: No), the display control unit 104 displays a predetermined warning message on the display 12 (step S210). And the vehicle equipment 10 advances a process to step S228.

  On the other hand, when it is determined that the video reproduced in the background is a videophone video (step S208: Yes), the in-vehicle device 10 performs the processing after step S212 (videophone video display control processing).

  When performing the process after step S212 for the first time (step S212: Yes), the image extraction unit 103 extracts the frame image of the first frame from the video reproduced in the background at step S202 (step S214). The display control unit 104 displays the frame image on the display 12 (step S216). At this time, the measurement unit 105 starts measuring the elapsed time after the frame image is displayed. When the process after step S212 is performed after the second time (step S212: No), the process of step S214 and the process of step S216 need not be executed.

  After that, the facial expression change detection unit 102 performs a change detection process that satisfies a certain condition in the facial expression of the communication partner from the video reproduced in the background in step S202 (step S218a). Here, when the change which satisfy | fills a certain condition in a call partner's facial expression is not detected (step S218a: No), the vehicle equipment 10 advances a process to step S228.

  On the other hand, when a change that satisfies a certain condition in the expression of the other party is detected in step S218a (step S218a: Yes), the elapsed time determination unit 106 displays the elapsed time measured by the measurement unit 105 (the frame image is displayed on the display). It is determined whether or not (elapsed time since being displayed at 12) has reached a predetermined time (step S220a). Here, when it determines with the elapsed time measured by the measurement part 105 not having reached predetermined time (step S220a: No), the vehicle equipment 10 advances a process to step S228.

  On the other hand, when it is determined that the elapsed time measured by the measurement unit 105 has reached a predetermined time (step S220a: Yes), the image extraction unit 103 displays the frame image at that time as a background reproduction in step S202. (Step S222). Then, the display control unit 104 displays the frame image extracted in step S222 on the display 12 (step S224). Further, the notification sound generator 108 generates a notification sound from the speaker 11 (step S226). And the vehicle equipment 10 advances a process to step S228.

  In step S228, the in-vehicle device 10 determines whether or not the reproduction of the video reproduced in the background in step S202 is finished (step S228). Here, when it is determined that the reproduction of the video has not ended (step S228: No), the in-vehicle device 10 executes the processes after step S204 again. On the other hand, when it is determined that the video reproduction has ended (step S228: Yes), the in-vehicle device 10 ends the series of processes illustrated in FIG.

[Example of display control by in-vehicle device 10]
FIG. 3 is a diagram illustrating an example of display control by the in-vehicle device 10 according to the present embodiment.

  In FIG. 3, the left column in the figure shows the structure of the frame image of the video 300 reproduced in the background by the video reproduction unit 101 while the vehicle is running. Video 300 is a videophone video. In the example illustrated in FIG. 3, the video 300 includes a frame image f1 to a frame image f10. These frame images f <b> 1 to f <b> 10 are frame images extracted from the video 300 every several frames as frame images to be detected as facial expression changes.

  In FIG. 3, the right column in the figure shows frame images (still images) extracted from the video 300 by the image extraction unit 103 and displayed on the display 12 under the control of the display control unit 104.

  As described above, the vehicle-mounted device 10 of the present embodiment detects a change that satisfies a certain condition in the expression of the other party in the videophone video when the videophone video display control process is performed for the first time. In this case, a frame image is extracted from the videophone video, and the frame image is displayed on the display 12. And after the vehicle equipment 10 of this embodiment displays a frame image on the display 12, it continues displaying the frame image on the display 12 until the next frame image is displayed.

  For example, in the example shown in FIG. 3, the frame image f1 is extracted from the video 300 when the videophone video display control process is performed for the first time, and the frame image f1 is displayed on the display 12. Thereafter, the frame image f <b> 1 continues to be displayed on the display 12 until a change in facial expression that satisfies a certain condition is detected in the video 300.

  In the example shown in FIG. 3, a change in facial expression that satisfies a certain condition is detected in the frame image f4. For example, in the frame image f4, as a change in facial expression that satisfies a certain condition, it is detected that the amount of increase in the mouth corner and the amount of decrease in the eyelash exceed predetermined thresholds (change to a laughed expression). Yes. At this time, a predetermined time has elapsed since the frame image f1 was displayed. For this reason, the frame image displayed on the display 12 is switched from the frame image f1 to the frame image f4. At this time, a notification sound is emitted from the speaker 11 to notify the user of the switching. Thereafter, the frame image f4 continues to be displayed on the display 12 until the next change in facial expression that satisfies a certain condition is detected in the video 300.

  In the example shown in FIG. 3, a change in facial expression that satisfies a certain condition is detected in the frame image f8. For example, in the frame image f8, it is detected that the amount of lowering of the mouth corner and the angle of the eyebrows exceed predetermined thresholds (changed to an angry facial expression) as a change in facial expression that satisfies a certain condition. . At this time, a predetermined time has elapsed since the frame image f4 was displayed. For this reason, the frame image displayed on the display 12 is switched from the frame image f4 to the frame image f8. At this time, a notification sound is emitted from the speaker 11 to notify the user of the switching. Thereafter, the frame image f <b> 8 continues to be displayed on the display 12 until the next change in facial expression that satisfies certain conditions is detected in the video 300.

  In the example shown in FIG. 3, in the frame image f2 to the frame image f3, the frame image f5 to the frame image f7, and the frame image f9 to the frame image f10, the expression of the other party is slightly changed from the previous frame image. ing. However, these frame images are not the display target because the change part is a part that is not the detection target or the change amount of the change part does not exceed a predetermined threshold. As described above, the frequency of switching the frame image displayed on the display 12 can be suppressed by limiting the part to be detected for the expression change or by setting the threshold value for the expression change amount.

  As described above, according to the in-vehicle device 10 of the present embodiment, every time a change in the facial expression of the other party is detected in the videophone video, the frame image on which the facial expression is displayed is displayed on the display 12 as a still image. Can be displayed. In particular, according to the in-vehicle device 10 of the present embodiment, even though the in-vehicle device is prohibited from displaying moving images while the vehicle is traveling, the videophone video is one of them even while the vehicle is traveling. The frame image of the part can be displayed on the display 12 as a still image. Thereby, even when the vehicle is running, the caller can make a voice call while grasping the expression of the other party by checking the still image displayed on the display 12 from time to time.

  Further, according to the vehicle-mounted device 10 of the present embodiment, when a predetermined time has not elapsed since the frame image was displayed, the frame image that has already been displayed is detected even if a change in the expression of the other party is detected. It is made to display on the display 12 as it is. For this reason, by setting a reasonably long time to the predetermined time, even if the face of the other party changes continuously in a short time, the frame image displayed on the display 12 can be switched. Can be done not frequently. That is, it can be avoided that a plurality of frame images are continuously displayed at short intervals like a moving image.

  Further, according to the vehicle-mounted device 10 of the present embodiment, the notification sound is emitted from the speaker 11 when the frame image is displayed on the display 12. As a result, even if the caller is not staring at the display 12, a change in the expression of the other party is detected by the notification sound emitted from the speaker 11, and a new frame image is thereby displayed on the display 12. I can understand that. In addition, although the structure which emits a notification sound from the speaker 11 is not an essential structure, it is a preferable structure if there exists.

[Modification]
FIG. 4 is a block diagram illustrating a modified example of the configuration of the in-vehicle device 10 according to the present embodiment. FIG. 5 is a flowchart showing a modified example of processing by the in-vehicle device 10 according to the present embodiment. In the vehicle-mounted device 10 of this modification shown in FIG. 4, the facial expression change detection unit 102a refers to the determination result by the elapsed time determination unit 106a. Then, the facial expression change detection unit 102a performs facial expression change detection processing when the elapsed time determination unit 106a determines that the elapsed time measured by the measurement unit 105 has reached a predetermined time.

  Accordingly, the flowchart of FIG. 5 is changed from the flowchart of FIG. 2 so that the elapsed time determination process (step S218b) is performed before the facial expression change detection process (step S220b). That is, in the flowchart of FIG. 5, when the elapsed time measured by the measurement unit 105 reaches a predetermined time (step S218b: Yes), the facial expression change detection unit 102a performs a facial expression change detection process (step S220b). . On the other hand, when the elapsed time measured by the measurement unit 105 has not reached the predetermined time (step S218b: No), the processing advances to step S228 without performing the facial expression change detection process (step S220b).

  Also according to this modification, when a predetermined time has not elapsed since the frame image was displayed, the already displayed frame image is displayed on the display 12 as it is. For this reason, by setting a reasonably long time to the predetermined time, even if the face of the other party changes continuously in a short time, the frame image displayed on the display 12 can be switched. Can be done not frequently. That is, it can be avoided that a plurality of frame images are continuously displayed at short intervals like a moving image.

  As another modification, the expression change detection process may be performed every predetermined time. In this case, when a change in facial expression is detected, since a predetermined time has elapsed since the frame image was displayed, the frame image in which the change in facial expression is detected is displayed on the display 12 as a still image. .

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the gist or the main features thereof.

DESCRIPTION OF SYMBOLS 10 Vehicle equipment 11 Speaker 12 Display 20 Communication network 30 Microphone 40 Camera 50 Vehicle speed sensor 100 Display control apparatus 101 Image | video reproduction | regeneration part 102 Expression change detection part 103 Image extraction part 104 Display control part 105 Measurement part 106 Elapsed time determination part 107 Running state determination Unit 108 notification sound generation unit 109 video type determination unit 110 call control unit

Claims (7)

A display control device for controlling the display of an in-vehicle videophone,
A video playback unit for background playback of videophone video sent from the other party's call device;
A facial expression change detection unit for detecting a change that satisfies a certain condition in the facial expression of the communication partner from the video reproduced in the background by the video reproduction unit;
When a change that satisfies the certain condition is detected by the facial expression change detection unit, an image extraction unit that extracts a frame image in which a change that satisfies the certain condition is detected from the video; and
A display control unit for displaying the frame image extracted by the image extraction unit on a display as a still image; A traveling state determination unit that determines whether or not the vehicle is traveling;
A video type determination unit that determines whether the video played back in the background by the video playback unit is the videophone video, or
The display control unit is extracted by the image extraction unit when it is determined that the vehicle is running and when the background-played video is determined to be a videophone video. The display control apparatus for an in-vehicle videophone according to claim 1, wherein the frame image is displayed on the display. The display control of the in-vehicle videophone according to claim 2, wherein when it is determined that the vehicle is not running, the display control unit displays the image reproduced in the background on the display. apparatus. A measuring unit that measures an elapsed time after the frame image is displayed on the display;
The display control unit is configured to display the display even when the elapsed time measured by the measurement unit does not reach a predetermined time or when the change in facial expression is detected by the facial expression change detection unit. The in-vehicle videophone display control apparatus according to any one of claims 1 to 3, wherein the frame image already displayed on the display is continuously displayed on the display. A measuring unit that measures an elapsed time after the frame image is displayed on the display;
The facial expression change detection unit does not detect a change in the facial expression when the elapsed time measured by the measurement unit has not reached a predetermined time. A display control device for an in-vehicle videophone according to claim 1. 6. The notification sound generation unit according to claim 1, further comprising a notification sound generation unit configured to generate a notification sound when the frame image extracted by the image extraction unit is displayed on the display. In-vehicle videophone display control device. A display control method for controlling the display of an in-vehicle videophone,
A video playback process in which the video playback unit of the in-vehicle device performs background playback of the videophone video sent from the other party's phone,
A facial expression change detection step in which a facial expression change detection unit of the in-vehicle device detects a change that satisfies a certain condition in the facial expression of the communication partner from the video reproduced in the background in the video reproduction step;
When the image extraction unit of the in-vehicle device detects a change that satisfies the certain condition in the facial expression changing step, the image extraction extracts a frame image in which the change satisfying the certain condition is detected from the video Process,
A display control method for an in-vehicle videophone, comprising: a display control unit for displaying a frame image extracted in the image extraction step on a display as a still image.

Images