JP2019004221A - Control device and program - Google Patents

Abstract

To reduce power consumed by a source device by control at a sink device side when a user sleeps during execution of display mirroring.SOLUTION: A control device 3 for a sink device 2 in a mirroring system 100 that executes display mirroring from a source device 1 to a sink device 2 by wireless communication between the source device 1 and the sink device 2 includes a photographed image acquiring unit 11 that acquires a photographed image by a camera 9, a sleepiness determining unit 21 that determines whether or not a user U who appears in the photographed image sleeps by executing image recognition processing for the photographed image, and a sleep-on-time control unit 22 that executes control to terminate the display mirroring when it is determined that the user U has slept.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system for executing display mirroring from a source device to a sink device (hereinafter referred to as “mirroring system”), and more particularly to a control device and a program for a sink device.

  Conventionally, a technique for mirroring a screen displayed on a source device to a sink device, so-called display mirroring, has been developed. Specifically, for example, display mirroring by wireless communication based on Wi-Fi Direct (registered trademark; hereinafter referred to as “Wi-Fi Direct”) standard between a source device and a sink device, so-called Miracast (registered trademark, hereinafter). "Miracast" is described.) Has been developed.

  For example, a portable information terminal such as a smartphone is used as a source device for Miracast. As the Miracast sink device, for example, a television receiver, a PC (Personal Computer), an in-vehicle information device, or the like is used. Thereby, the user can view the moving image content being reproduced on the portable information terminal on a display larger than the display of the portable information terminal. Patent Document 1 discloses a mirroring system in which a portable information terminal is used as a source device and an in-vehicle information device is used as a sink device.

Japanese Patent Laying-Open No. 2015-97314

  During the execution of Miracast, the source device continuously executes a process for encoding a screen displayed on the source device and a process for transmitting the encoded data to the sink device. These processes consume a large amount of power in the source device. For this reason, when the user falls asleep during the execution of Miracast, there is a problem that a large amount of power is consumed in the source device. Further, when a portable information terminal such as a smartphone is used as the source device, there is a problem that the remaining amount of the battery is significantly reduced, and the portable information terminal cannot be used in an environment where the battery cannot be charged.

  The present invention has been made to solve the above-described problems. When a user sleeps during display mirroring, the power consumed by the source device is reduced by the control on the sink device side. The purpose is to do.

  The control device of the present invention is a control device for a sink device in a mirroring system that performs display mirroring from a source device to a sink device by wireless communication between the source device and the sink device, and acquires a photographed image by a camera An image acquisition unit, an image recognition process for the captured image, and a sleep onset determination unit that determines whether or not the user captured in the captured image has fallen asleep; display mirroring when it is determined that the user has fallen asleep And a sleep control unit that executes control for terminating the control.

  Alternatively, the control device of the present invention is a control device for a sink device in a mirroring system that performs display mirroring from the source device to the sink device by wireless communication between the source device and the sink device, and acquires a photographed image by the camera A captured image acquisition unit that performs image recognition processing on the captured image to determine whether or not the user imaged in the captured image has fallen asleep, and after determining that the user has fallen asleep, A sleep continuation determination unit that determines whether or not the user's sleep state has continued for a reference time or more, and a sleep continuation time that executes control to end display mirroring when it is determined that the user's sleep state has continued for a reference time or more And a control unit.

  According to the present invention, since it is configured as described above, when a user falls asleep during execution of display mirroring, the power consumed in the source device can be reduced by the control on the sink device side.

It is a block diagram which shows the system configuration | structure of the mirroring system which concerns on Embodiment 1 of this invention. FIG. 2A is an explanatory diagram showing an example of a display screen in the sink device according to Embodiment 1 of the present invention. FIG. 2B is an explanatory diagram showing an example of a display screen in the source device according to Embodiment 1 of the present invention. It is a block diagram which shows the principal part of the control apparatus which concerns on Embodiment 1 of this invention. FIG. 4A is a block diagram showing a hardware configuration of the control device according to Embodiment 1 of the present invention. FIG. 4B is a block diagram showing another hardware configuration of the control apparatus according to Embodiment 1 of the present invention. It is a flowchart which shows operation | movement of the control apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the system configuration | structure of the mirroring system which concerns on Embodiment 2 of this invention. It is a block diagram which shows the principal part of the control apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows operation | movement of the control apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the system configuration | structure of the mirroring system which concerns on Embodiment 3 of this invention. It is a block diagram which shows the principal part of the control apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the control apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the control apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the control apparatus which concerns on Embodiment 3 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a system configuration of a mirroring system according to Embodiment 1 of the present invention. FIG. 2A is an explanatory diagram showing an example of a display screen in the sink device according to Embodiment 1 of the present invention. FIG. 2B is an explanatory diagram showing an example of a display screen in the source device according to Embodiment 1 of the present invention. FIG. 3 is a block diagram showing a main part of the control device according to Embodiment 1 of the present invention. FIG. 4A is a block diagram showing a hardware configuration of the control device according to Embodiment 1 of the present invention. FIG. 4B is a block diagram showing another hardware configuration of the control apparatus according to Embodiment 1 of the present invention. The mirroring system 100 according to the first embodiment will be described with reference to FIGS.

  The mirroring system 100 executes display mirroring from the source device 1 to the sink device 2 by one-to-one type wireless communication between the source device 1 and the sink device 2. The source device 1 is configured by a portable information terminal such as a smartphone, for example. The sink device 2 includes a control device 3, a wireless communication device 4, a display device 5, an audio output device 6, a storage device 7, and an operation input device 8.

  Wireless communication between the source device 1 and the sink device 2 is based on the Wi-Fi direct standard, and display mirroring from the source device 1 to the sink device 2 is based on Miracast. This Miracast is controlled by cooperation between a control unit (not shown) of the source device 1 and a control device 3 of the sink device 2.

  In other words, the control device 3 has a function of starting a Miracast session by transmitting a Miracast session start request to the source device 1 by controlling the wireless communication device 4. The control device 3 also has a function of transmitting a Miracast session end request to the source device 1 by controlling the wireless communication device 4 to end the Miracast session.

  The source device 1 encodes a screen displayed on a display unit (not shown) of the source device 1 and transmits the encoded data to the sink device 2. In addition, when the source device 1 is reproducing the moving image content, the source device 1 encodes the sound of the moving image content and transmits the encoded data to the sink device 2.

  The wireless communication device 4 receives data transmitted by the source device 1. The wireless communication device 4 outputs the data to the control device 3. The wireless communication device 4 includes, for example, a Wi-Fi direct receiver, a transmitter, an antenna, and the like.

  The control device 3 has a function of decoding data input from the wireless communication device 4 and causing the display device 5 to display the same screen as that displayed on the display unit (not shown) of the source device 1. ing. The display device 5 includes a display such as a liquid crystal display or an organic EL (Electro Luminescence) display.

  Hereinafter, a screen displayed on the display unit (not shown) of the source device 1 or the display device 5 of the sink device 2 is referred to as a “display screen”. Here, FIG. 2A shows an example of a display screen in the sink device 2, and FIG. 2B shows an example of a display screen in the source device 1. As illustrated in FIG. 2, the source device 1 includes a smartphone, and a menu screen for the smartphone is displayed on the display unit of the source device 1. On the display device 5, the same screen as the menu screen is displayed.

  Further, the control device 3 has a function of decoding data input from the wireless communication device 4 and causing the audio output device 6 to output the sound of the moving image content being played back by the source device 1. The audio output device 6 is configured by, for example, a speaker or headphones.

  The control device 3 has a function of capturing a display screen on the display device 5 and storing the captured data in the storage device 7. The storage device 7 is configured by an auxiliary storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

  In addition, the control device 3 has a function of acquiring capture data stored in the storage device 7 in accordance with an operation input to the operation input device 8 and displaying the image indicated by the capture data on the display device 5. doing. The operation input device 8 is configured by, for example, a touch panel integrated with the display of the display device 5 or a remote controller.

  Hereinafter, main parts of the control device 3 will be described.

  The captured image acquisition unit 11 acquires an image captured by the camera 9 (hereinafter referred to as “captured image”). The camera 9 captures the user U of the sink device 2. Specifically, for example, the camera 9 is provided adjacent to the display of the display device 5 and captures a range including the face of the user U from the front with respect to the user U.

  The sleep onset determination unit 21 performs image recognition processing on the captured image acquired by the captured image acquisition unit 11. This image recognition process determines whether or not the user U has shifted from the awake state to the sleep state (hereinafter referred to as “sleeping”) when the user U is captured in the captured image.

  Specifically, for example, the sleep onset determination unit 21 determines whether the user U is face down or face up by image recognition processing. When the user U is raising his / her face, the sleep determination unit 21 determines whether the eyes of the user U are closed or open by image recognition processing. The sleep onset determination unit 21 determines that the user U has fallen asleep when the user U is face down or the eyes of the user U are closed. On the other hand, when the user U is looking up and the user U's eyes are open, the sleep onset determination unit 21 determines that the user U is not asleep.

  The sleep on-time controller 22 captures a display screen on the display device 5 and stores the captured data in the storage device 7 when the sleep onset determination unit 21 determines that the user U has fallen asleep. Further, the sleep control unit 22 stores the capture data (hereinafter referred to as “sleep capture data”) in the storage device 7 and then controls the wireless communication device 4 so as to make a request to end the Miracast session. This is sent to the source device 1 to end the Miracast session.

  The power saving control unit 23 is configured by the sleep determination unit 21 and the sleep control unit 22. The captured image acquisition unit 11 and the power saving control unit 23 constitute a main part of the control device 3.

  FIG. 4 shows a hardware configuration of a main part of the control device 3. As shown in FIG. 4A, the control device 3 is configured by a computer, and includes a processor 51 and a memory 52. The memory 52 stores a program for causing the computer to function as the captured image acquisition unit 11, sleep onset determination unit 21, and sleep onset control unit 22 illustrated in FIG. 3. When the processor 51 reads out and executes the program stored in the memory 52, the functions of the captured image acquisition unit 11, the sleep determination unit 21, and the sleep control unit 22 are realized.

  The processor 51 is composed of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). The memory 52 is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Memory). Yes.

  Alternatively, as shown in FIG. 4B, the functions of the captured image acquisition unit 11, sleep onset determination unit 21, and sleep onset control unit 22 may be realized by a dedicated processing circuit 53. The processing circuit 53 includes, for example, ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field-Programmable Gate Array), SoC (System LSI-L). Or a combination of these.

  It should be noted that the functions of each unit of the captured image acquisition unit 11, the sleep onset determination unit 21, and the sleep onset control unit 22 may be realized by the processing circuit 53, or the functions of the respective units may be collectively realized by the processing circuit 53. . In addition, some of the functions of the captured image acquisition unit 11, the sleep detection unit 21, and the sleep control unit 22 are realized by the processor 51 and the memory 52 illustrated in FIG. 4A, and the remaining functions are the processing circuit 53 illustrated in FIG. 4B. May be realized.

  Next, the operation of the control device 3 will be described with reference to the flowchart of FIG.

  In the initial state, the mirroring system 100 is executing display mirroring by Miracast. The control device 3 executes the following steps ST1 to ST4 in the background of display mirroring.

  First, in step ST1, the captured image acquisition unit 11 acquires an image captured by the camera 9.

  Next, in step ST2, the sleep determination unit 21 performs image recognition processing on the captured image acquired in step ST1, and determines whether or not the user U has fallen asleep. When the user U is not asleep (step ST2 “NO”), the process of the control device 3 returns to step ST1. On the other hand, when the user U falls asleep ("YES" in step ST2), the process of the control device 3 proceeds to step ST3.

  Next, in step ST <b> 3, the sleep control unit 22 captures the display screen on the display device 5 and stores the sleep capture data in the storage device 7.

  Next, in step ST <b> 4, the sleep control unit 22 controls the wireless communication device 4 to transmit a Miracast session end request to the source device 1 to end the Miracast session.

  When the Miracast session is ended, the encoding process and the transmission process in the source device 1 are also ended. For this reason, the power consumed in the source device 1 can be reduced.

  Moreover, the control apparatus 3 may perform the process which turns off the display of the display apparatus 5, or the process which turns off the power supply of the sink device 2 after step ST4. Thereby, the power consumed in the sink device 2 can be reduced.

  Further, after step ST4, when the user U shifts from the sleep state to the awake state (hereinafter referred to as “wake up”), the control device 3 stores the information in the storage device 7 in accordance with the operation input to the operation input device 8. The stored sleep capture data is acquired, and an image indicated by the sleep capture data is displayed on the display device 5. Thereby, the user U can confirm the display screen immediately before the end of the Miracast, that is, the display screen immediately after the user U falls asleep.

  By checking the display screen, when the source device 1 is reproducing the moving image content when the user U falls asleep, it becomes easy to search for a reproduction portion at the time of falling asleep after the user U wakes up. As a result, it becomes easy to resume the reproduction of the moving image content from the relevant part.

  Note that the control device 3 can freely set whether or not to execute the control by the power saving control unit 23 (that is, whether or not the processing of steps ST1 to ST4 shown in FIG. 5 is executed) by an operation input to the operation input device 8. It may be anything. Whether or not the control is necessary is set by, for example, operating the operation input device 8 on the setting screen while the setting screen of the sink device 2 is displayed on the display device 5. By setting the execution of the control to off, it is possible to prevent the control unnecessary for the user U from being executed.

  Further, wireless communication between the source device 1 and the sink device 2 is not limited to Wi-Fi direct, and display mirroring from the source device 1 to the sink device 2 is not limited to Miracast. The mirroring system 100 may execute display mirroring by one-to-one type wireless communication based on another standard different from Wi-Fi Direct, for example, the BLUETOOTH (registered trademark) standard or the ZIGBEE (registered trademark) standard.

  As described above, the control device 3 according to the first embodiment is for the sink device 2 in the mirroring system 100 that performs display mirroring from the source device 1 to the sink device 2 by wireless communication between the source device 1 and the sink device 2. The control device 3 determines whether or not the user U captured in the captured image has fallen asleep by executing a captured image acquisition unit 11 that acquires a captured image captured by the camera 9 and image recognition processing on the captured image. And a sleep onset control unit 22 that executes control to end display mirroring when it is determined that the user U has fallen asleep. Thereby, when the user U falls asleep during execution of display mirroring, the power consumed in the source device 1 can be reduced by the control on the sink device 2 side.

  The sleep control unit 22 captures the display screen on the sink device 2 and then ends the display mirroring. Thereby, the user U can confirm the display screen immediately before the end of the display mirroring, that is, the display screen immediately after the user U falls asleep.

Embodiment 2. FIG.
FIG. 6 is a block diagram showing a system configuration of the mirroring system according to Embodiment 2 of the present invention. FIG. 7 is a block diagram showing a main part of the control device according to the second embodiment of the present invention. With reference to FIG.6 and FIG.7, the mirroring system 100 of Embodiment 2 is demonstrated centering on the principal part of the control apparatus 3a.

  In FIG. 6, the same blocks as those shown in FIG. In FIG. 7, the same blocks as those shown in FIG. Since the hardware configuration of the control device 3a is the same as that described with reference to FIG. 4 in the first embodiment, illustration and description thereof are omitted.

  The sleep onset determination unit 31 performs image recognition processing on the captured image acquired by the captured image acquisition unit 11. This image recognition process determines whether or not the user U has fallen asleep. A specific example of the image recognition process by the sleep determination unit 31 is the same as the specific example of the image recognition process by the sleep determination unit 21 shown in FIG.

  The awakening determination unit 32 performs image recognition processing on the captured image acquired by the captured image acquisition unit 11 after the sleep determination unit 31 determines that the user U has fallen asleep. This image recognition process determines whether or not the user U has awakened.

  Specifically, for example, the awakening determination unit 32 determines whether the user U is face down or face up by image recognition processing. When the user U is raising his / her face, the awakening determination unit 32 determines whether the eyes of the user U are closed or open by an image recognition process. The awakening determination unit 32 determines that the user U has awakened when the user U is looking up and the user U's eyes are open. On the other hand, when the user U is face down or when the user U's eyes are closed, the awakening determination unit 32 determines that the user U is not awake.

  The sleep continuation determination unit 33 counts the duration of the sleep state after the sleep determination unit 31 determines that the user U has fallen asleep. When the sleep continuation determination unit 33 determines that the user U is not awake by the awakening determination unit 32, the sleep continuation determination unit 33 compares the duration with a predetermined time (hereinafter referred to as “reference time”), thereby It is determined whether or not the sleep state has continued for a reference time or more.

  The reference time is set to a value corresponding to the time from when a person goes to sleep until it shifts to a deeper sleep state (hereinafter referred to as “deep sleep state”) than immediately after going to sleep. Specifically, for example, the reference time is set to a value of about several minutes.

  When the sleep continuation determination unit 33 determines that the sleep state of the user U has continued for a reference time or longer, the sleep continuation time control unit 34 captures the display screen on the display device 5 and stores the capture data in the storage device 7. It is something to be made. In addition, the sleep duration control unit 34 stores the capture data (hereinafter referred to as “sleep sleep capture data”) in the storage device 7 and then controls the wireless communication device 4 to end the Miracast session. A request is transmitted to the source device 1 to end the Miracast session.

  The power saving control unit 35 includes the sleep determination unit 31, the awakening determination unit 32, the sleep continuation determination unit 33, and the sleep continuation control unit 34. The captured image acquisition unit 11 and the power saving control unit 35 constitute a main part of the control device 3a.

  Next, the operation of the control device 3a will be described with reference to the flowchart of FIG.

  In the initial state, the mirroring system 100 is executing display mirroring by Miracast. The control device 3a executes the following steps ST11 to ST19 in the background of display mirroring.

  First, in step ST11, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST12, the sleep determination unit 31 performs image recognition processing on the captured image acquired in step ST11, and determines whether or not the user U has fallen asleep. When the user U is not asleep (step ST12 “NO”), the process of the control device 3a returns to step ST11. On the other hand, when the user U falls asleep (“YES” in step ST12), the process of the control device 3a proceeds to step ST13.

  Subsequently, in step ST13, the sleep continuation determination unit 33 starts counting the duration of the sleep state.

  Next, in step ST <b> 14, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST15, the awakening determination unit 32 performs image recognition processing on the captured image acquired in step ST14, and determines whether or not the user U has awakened. When the user U is not awake (step ST15 “NO”), in step ST16, the sleep continuation determination unit 33 determines whether or not the sleep state of the user U has continued for a reference time or more. When the duration of the sleep state is less than the reference time (step ST16 “NO”), the process of the control device 3a returns to step ST14.

  If the user U wakes up after the user U falls asleep before the reference time has elapsed (step ST15 “YES”), the process of the control device 3a proceeds to step ST17. In step ST17, the sleep continuation determination unit 33 resets the count of the duration of the sleep state. Next, the process of the control device 3a returns to step ST11.

  On the other hand, after the user U falls asleep, if the reference time has passed without the user U waking up (step ST16 “YES”), the process of the control device 3a proceeds to step ST18. In step ST <b> 18, the sleep continuation time control unit 34 captures the display screen on the display device 5 and stores the sleep continuation time capture data in the storage device 7.

  Next, in step ST <b> 19, the sleep duration control unit 34 controls the wireless communication device 4 to transmit a Miracast session end request to the source device 1 to end the Miracast session.

  When the Miracast session is ended, the encoding process and the transmission process in the source device 1 are also ended. For this reason, the power consumed in the source device 1 can be reduced.

  Moreover, the control apparatus 3a may perform the process which turns off the display of the display apparatus 5, or the process which turns off the power supply of the sink device 2 after step ST19. Thereby, the power consumed in the sink device 2 can be reduced.

  Moreover, when the user U wakes up after step ST19, the control device 3a acquires the sleep continuation capture data stored in the storage device 7 in accordance with the operation input to the operation input device 8, and sleeps. The image indicated by the continuous capture data is displayed on the display device 5. Thereby, the user U can confirm the display screen immediately before the end of the Miracast, that is, the display screen immediately after the user U shifts to the deep sleep state.

  The control device 3a can freely set whether or not to execute the control by the power saving control unit 35 (that is, whether or not the processing of steps ST11 to ST19 shown in FIG. 8 is executed) by an operation input to the operation input device 8. It may be anything. By setting the execution of the control to off, it is possible to prevent the control unnecessary for the user U from being executed.

  Further, wireless communication between the source device 1 and the sink device 2 is not limited to Wi-Fi direct, and display mirroring from the source device 1 to the sink device 2 is not limited to Miracast. The mirroring system 100 may perform display mirroring by one-to-one type wireless communication based on another standard different from Wi-Fi Direct.

  As described above, the control device 3a according to the second embodiment is for the sink device 2 in the mirroring system 100 that performs display mirroring from the source device 1 to the sink device 2 through wireless communication between the source device 1 and the sink device 2. The control device 3a determines whether or not the user U captured in the captured image has fallen asleep by executing a captured image acquisition unit 11 that acquires a captured image from the camera 9 and image recognition processing on the captured image. Sleep determination unit 31 to perform, sleep determination unit 33 to determine whether or not the sleep state of user U has continued for a reference time or more after determination that user U has fallen asleep, and sleep state of user U is the reference time When it determines with having continued above, the sleep continuation time control part 34 which performs control which complete | finishes display mirroring is provided. Thereby, when the user U falls asleep during execution of display mirroring, the power consumed in the source device 1 can be reduced by the control on the sink device 2 side.

  Further, the sleep continuation time control unit 34 captures the display screen in the sink device 2 and then ends the display mirroring. Thereby, the user U can confirm the display screen immediately before display mirroring is completed, that is, the display screen immediately after the user U shifts to the deep sleep state.

Embodiment 3 FIG.
FIG. 9 is a block diagram showing a system configuration of the mirroring system according to the third embodiment of the present invention. FIG. 10 is a block diagram showing a main part of a control device according to Embodiment 3 of the present invention. A mirroring system 100 according to the third embodiment will be described with reference to FIGS. 9 and 10.

  In FIG. 9, the same blocks as those shown in FIG. In FIG. 10, the same blocks as those shown in FIG. Since the hardware configuration of the control device 3b is the same as that described with reference to FIG. 4 in the first embodiment, illustration and description thereof are omitted.

  In the third embodiment, the source device 1 and the sink device 2 correspond to UIBC (User Interface Back Channel) which is an optional function of Miracast.

  That is, when an operation on the display screen of the display device 5 is input to the operation input device 8, the control device 3 b regards the operation as being for the display screen of the source device 1 and controls the wireless communication device 4. Thus, it has a function of transmitting an event corresponding to the operation to the source device 1. A control unit (not shown) of the source device 1 has a function of executing various controls in the source device 1 assuming that an operation corresponding to the event is input to a GUI (Graphical User Interface) of the source device 1. doing. Thus, the user U can operate the source device 1 using the operation input device 8.

  In addition, the control device 3b has a function of capturing a display screen on the display device 5 at a predetermined time interval during accumulation of Miracast and storing the captured data in the storage device 7 for storage. Hereinafter, the capture data stored in the storage device 7 by the function is referred to as “accumulated capture data”. The accumulated capture data is used for processing by a moving image presence / absence determination unit 12, a first GUI detection unit 13, and a second GUI detection unit 14 to be described later.

  Hereinafter, the main part of the control device 3b will be described.

  The moving image presence / absence determination unit 12 acquires accumulated capture data stored in the storage device 7. The moving image presence / absence determining unit 12 determines whether or not moving image content is included in the display screen of the source device 1 by executing an image recognition process on the image indicated by the stored capture data, and also displays the display screen of the source device 1. If the video content is included, the source device 1 determines whether or not the video content is being reproduced.

  For example, when the media player is activated in the source device 1 and the media player plays the moving image content using the entire display screen, the moving image content is detected by the image recognition process. In addition, for example, when a web browser is activated on the source device 1 and the web browser accesses the video sharing service and reproduces video content using a part of the display screen, The moving image content is detected by the recognition process. In such a case, the moving image presence / absence determining unit 12 determines that moving image content is included in the display screen of the source device 1 and that the source device 1 is reproducing the moving image content.

  Further, when the moving image content as described above is detected by the image recognition processing, but the moving image content is still at the time of detection, the moving image presence / absence determination unit 12 includes the moving image content on the display screen of the source device 1. And the source device 1 determines that the moving image content is not being reproduced.

  On the other hand, for example, when the source device 1 displays the menu screen using the entire display screen, the moving image content is not detected by the image recognition process. For example, when the photo viewer is activated in the source device 1 and the photo viewer displays a still image such as a photograph using the entire display screen, the moving image content is not detected by the image recognition processing. In such a case, the moving image presence / absence determining unit 12 determines that moving image content is not included in the display screen of the source device 1.

  The first GUI detection unit 13 acquires accumulated capture data stored in the storage device 7. The first GUI detection unit 13 performs a GUI (hereinafter referred to as “first GUI”) corresponding to an operation for temporarily stopping the reproduction of the moving image content in the source device 1 by executing an image recognition process on the image indicated by the stored capture data. It is to detect.

  For example, when a media player or a web browser is playing back video content in the source device 1, playback of video content is included in at least some of the past timing images indicated by the stored capture data. There is a high probability that a software button for pausing is included. The first GUI detection unit 13 detects the position of the software button on the display screen of the source device 1 by image recognition processing.

  The second GUI detection unit 14 acquires accumulated capture data stored in the storage device 7. The second GUI detection unit 14 detects a GUI (hereinafter referred to as “second GUI”) corresponding to an operation for resuming the reproduction of the moving image content in the source device 1 by executing an image recognition process on the image indicated by the stored capture data. To do.

  For example, when a media player or a web browser is playing a video content in the source device 1, the image at at least some of the past timing images indicated by the stored capture data is paused. There is a high probability that a software button for resuming the playback of video content is included. The second GUI detection unit 14 detects the position of the software button on the display screen of the source device 1 by image recognition processing.

  The sleep onset determination unit 41 performs image recognition processing on the captured image acquired by the captured image acquisition unit 11. This image recognition process determines whether or not the user U has fallen asleep. A specific example of the image recognition process performed by the sleep determination unit 41 is the same as the specific example of the image recognition process performed by the sleep determination unit 21 illustrated in FIG. 1 or the sleep determination unit 31 illustrated in FIG.

  The awakening determination unit 42 performs image recognition processing on the captured image acquired by the captured image acquisition unit 11 after the sleep determination unit 41 determines that the user U has fallen asleep. This image recognition process determines whether or not the user U has awakened. Since the specific example of the image recognition process by the awakening determination part 42 is the same as the specific example of the image recognition process by the awakening determination part 32 shown in FIG. 7, description is abbreviate | omitted.

  The sleep continuation determination unit 43 counts the duration of the sleep state after the sleep determination unit 41 determines that the user U has fallen asleep. When the sleep continuation determination unit 43 determines that the user U is not awake by the awakening determination unit 42, whether or not the sleep state of the user U has continued for a reference time or more by comparing the duration with a reference time. This is a judgment. Since the specific example of the reference time in the sleep continuation determination unit 43 is the same as the specific example of the reference time in the sleep continuation determination unit 33 shown in FIG.

  When the moving image presence / absence determining unit 12 determines that the moving image content is included in the display screen of the source device 1, the sleep onset control unit 44 determines that the user U has fallen asleep when the sleep onset determining unit 41 determines that the user U has fallen asleep. Is used to temporarily stop the reproduction of the moving image content in the source device 1. That is, the sleep control unit 44 controls the wireless communication device 4 to transmit an event corresponding to the operation of the first GUI to the source device 1. Thereby, the reproduction of the moving image content in the source device 1 is paused.

  When the awakening control unit 45 determines that the moving image content is included in the display screen of the source device 1 by the moving image presence / absence determining unit 12, the UIBC determines that the user U has been awakened by the awakening determination unit 42. Is used to restart the reproduction of the moving image content in the source device 1. That is, the awakening control unit 45 controls the wireless communication device 4 to transmit an event corresponding to the operation of the second GUI to the source device 1. Thereby, the reproduction of the moving image content in the source device 1 is resumed.

  When the sleep continuation determination unit 43 determines that the sleep state of the user U has continued for a reference time or longer, the sleep continuation time control unit 46 captures the display screen on the display device 5 and stores the capture data in the storage device 7. It is something to be made. In addition, the sleep duration control unit 46 stores the capture data (hereinafter referred to as “sleep-continuation capture data”) in the storage device 7 and then controls the wireless communication device 4 to end the Miracast session. A request is transmitted to the source device 1 to end the Miracast session.

  The power saving control unit 47 is configured by the sleep onset determination unit 41, the awakening determination unit 42, the sleep continuation determination unit 43, the sleep onset control unit 44, the awakening time control unit 45, and the sleep continuation time control unit 46. The captured image acquisition unit 11, the moving image presence / absence determination unit 12, the first GUI detection unit 13, the second GUI detection unit 14, and the power saving control unit 47 constitute the main part of the control device 3 b.

  Next, the operation of the control device 3b will be described with reference to the flowchart of FIG.

  In the initial state, the mirroring system 100 is executing display mirroring by Miracast. The storage device 7 stores accumulated capture data. The control device 3b executes the following steps ST21 to ST22, ST31 to ST43, and ST51 to ST59 in the background of display mirroring.

  First, the moving image presence / absence determination unit 12 acquires accumulated capture data stored in the storage device 7. The moving image presence / absence determining unit 12 determines whether or not moving image content is included in the display screen of the source device 1 by executing an image recognition process on the image indicated by the stored capture data (step ST21). When the moving image content is included in one display screen, it is determined whether or not the source device 1 is reproducing the moving image content (step ST22).

  When the moving image content is included in the display screen of the source device 1 and the source device 1 is reproducing the moving image content (“YES” in step ST21 and “YES” in step ST22), the processing of the control device 3b is performed in step Proceed to ST31.

  Next, in step ST31, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST32, the sleep determination unit 41 performs image recognition processing on the captured image acquired in step ST31, and determines whether or not the user U has fallen asleep. When the user U is not asleep (step ST32 “NO”), the process of the control device 3b returns to step ST21. On the other hand, when the user U falls asleep ("YES" in step ST32), the process of the control device 3b proceeds to step ST33.

  Next, in step ST33, the sleep continuation determination unit 43 starts counting the duration of the sleep state.

  Next, in step ST <b> 34, the first GUI detection unit 13 acquires accumulated capture data stored in the storage device 7. The first GUI detection unit 13 detects the first GUI in the source device 1 by executing an image recognition process on the image indicated by the stored capture data.

  Next, in step ST <b> 35, the sleep control unit 44 temporarily stops the reproduction of the moving image content in the source device 1 using the UIBC. That is, the sleep control unit 44 controls the wireless communication device 4 to transmit an event corresponding to the operation of the first GUI detected in step ST34 to the source device 1.

  Through the process of step ST35, the reproduction of the moving image content in the source device 1 is paused. As a result, the power used for the reproduction processing of the moving image content becomes unnecessary, so that the power consumed by the source device 1 can be reduced. However, since the Miracast session is continued, the encoding process and the transmission process are continuously executed by the source device 1.

  Next, in step ST36, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST37, the awakening determination unit 42 performs image recognition processing on the captured image acquired in step ST36, and determines whether or not the user U has been awakened. When the user U is not awake (step ST37 “NO”), in step ST38, the sleep continuation determination unit 43 determines whether the sleep state of the user U has continued for a reference time or more. When the duration of the sleep state is less than the reference time (step ST38 “NO”), the process of the control device 3b returns to step ST36.

  If the user U wakes up after the user U falls asleep before the reference time has elapsed (step ST37 “YES”), the process of the control device 3b proceeds to step ST39. In step ST39, the sleep continuation determination unit 43 resets the count of the duration time of the sleep state.

  Next, in step ST <b> 40, the second GUI detection unit 14 acquires accumulated capture data stored in the storage device 7. The second GUI detection unit 14 detects the second GUI in the source device 1 by executing an image recognition process on the image indicated by the stored capture data.

  Next, in step ST41, the awakening control unit 45 restarts the reproduction of the moving image content in the source device 1 using UIBC. That is, the awakening control unit 45 controls the wireless communication device 4 to transmit an event corresponding to the operation of the second GUI detected in step ST40 to the source device 1.

  Through the process in step ST41, the reproduction of the moving image content in the source device 1 is resumed. As a result, the reproduction of the moving image content can be automatically restarted from the place where it was temporarily stopped in step ST35. Following step ST41, the process of the control device 3b returns to step ST21.

  On the other hand, after the user U falls asleep, if the reference time has passed without the user U waking up (step ST38 “YES”), the process of the control device 3b proceeds to step ST42. In step ST <b> 42, the sleep continuation time control unit 46 captures the display screen on the display device 5 and stores the sleep continuation time capture data in the storage device 7.

  Next, in step ST43, the sleep continuation time control unit 46 controls the wireless communication device 4 to transmit a Miracast session end request to the source device 1 to end the Miracast session.

  When the Miracast session is ended, the encoding process and the transmission process in the source device 1 are also ended. For this reason, the power consumed in the source device 1 can be further reduced.

  Moreover, the control apparatus 3b may perform the process which turns off the display of the display apparatus 5, or the process which turns off the power supply of the sink device 2 after step ST43. Thereby, the power consumed in the sink device 2 can be reduced.

  Moreover, when the user U wakes up after step ST43, the control device 3b acquires the sleep continuation capture data stored in the storage device 7 according to the operation input to the operation input device 8, and sleeps. The image indicated by the continuous capture data is displayed on the display device 5. Thereby, the user U can confirm the display screen immediately before the end of the Miracast, that is, the display screen immediately after the user U shifts to the deep sleep state.

  Here, since the reproduction of the moving image content is paused when the user U falls asleep (step ST35), the display screen when the user U enters the deep sleep state includes the moving image content in the same state as when the user U fell asleep. By confirming the display screen, it becomes easy to search for a reproduction part at the time of falling asleep after the user U wakes up. As a result, it becomes easy to resume the reproduction of the moving image content from the relevant part.

  On the other hand, when the video content is not included in the display screen of the source device 1 (step ST21 “NO”), or the video content is included in the display screen of the source device 1 and the source device 1 is the video content. Is not being reproduced (step ST21 “YES” and step ST22 “NO”), the process of the control device 3b proceeds to step ST51.

  Next, in step ST51, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST52, the sleep determination unit 41 performs image recognition processing on the captured image acquired in step ST51, and determines whether or not the user U has fallen asleep. When the user U is not asleep (step ST52 “NO”), the process of the control device 3b returns to step ST21. On the other hand, when the user U falls asleep ("YES" in step ST52), the process of the control device 3b proceeds to step ST53.

  Next, in step ST <b> 53, the sleep continuation determination unit 43 starts counting the duration of the sleep state.

  Next, in step ST54, the captured image acquisition unit 11 acquires a captured image by the camera 9.

  Next, in step ST55, the awakening determination unit 42 performs image recognition processing on the captured image acquired in step ST54, and determines whether or not the user U has been awakened. When the user U is not awake (step ST55 “NO”), in step ST56, the sleep continuation determination unit 43 determines whether or not the sleep state of the user U has continued for a reference time or more. When the duration of the sleep state is less than the reference time (step ST56 “NO”), the process of the control device 3b returns to step ST54.

  If the user U wakes up after the user U falls asleep before the reference time has elapsed (step ST55 “YES”), the process of the control device 3b proceeds to step ST57. In step ST57, the sleep continuation determination unit 43 resets the count of the duration time of the sleep state. Next, the process of the control device 3b returns to step ST21.

  On the other hand, after the user U falls asleep, when the reference time has passed without the user U waking up (step ST56 “YES”), the process of the control device 3b proceeds to step ST58. In step ST58, the sleep continuation time control unit 46 captures the display screen on the display device 5 and stores the sleep continuation time capture data in the storage device 7.

  Next, in step ST <b> 59, the sleep continuation time control unit 46 controls the wireless communication device 4 to transmit a miracast session end request to the source device 1 to end the miracast session.

  When the Miracast session is ended, the encoding process and the transmission process in the source device 1 are also ended. For this reason, the power consumed in the source device 1 can be reduced.

  Moreover, the control apparatus 3b may perform the process which turns off the display of the display apparatus 5, or the process which turns off the power supply of the sink device 2 after step ST59. Thereby, the power consumed in the sink device 2 can be reduced.

  Moreover, when the user U wakes up after step ST59, the control device 3b acquires the sleep continuation capture data stored in the storage device 7 according to the operation input to the operation input device 8, and sleeps. The image indicated by the continuous capture data is displayed on the display device 5. Thereby, the user U can confirm the display screen immediately before the end of the Miracast, that is, the display screen immediately after the user U shifts to the deep sleep state.

  Note that the control device 3b determines whether or not the control by the power saving control unit 47 is necessary by the operation input to the operation input device 8 (that is, the processing of steps ST21 to ST22, ST31 to ST43, ST51 to ST59 shown in FIG. 11). It may be possible to set whether or not execution is necessary. By setting the execution of the control to off, it is possible to prevent the control unnecessary for the user U from being executed.

  Further, wireless communication between the source device 1 and the sink device 2 is not limited to Wi-Fi direct, and display mirroring from the source device 1 to the sink device 2 is not limited to Miracast. The mirroring system 100 may perform display mirroring by one-to-one type wireless communication based on another standard different from Wi-Fi Direct.

  As described above, the control device 3b according to the third embodiment is for the sink device 2 in the mirroring system 100 that performs display mirroring from the source device 1 to the sink device 2 through wireless communication between the source device 1 and the sink device 2. The control device 3b determines whether or not the user U captured in the captured image has fallen asleep by executing a captured image acquisition unit 11 that acquires a captured image from the camera 9 and image recognition processing on the captured image. A sleep continuation determination unit 41, a sleep continuation determination unit 43 that determines whether or not the sleep state of the user U has continued for a reference time or more after it is determined that the user U has fallen asleep, and the sleep state of the user U is a reference time When it determines with having continued above, it has the sleep continuation time control part 46 which performs control which complete | finishes display mirroring. Thereby, when the user U falls asleep during execution of display mirroring, the power consumed in the source device 1 can be reduced by the control on the sink device 2 side.

  In addition, the sleep continuation time control unit 46 ends the display mirroring after capturing the display screen in the sink device 2. Thereby, the user U can confirm the display screen immediately before display mirroring is completed, that is, the display screen immediately after the user U shifts to the deep sleep state.

  In addition, the control device 3b uses the capture data of the display screen in the sink device 2 to determine whether the display screen of the source device 1 includes moving image content, and the sink device 2 Using the capture data of the display screen, the first GUI detection unit 13 that detects the first GUI corresponding to the operation of pausing the reproduction of the moving image content in the source device 1, and the moving image content is included in the display screen of the source device 1. When it is determined that the user U has fallen asleep, the sleep control unit 44 includes a sleep-time control unit 44 that executes control for temporarily stopping the reproduction of the moving image content in the source device 1 using the first GUI. Thereby, the power consumed in the source device 1 can be further reduced.

  In addition, the control device 3b uses the display screen capture data in the sink device 2 to detect a second GUI corresponding to an operation for resuming the reproduction of the moving image content in the source device 1, and the user U After it is determined that the user has fallen asleep, the image recognition process is performed on the captured image to determine whether or not the user U has been awakened, and the display screen of the source device 1 includes moving image content. When it is determined that the user U has awakened, the awakening control unit 45 executes control for resuming the reproduction of the moving image content in the source device 1 using the second GUI. Thereby, after the user U falls asleep, when the user U wakes up without transitioning to the deep sleep state, the reproduction of the moving image content can be automatically resumed from the place where it was paused.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  1 Source device, 2 Sink device, 3, 3a, 3b Control device, 4 Wireless communication device, 5 Display device, 6 Audio output device, 7 Storage device, 8 Operation input device, 9 Camera, 11 Captured image acquisition unit, 12 Movie Presence / absence determination unit, 13 1st GUI detection unit, 14 2nd GUI detection unit, 21 sleep onset determination unit, 22 sleep onset control unit, 23 power saving control unit, 31 sleep onset determination unit, 32 awakening determination unit, 33 sleep continuation determination unit, 34 Sleep continuation control unit, 35 Power saving control unit, 41 Sleep onset determination unit, 42 Awakening determination unit, 43 Sleep continuation determination unit, 44 Sleep onset control unit, 45 Awakening control unit, 46 Sleep continuation control unit, 47 Power saving Control unit, 51 processor, 52 memory, 53 processing circuit, 100 mirroring system.

Claims (10)

A control device for the sink device in a mirroring system that performs display mirroring from the source device to the sink device by wireless communication between the source device and the sink device,
A captured image acquisition unit for acquiring a captured image by the camera;
A sleep determination unit that determines whether or not the user imaged in the captured image has fallen asleep by executing image recognition processing on the captured image;
When it is determined that the user has fallen asleep, a sleep control unit that executes control to end the display mirroring;
A control device comprising:   The control device according to claim 1, wherein the sleep control unit ends the display mirroring after capturing a display screen on the sink device.   The control device according to claim 1, wherein whether or not the control by the sleep control unit is necessary can be set by an operation input to the operation input device. A control device for the sink device in a mirroring system that performs display mirroring from the source device to the sink device by wireless communication between the source device and the sink device,
A captured image acquisition unit for acquiring a captured image by the camera;
A sleep determination unit that determines whether or not the user imaged in the captured image has fallen asleep by executing image recognition processing on the captured image;
After determining that the user has fallen asleep, the sleep continuation determining unit that determines whether the user's sleep state has continued for a reference time or longer,
When it is determined that the sleep state of the user has continued for a reference time or longer, a sleep duration control unit that executes control to end the display mirroring;
A control device comprising:   The control device according to claim 4, wherein the sleep duration control unit ends the display mirroring after capturing a display screen in the sink device.   The control device according to claim 4 or 5, wherein whether or not the execution of control by the sleep duration control unit can be set by an operation input to the operation input device. A moving image presence / absence determining unit that determines whether moving image content is included in the display screen of the source device using capture data of the display screen in the sink device;
A first GUI detection unit for detecting a first GUI corresponding to an operation for temporarily stopping reproduction of moving image content in the source device, using capture data of a display screen in the sink device;
When it is determined that moving image content is included in the display screen of the source device, when the user is determined to fall asleep, control is performed to pause the reproduction of the moving image content on the source device using the first GUI. A sleep control unit,
The control device according to any one of claims 4 to 6, further comprising: A second GUI detection unit that detects a second GUI corresponding to an operation of resuming reproduction of moving image content in the source device, using capture data of a display screen in the sink device;
An awakening determination unit that determines whether or not the user has awakened by executing an image recognition process on the captured image after it is determined that the user has fallen asleep;
When it is determined that the display screen of the source device includes moving image content, the second GUI executes control to resume playback of the moving image content on the source device when it is determined that the user has awakened An awakening control unit;
The control device according to claim 7, further comprising: A program for the sink device in a mirroring system that performs display mirroring from the source device to the sink device by wireless communication between the source device and the sink device,
A computer for the sink device;
A captured image acquisition unit for acquiring a captured image by the camera;
A sleep determination unit that determines whether or not the user imaged in the captured image has fallen asleep by executing image recognition processing on the captured image;
When it is determined that the user has fallen asleep, a sleep control unit that executes control to end the display mirroring;
Program to function as. A program for the sink device in a mirroring system that performs display mirroring from the source device to the sink device by wireless communication between the source device and the sink device,
A computer for the sink device;
A captured image acquisition unit for acquiring a captured image by the camera;
A sleep determination unit that determines whether or not the user imaged in the captured image has fallen asleep by executing image recognition processing on the captured image;
After determining that the user has fallen asleep, the sleep continuation determining unit that determines whether the user's sleep state has continued for a reference time or longer,
When it is determined that the sleep state of the user has continued for a reference time or longer, a sleep duration control unit that executes control to end the display mirroring;
Program to function as.

Images