FR3058607A1 - VIDEO CAPTURE DEVICE WITH CAPTURE OF USER EYE MOVEMENT - Google Patents

Abstract

The present invention relates to a device (1) for capturing video images comprising: - a front camera (10) for capturing images of a front scene initially comprising an object of interest for the user of the device, - a camera rear view (11) configured to capture images of the head of a user of the device, - a processing circuit (12) of the images of the rear camera to estimate the movement of the user's eyes and estimate the movement of the object of interest in the frontal scene captured from the estimated movement of the eyes of the user, and - a decision circuit (13) for determining, from the estimated displacement of the object of interest, whether the object interest gets out of the captured frontal scene and, if so, trigger an action.

Description

058 607

60685 ® FRENCH REPUBLIC

NATIONAL INSTITUTE OF INDUSTRIAL PROPERTY © Publication number:

(to be used only for reproduction orders)

©) National registration number

COURBEVOIE © Int Cl ⁸ : H 04 N 5/225 (2017.01)

PATENT INVENTION APPLICATION

A1

©) Date of filing: 04.11.16. © Applicant (s): THOMSON LICENSING Company by (© Priority: simplified actions - FR. @ Inventor (s): DEMARTY CLAIRE-HELENE, CHE- VALLIER LOUIS and KULKARNI PRAVEEN ANIL. (43) Date of public availability of the request: 11.05.18 Bulletin 18/19. ©) List of documents cited in the report preliminary research: Refer to end of present booklet (© References to other national documents ® Holder (s): THOMSON LICENSING Company by related: simplified actions. ©) Extension request (s): © Agent (s): TECHNICOLOR.

104 / VIDEO CAPTURE DEVICE WITH CAPTURE OF THE USER'S EYE MOVEMENT.

FR 3,058,607 - A1 \ b /) The present invention relates to a device (1) for capturing video images comprising:

- a front camera (10) for capturing images of a frontal scene initially comprising an object of interest for the user of the device,

- a rear camera (11) configured to capture images of the head of a user of the device,

a processing circuit (12) for images from the rear camera to estimate the movement of the eyes of the user and to estimate the displacement of the object of interest in the frontal scene captured from the estimated movement of the eyes of the user, and

- a decision circuit (13) for determining, from the estimated displacement of the object of interest, whether the object of interest leaves the captured frontal scene and, if so, triggering an action.

PF160142 - FR-NP

VIDEO CAPTURE DEVICE WITH MOTION CAPTURE

EYES OF THE USER Technical area The present invention is reports at field of the video clip capture using of devices of capture provided with a front camera and of a camera

back.

State of the art

Mobile phones and tablets are nowadays widely used for filming any type of event, such as family meals, weddings, shows, conferences. During the capture, the user generally looks at the screen of his phone or tablet to make sure that the captured scene is well framed and that it shows the desired people and / or objects. The user is therefore forced to watch the scene through the telephone screen. This can result in a feeling of frustration at not being able to look directly at the scene.

There is therefore a need to help the user to be able to film a scene without having to constantly monitor the screen of the capture device. This need is all the more difficult to satisfy when the scene includes moving elements and that the capture must follow these elements.

PF160142 - FR-NP

FIG. 1 schematically illustrates the case of a father P filming with his mobile phone T the dance performance of his daughter F. This father particularly wishes to film his daughter. As the figure shows, the father is then forced to watch the show through the screen of his phone to make sure that he is filming his daughter. He redirects his phone as his daughter moves.

Summary of the invention

The invention proposes a solution making it possible to avoid the user having to permanently watch the scene filmed through the screen of his video capture device in order to reorient the video capture device if necessary.

According to the invention, it is considered that the user wishes to film what he is watching. Furthermore, video capture devices, such as mobile phones or tablets, are now, and as a standard, equipped with a front camera for filming a front scene and a rear camera for capturing the user, in particular his face, especially his eyes.

According to the invention, it is therefore proposed to use the rear camera of the video capture device to follow the gaze of the user, to estimate the displacement of the object, called object of interest, which the user follows with his gaze. and deduce therefrom if the object of interest is still in the scene captured by the front camera of the device.

PF160142 - FR-NP

More particularly, the invention relates to a device for capturing video images comprising:

a camera, known as a front camera, for capturing images, known as front images, of a frontal scene initially comprising an object of interest for the user of the device, said object of interest being potentially mobile and capable of leaving the captured front scene,

a camera, known as the rear camera, for capturing images, called rear images, of a rear scene, said rear camera being configured to capture images of the eyes of a user of the device, forming the rear images,

a rear image processing circuit for estimating the movement of the user's eyes and estimating the displacement of the object of interest in the frontal scene captured from the estimated movement of the user's eyes, and

- a decision circuit for determining, from the estimated displacement of the object of interest, whether the object of interest leaves the captured frontal scene and, if so, triggering an action.

According to a particular embodiment, the action triggered is the emission of a sound signal and / or the emission of a vibration.

According to another particular embodiment, the front camera is mounted mobile in said device and the action consists in moving the front camera to modify the axis of shooting.

PF160142 - FR-NP

According to another particular embodiment, the device comprises an accelerometer for estimating the movement of the device and the estimated movement of the device is supplied to the processing circuit. The processing circuit then estimates the displacement of

the object of interest from of the difference Between the estimated movement eyes and the movement valued of device.

The invention also relates to a method of capturing video images comprising the following steps:

- Capture with a so-called front camera, images, called front images, of a frontal scene initially comprising an object of interest for the user of the device, said object of interest being potentially mobile and capable of leaving the scene frontal captured,

- capture, with a camera called rear camera, images, called rear images, of a rear scene, and characterized in that the rear camera is configured to capture images of the eyes of a user, forming the rear images, and in that the method further comprises the following steps:

- process the rear images to estimate the movement of the user's eyes and estimate the displacement of the object of interest in the frontal scene captured from the estimated movement of the user's eyes, and 'object to be determined, interest, from the estimated displacement of if the object of interest leaves the

PF160142 - FR-NP frontal scene captured and, if so, trigger an action.

As mentioned previously, the action triggered is the emission of a sound signal and / or the emission of a vibration or, if the front camera is mobile, the action consists in moving the front camera to modify its axis. of shooting.

According to another particular embodiment, the method further comprises a step of estimating the movement of the front camera and the rear camera. The movement of the object of interest is then estimated from the difference between the estimated movement of the eyes and the estimated movement of the device.

Other advantages may come from the profession of reading illustrated by the illustrative figures.

still appear to humans from the examples below, appended, given as

Brief description of the figures - Figure 1, already described, is a schematic view illustrating the video capture of a scene with a device according to the prior art;

- Figure 2 is a schematic view illustrating the video capture of the same scene with a device according to the invention;

- Figure 3 is a block diagram of a video capture device embodiment of the invention, and according to a mode of

PF160142 - FR-NP - Figure 4 is a flow diagram of the steps of the method according to the invention.

Detailed description of the invention

FIG. 2 illustrates a video capture operation with a video capture device according to the invention. According to the invention, the front camera of the device is used to film the desired scene while the rear camera is used to capture images of the eyes, at least, of the user. An eye movement detection algorithm is then applied to the images captured by the rear camera to estimate the movement of the user's eyes and deduce the displacement of the object of interest in the scene filmed by the front camera. The user can then position the video capture device outside of his field of vision as illustrated in FIG. 2. The position of the object of interest in the scene filmed by

the camera East permanently estimated and the user East warned when the object of interest comes out of the scene filmed by the front camera. The figure 3 represents a diagram functional of a device of video capture 1 compliant to the invention.

The video capture device 1 comprises a front camera 10 for capturing images of a front scene, for example a spectacle as in FIGS. 1 and 2, comprising an object of interest and a rear camera 11 for capturing images of a rear scene. The rear camera 11 is more particularly configured to capture images of the eyes of the user of the

PF160142 - FR-NP device. It can capture a wider scene than the user's eyes, for example his head.

The device 1 also includes an image processing circuit 12 for processing the images captured by the rear camera 11 and a decision circuit 13. The image processing circuit 12 implements an algorithm for estimating the movement of the eyes of the user and an algorithm for estimating the displacement of the object of interest from the estimated movement of the eyes. The decision circuit 13 then determines, from the estimated displacement of the object of interest, whether the object of interest leaves the frontal scene and, if so, triggers an action.

This action can be multiple. It can consist in warning the user that the object of interest is leaving the filmed frontal scene and that he must therefore modify the orientation or the position of the video capture device to keep the object of interest in the filmed scene. The decision circuit 13 sends for example a command to a vibrator 14 of the device so that the latter starts to vibrate. It can also trigger the emission of a sound signal by a loudspeaker 15. The vibrator 14 and the loudspeaker 15 are an integral part of the capture device. As a variant, the vibrator 14 and the loudspeaker 15 are elements separate from the capture device so as not to hinder the recording of the video; for example the vibrator 14 can be integrated into a watch or a garment and the loudspeaker 15 can be an earpiece. Thus vibration and / or sound will not be

PF160142 - FR-NP not recorded by the capture device. The action triggered by the decision circuit 13 corresponds to a signal representative of a vibration intended for the vibrator 14 or to a signal representative of a sound intended for the loudspeaker 15. As a variant, the action triggered by the decision circuit 13 comprises several signals intended for several effectors, for example vibrators or loudspeakers.

If the front camera 10 is a camera mounted mobile in the capture device, the action can also consist in moving the front camera to modify its shooting axis and thus preserve the object of interest in the filmed scene.

The estimation of the displacement of the object of interest in the filmed scene requires a calibration of the image processing circuit 12 and / or of the decision circuit 13,

especially for determine a parameter of correspondence ( or postman scale) Between ' the movement of eyes and the displacement from 1 ' object of interest in the scene captured by the camera before

.

According to the invention, it is assumed that the way in which the user holds the video capture device relative to his head is known by circuits 12 and 13.

The match parameter can then be determined as follows. The user fixes the four corners of a frame, real or virtual, in the real scene for a so-called reference adjustment of the zoom

PF160142 - FR-NP of the front camera so that the scene captured by the camera matches the frame mentioned above.

The rear camera captures the movement of the user's eyes. This eye movement is estimated in the rear images by an appropriate algorithm, such as that described in American application US 20150092983 A1. We can then deduce that, for this so-called reference zoom, the eye movement estimated in the rear images corresponds to a displacement of the object of interest on the periphery of the scene. More particularly, the displacement of the eyes between the upper left corner and the upper right corner corresponds to a displacement of the object of interest between the left edge and the right edge of the scene of the scene. Similarly, the movement of the eyes between the upper right corner and the lower right corner corresponds to a movement of the object of interest between the upper edge and the lower edge of the scene of the scene.

We can then deduce a correspondence between the movement of the eyes and the movement of the object of interest in the scene for the reference zoom.

If we initially consider an object of interest in the center of the filmed scene and if we detect a movement of the eyes equal to half the estimated movement for a displacement of the object of interest between the left edge and the right edge, we can deduce that the object of interest is about to leave the filmed scene.

In operation, the capture device of the invention is used in the following manner.

PF160142 - FR-NP

The user before. That of rear pictures.

launches the video capture eyes trigger by the camera automatically captures the user by the camera

According to a basic embodiment, the user places the object of interest at the center of the scene filmed by the front camera. This allows to know the initial position of it in the filmed scene without having to process the images before.

As a variant, the user can select the initial position of the object of interest by pointing it on a touch screen of the capture device displaying the images captured by the front camera.

Assuming that the user follows the object of interest, the movement of his eyes gives information on the movement of the object of interest. The image processing circuit then estimates the displacement of the object of interest from the estimated movement of the user's eyes and the zoom factor of the front camera. If the zoom factor used (Z) is different from the reference zoom (Zref), the displacement of the object of interest estimated from the estimated movement of the eyes must be multiplied by the ratio Z / Zref.

When the decision circuit determines, from the estimated displacement of the object of interest, that the latter has reached has reached an edge of the filmed scene, it triggers an alarm, for example the emission of a sound signal and / or a vibration. Advantageously, this alarm is triggered before the object reaches

PF160142- FR-NP one edge of the stage. It is triggered for example at a predetermined threshold distance before the object of interest reaches the edge. This threshold distance can vary during capture and depend in particular on the speed of movement of the object of interest.

the user to capture and to if necessary, for example in order to capture by

The reception of this alarm prompts to check the position of the repositioning device in relation to the scene. The user must reposition, angularly pivoting on his feet, to keep the position of the device in relation to his head.

As a variant, the user can be authorized to move the capture device during video capture. The displacement of the device is then measured, for example with an inertial measurement unit comprising for example one or more accelerometer (s) (such as the accelerometer shown in FIG. 3) and / or one or more gyroscope (s) and / or a compass and / or a level, embedded in the capture device. As a variant, the movement of the capture device can be detected by a positioning system external to the capture device, for example a network of infrared cameras. The processing circuit then estimates the displacement of the object of interest from the difference between the estimated movement of the eyes and the estimated movement of the device and as a function of the zoom factor applied to the capture device. Advantageously, this variant makes it possible to correct the

PF160142- FR-NP errors linked to involuntary movements of the capture device by the user.

The invention also relates to a video capture method with a capture device comprising a front camera and a rear camera. Figure 4 illustrates the steps in this process.

According to a first step E1, a frontal scene to be filmed with the front camera of the capture device is captured. The front scene includes an object of interest that the user wishes to appear in the images throughout the capture.

According to a step E2, images of the eyes, at least with the rear camera, of the user are captured.

According to a step E3, the images from the rear camera are processed to estimate the movement of the eyes of the user and to estimate the displacement of the object of interest in the frontal scene from the estimated movement of the eyes of the user.

According to a step E4, it is determined, from the estimated displacement of the object of interest, whether the object of interest leaves the captured frontal scene and, if so, an alarm is triggered to warn the user .

The invention is described in the foregoing by way of example. It is understood that the person skilled in the art is able to carry out different variant embodiments of the invention, by combining for example the different characteristics above taken alone or

PF160142- FR-NP in combination, without departing from the scope of the invention.

PF160142- FR-NP

Claims (8)

1) Device comprising: - a camera, (1) for capturing video images for said front camera (10), for capturing images, called front images, of a front scene initially comprising an object of interest for a user of the device, said object interest being potentially mobile and likely to leave the frontal scene captured, - a camera, called rear camera (11), for capturing images, called rear images, of a rear scene, and characterized in that the rear camera is configured to capture images of the eyes of the user of the device, forming rear images, and in that the device further comprises a rear image processing circuit (12) for estimating the movement of the user's eyes and estimating the displacement of the object of interest in the frontal scene captured from the estimated movement of the user's eyes, and - a decision circuit (13) for determining, from the estimated displacement of the object of interest, whether the object of interest leaves the captured frontal scene and, if so, triggering an action. 2) Device according to claim 1, wherein the action triggered is the emission of a sound signal and / or the emission of a vibration. PF160142- FR-NP 3) Device according to claim 1, wherein the front camera is mounted mobile in said device and in which the action consists in moving the front camera to modify the axis of shooting. 4) Device according to any one of the preceding claims comprising an accelerometer (16) for estimating the movement of the device, in which the estimated movement of the device is supplied to the decision circuit and in which the processing circuit estimates the displacement of the object of interest from the difference between the estimated movement of the eyes and the estimated movement of the device. 5) A method of capturing video images comprising the following steps; - capture (El), with a camera called front camera, images, called front images, of a frontal scene initially comprising an object of interest for a user of the device, said object of interest being potentially mobile and capable of coming out of the captured frontal scene, - capture (E2), with a camera called rear camera, images, called rear images, of a rear scene, and characterized in that the rear camera is configured to capture images of the user's eyes, forming the images rear, and in that the method further comprises the following steps: - process (E3) the rear images to estimate the movement of the user's eyes and estimate the PF160142 - FR-NP displacement of the object of interest in the frontal scene captured from the estimated movement of the user's eyes, and - determine (E4), from the estimated displacement of the object of interest, if the object of interest leaves the captured frontal scene and, if so, trigger an action. 6) Method according to claim 5, wherein the action triggered is the emission of a sound signal and / or the emission of a vibration. 7) Method according to claim 5, wherein the front camera is mounted mobile in said device and wherein the action consists in moving the front camera to modify the axis of shooting. 8) Method according to claim 5, further comprising a step of estimating the movement of the front camera and the rear camera and in which the displacement of the object of interest is estimated from the difference between the estimated movement of the eyes and the estimated movement of the device. 1/2 F