WO2024065775A1

WO2024065775A1 - A method for extending the display of a content on a first screen of a first device to a second screen of a second device, a device for enabling screen extension and a screen extension system

Info

Publication number: WO2024065775A1
Application number: PCT/CN2022/123513
Authority: WO
Inventors: Nan Ye; Zhihong Guo
Original assignee: Orange
Priority date: 2022-09-30
Filing date: 2022-09-30
Publication date: 2024-04-04
Also published as: WO2024069228A1

Abstract

The invention relates method for extending the display of a content on a first screen of a first device to a second screen of a second device, comprising: -detecting a movement of the second device; -determining a position of the second device relatively to the first device based on the detected movement; and -extending the display content on the first screen of the first device to the second screen of the second device based on the determined position. The invention improves the connection efficiency for screen extension on multiple devices.

Description

A METHOD FOR EXTENDING THE DISPLAY OF A CONTENT ON A FIRST SCREEN OF A FIRST DEVICE TO A SECOND SCREEN OF A SECOND DEVICE, A DEVICE FOR ENABLING SCREEN EXTENSION AND A SCREEN EXTENSION SYSTEM

FIELD OF INVENTION

The present invention relates toa field of image processing, in particular screen extension method, device and system.

BACKGROUND

With more and more screens of various devices being simultaneously used by a user, screen extension technology has been developed to extent the display of a content on a first screen of a first device (such as a host device) to at least one additional screen of other additional device (s) . With such a technology, the user can use multiple screens simultaneously for collaborative work, entertainment and other professional or individual purposes.

For example, in a particular scenario, when a userwants to use a larger display size, in addition to a first screen, e.g. a screen of a mobile device, to get a better experience, he can combine more than one screen of additional device (s) , e.g. a second screen of another mobile device, with the first screen into an integrated virtual screen for extending the display content of the first screen to the second screen.

To that end, a display extension technology is already known especially for PC/laptops, which allows to add additional devices/screens to a host device in order to extend the screen on the additional devices. In such a prior art technology, for instance, when one connects an additional display to the host computer through, for example, a USB cable, a first connected additional display is usually assigned by default a first predefined position, such as on the right side of the host computer for a right screen extension, while a second one is assigned by default a second predefined position, such as on the left side of the host computer for the left screen extension. In other words, in the art, the order of connection of the additional display to the host computer defines by default the position of the additional display with respect to the host computer.

Although the abovementioned solution is well developed and widely used to extent screen, such a solution has some drawbacks. Since the extension position of the additional display is assigned by default, when the actual position is different from the predefined position, the display content will not be correctly extended. For example, when the first additional display is positioned on the left side of a host computer and is assigned by default a right position relatively to the host computer, the right extension display content will be incorrectly extended to the first additional display. Similar, when the second additional display is positioned on the right side of a host computer and is assigned by default a left position relatively to the host computer, the left extension display content will be incorrectly extended to the second additional display. Therefore, the extended display content are mismatched on the incorrect extension screen. To solve this problem, in the art, one can manually adjust configuration on the host computer to correctly select the actual positions for the additional displays. Alternatively, one can manually change the physical positions or connection order of the first and second additional displays, so as to match the predefined positions.

Therefore, in the art, to overcome the problem of misconfigured screen extension due to additional displays for screen extension not being correctly positioned relatively to the host computer according to the predefined position arrangements as mentioned above, the use needs to pay more attention beforehand to the predefined positions rules of the host computer and correctly connect the additional displays in right order, needs to manually adjust the screen extension configuration on the host computer, or needs to manually reconnect the additional displays in a correct order or change the physical positions of the additional displays.

The present invention aims to address these problems in the art to improve the overall performance and efficiency of screen extension.

SUMMARY

In this regard, according to one aspect of the invention, it is provided amethod for extending the display of a content on a first screen of a first device to a second screen of a second device, which comprises:

- detecting a movement of the second device;

- determining a position of the second device relatively to the first device based on the detected movement; and

- extending the display content on the first screen of the first device to the second screen of the second device based on the determined position.

With such a method according to the present invention, it provides an automatic way to correctly extent screen, and therefore a user no longer need to be aware in advance of the predefined connection order for screen extension and/or to manually adjust screen extension configuration based on the actual screen positions.

In an embodiment, the second device comprises an on-device camera, and the step of detecting the movement of the second device relatively to the first device comprises:

- capturing a plurality of first images by the on-device camera of the second device;

- comparing the captured first images in order to detect the movement of the second device.

Therefore, it is possible to wisely use normal on-device camerasfor an unexpected purpose.

In an embodiment, the on-device camera is a front camera.

Furthermore, the step of comparing the captured a plurality of images comprises:

- detecting a reference object in the captured images;

- detecting a displacement of the reference object in the captured images; and

- determining the movement of the second device relative to the first device based on the detected displacement of the reference object in the captured images.

By virtue of such an arrangement, it provides an efficient and simple way to detect the movement.

In an embodiment, the displacement of the reference object is a translation and/or a rotation thereof.

In an embodiment, wherein the reference object is a user face in front of the on-device cameras, especially when the on-device camera is a front camera.

Alternatively, the first device may also comprise an on-device camera, and the method further comprises:

- capturing a second image by the on-device camera of the first device; and

- verifying the determined position of the second device relatively to the first device, by comparing the second image with one of the first images;

wherein the step of extending display content is performed only if the verification of the determined position of the second device relatively to the first device outputs a positive result.

By virtue of such an arrangement, it provides more accurate movement detection and position determination.

In an embodiment, the position of the second device is on the left, right, top or bottom side of the first device. That being said, other positions of the second device relatively to the first device can also be determined.

In an embodiment, the method it comprises, after the step of detecting a position of the second device based on the detected movement, connecting the second device to the first device. Therefore, a position of the second device relatively to the first device can be learned before the actual display connection, so that it saves time to determine the extension position of the display content which further improves user experience.

In an embodiment, the step of connecting the second device to the first device comprises:

- calculating a first hash code of an image captured by the first device;

- calculating a second hash code of one of the images captured by the second device;

- determining whether these first and second hash codes match each other; and

- when said first and second hash codes are matching, connecting the second device with the first device and transmitting, from the second device to the first device, an information about the detected movement and/or the determined position of the second device.

Alternatively, other methods for the connection can also be implemented, as long as it requires less system resources.

Moreover, the second device is connected to the first device using a wire or wireless communication technology, for example a short-range wireless communication technology, such as Bluetooth.

In an embodiment, the step of extending display content on the first screen of the first device to the second screen of the second device based on the detected position comprises transmitting extended content corresponding to the detected position to second device for displaying said extended content on the second screen.

According to another aspect of the invention, it is provided a device for enabling screen extension, comprising:

- a screen;

- a movement detection module for detecting a movement of said device relatively to another device;

- a position detection module for determining a position of said device relatively to said other device based on the detected movement; and

- display content extension module for extending display content on the screen of the other device to the screen of said device for enabling screen extension, based on the detected position.

Such a device provides an efficient and automatic screen extension.

In an embodiment, the device as mentioned abovecomprises an on-device camera and the movement detection module comprises:

- a module for capturing a plurality of images by the on-device camera;

- a module for comparing the captured first images in order to detect the movement of the devicefor enabling screen extension.

According to yet another aspect of the invention, it is provided a screen extension system comprising a first device with a first screen and a second device as mentioned above, wherein the display content extension module of the second device is for extending display content on the first screen of the first device to the screen of the second device.

According to yet another aspect of the invention, it is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method as mentioned above.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present invention will appear in the description hereinafter, in reference to the appended drawings, where:

Figure 1 isa flowchart illustrating an exemplary method according to the present invention;

Figure 2 illustrates left and right screen extension according to the present invention;

Figure 3 illustrates an example of detecting movement of the screen extension devices and determining a position of screen extension devicesrelatively to the first device based on the detected movement;

Figure 4 illustrates an alternative example of detecting movement of the screen extension devicesand determining a position of screen extension devices relatively to the first device based on the detected movement; and

Figure 5illustrates an example of connecting screen extension devices to the first device after detecting a position of screen extension devicesbased on the detected movement.

DESCRIPTION OF EMBODIMENTS

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

The method according to the present invention is adapted to be used on various kinds of devices including screens that can be connected for screen extension. These devices include, but are not limited to, smartphones, tablets, in-car devices, augmented reality (AR) /virtual reality (VR) devices, laptops, ultra-mobile personal computer (UMPC) , netbooks, personal digital assistants (PDA) , artificial intelligence (AI) terminals and other terminal devices. The embodiments of this application do not limit any specific type of these devices.

An exemplary method according to the present invention is now discussed with reference to the flowchart offigure 1. In this exemplary method, a first and a second devices are implemented. The first device contains a first screen, and the second device contains a second screen, wherein the first screen is, for example, a host screen and the second screen is an additional screen to which the display content of the first screen can be extended.

The exemplary method comprises the following steps as shown in figure 1:

- at step S1, a movement of the second device is detected, for example, by movement detection module, such as gyroscope, or an on-device camera, which will be detailed later;

- at step S2, once the movement is detected, a position of the second device relatively to the first device is determined based on the detected movement, such a position could be on the left, right, top or bottom side of the first device; and

- at step S3, based on the determined position, the display content on the first screen of the first device is extended to the second screen.

For example, as shown in figure 2, by virtue of the abovementioned method, a first device 1 with a screen is present in the middle as a central host screen. A second device 2 with a second screen, and possibly a third device 3 with a third screen, are also provided to connect to the first device for extending the display content of the host screen to the second and/or the third screens.

Before the screen extension, the first device 1 usually only displays content on its own screen, i.e. the first screen, as a host screen, which is for example stationary at a position. In order to extend the screen, the second device 2 and/or the third device 3 are moved closer to the first device 1, and their movementsare detected. For example, when the second device 2 moves and approaches the left side of the first device 1 and/or the third device 3 moves and approaches the right side of the first device 1, these movements (especially the relative movements between the devices) are detected for example by a movement detection module equipped on the second and the third devices or by other external movement detection module.

Based on such a movement detection, the positions of the second device 2 and the third device3 relatively to the first device 1 are determined, for example by a position detection module equipped on the second and the third devices or by other external movement detection module. As shown in figure 2, the position of the second device 2 is determined as being on the left side of the first device 1, and the position of third device 3 is determined as being on the right side of the first device 1.

Accordingly, based on the determined positions, which correspond to the actual physical positions of the second and

third devices

2, 3 relatively to the first device 1, the display content on the first screen of the first device 1 may then be extended to the second screen and/or the third screen by a display content extension module with correctextension configurations, i.e. the left screen extension on the second device 2, and the right screen extension on the third device 3.

With such an arrangement, the extension configuration can be automatically adjusted based on the actual relative position between the first device and the second and/or third device, without manual adjustments as required in the art.

In order to detect the movement and determine the position of the second and/or third devices relatively to the first device, several movement detection technologies can be implemented according to the present invention. For example, gyroscope, G-Sensor and GPS can be used. More interestingly, in the present invention, another wise solution is proposed to detect and determine the position by using on-device camera on the device, which will be discussed below.

In one exemplary embodiment, the second device 2 and the third device 3 respectively comprise on-device cameras, preferably front cameras. As an example, hereinafter an exemplary screen extension on the second and the third devices will be discussed. However, the screen extension on different number of devices, i.e. two or more than three, can be respectively implemented in a similar fashion according to the present invention.

In particular, to detect the movement of the second device 2 and the third device 3 relatively to the first device 1, the on-device cameras of the second device 2 and the third device 3 respectively capture a plurality of images, and then these captured images are respectively compared to detect the movement of the second and the third devices.

More particularly, as shown in figures 3 and 4, the second device 2 captures one image before its movement and another image after its movement, and these two images contain the same reference object with different positions, i.e. displacement, in the image. For example, when the on-device camera is a front camera as shown in figures 3 and 4, the reference object is the portrait of a user who is in front of the device and is captured in the image.

Afterwards, a displacement of the reference object in the captured images is detected. For example, in the two images respectively captured before and after the movement of the device 2, the position of the portrait in the images moves with the displacement, i.e. a translation movement in figure 3 and a rotation movement in figure 4. By calculating such a displacement, the movement of the second devices relatively to the first device can be determined. For example, in figures 3 and 4, the movement of the second device 2 is detected as approaching to the left side of the first device 1 based on the displacement of the user’s portrait in the captured images, and therefore the position of the second device is then determined.

Similarly, the third device 3 in figures 3 and 4 can be used to detect its movement and determine its position in a similar manner.

Moreover, regarding the movement of the devices, figures 3 and 4 show different types of the movement.

In figure 3, the second and the

third devices

2, 3 make translational movements. In particular, the second device 2 moves towards to the first device from the left side and therefore the user’s portrait in the images have an obvious and detectable movement from right to left. With this characteristic, a right-to-left movement of the user’s portrait is detected, based on which a left extension position of the second device 2 is determined. Similarly, the third device 3moves towards to the first device from the right side and therefore the user’s portrait in the images have an obviousand detectable movement from left to right. With this characteristic, a left- to-right movement of the user’s portrait is detected, based on which a right extension position of the second device 3 is determined.

In figure 4, the second and the

third devices

2, 3 make rotational movements. In particular, the second and the third devices rotate along a Z-axis. The second device 2 rotates anti-clockwise and the user’s face in the portrait moves to the left part of the screen, and his left face directly faces to the screen which is captured by the front camera of the second device 2. Based on such a movement, the corresponding displacement is determined and then a left extension position is the determined for the second device 2. Similarly, the third device 3 rotates clockwise and the user’s face in the portrait moves to the right part of the screen, and his right face directly faces to the screen which is captured by the front camera of the third device 3. Based on such a movement, the corresponding displacement is determined and then a right extension position is the determined for the third device 3.

The translational and rotational movements may occur independently or simultaneously. Therefore, in an embodiment, in the present invention, both types of movements can be detected simultaneously, and one can be implemented as a complementary solution to the other, especially when the movement is not clear or stable.

Moreover, in addition to capturing user’s portrait in front of the devices, any other objects can also be captured as reference object by either front or back on-device cameras, as long as the movement and displacement of such objects can be detected in the images.

Alternatively, as shown in figures 3 and 4, the first device may also comprise an on-device camera, preferably a front camera. In this case, the method according to the present invention further comprises:

- capturing a second image by the on-device camera of the first device, wherein this second image stands still as the first device is usually used as a host device with a stationary position; and

- verifying the determined position of the second/third devices relatively to the first device, by comparing the second image with one of the first images;

Only if the verification of the determined position of the second device relatively to the first device outputs a positive result, the following step of extending display content will be performed.

In an embodiment, after detecting a position of the second/third device based on the detected movement, an exemplary method according to the present invention comprises a step of connecting the second device to the first device comprisesin order toextend screen. Alternatively, the connection can be established beforehand, e.g. before the movement detection.

In an advantageous embodiment, the step of connecting the second device to the first device comprises:

- calculating a first hash code of an image captured by the first device;

- determining whether these first and second hash codes match each other;

- when said first and second hash codes are matching, connecting the second device with the first deviceand transmitting, from the second device to the first device, an information about the detected movement and/or the determined position of the second device.

In a specific embodiment of this embodiment, the above-mentioned connecting and transmitting steps are only performed when the first and second hash codes matches each other, i.e. if they do not match, the second device does not connect with the first device, no information is thus transmitted to the first device and the screen of the first device is not extended to the screen of the second device.

In particular, in order to determine whether the first and second hash codes match each other, these hash codes are compared in order in order to provide similarity score, which can be compared to a similarity threshold above which both hash codes (thus both images they are derived from) are considered as matching. For performing such a comparison and similarity code calculation, a pHashalgoritm can be used, which is known in the art and will not be discussed in detail in the present application.

In a more particular example as shown in figure 5, the first, the second and the

third devices

1, 2, 3may communicate via cables such as USB cable or wireless communication technology, especially a short range wireless communication technology, such as Bluetooth Advertising technology, to exchange the information among these devices.

According to one exemplary embodiment, the first device 1, which captures a portrait image via its on-device camera, puts a hash code of the captured portrait (e.g. “bc10abcde21efbcc” in figure 5) in the Protocol DataUnit (PDU) of the Bluetooth Advertising protocol. Then the first device 1 waits for the extension device (the second and/or the third devices 2, 3) todiscover it and informs it about the movements of the extension device.

Afterwards, the extension devices, e.g. the second and the

third devices

2, 3, discover the first device nearby. Once finding a candidate, the extension device (s) will usethe hash code in the candidate's PDU to compare it with the hash code of the portrait image captured by their own on-device camera (s) (e.g. “bc11abdde21efbb9” of the portrait image caputed by the second device 2 and “bc11dadde22efbb2” of the portrait image caputed by the second device 2, in figure 5) . If the similarityof two codes (e.g. 0.91 between the first and the second devices, and 0.94 between the first and the third devices) is higher than a threshold, e.g. 0.85, the extension device will consider that is exactly the hostscreen (the first device) the extension devices are looking for. Then, the extension devices 2 and 3may connect to the first device 1 and inform the latter aboutthe movement direction of the portrait.

According to this movement direction as received from the

extension devices

2 and 3, the first device 1 will derive the extension position of the

devices

2 and 3.

It should be noted that, in this embodiment, it is the extension device (i.e. the second and/or the third devices 2, 3) that detects its movement direction and the first device that derives the extension position from this movement direction, hence the sending of an information about the movement direction from the extension device to the fi rst device.

However, alternatively, sincethe movement direction of the extension deviceis naturally detected by the extension device, the position of this extension device does not have to be derived by the first device, but could also be derived by the extension device itself. In such a case, it is an information about the position of the extension device, not about itsmovement direction, that is communicated to the first device, for the purpose of content transmission.

In another alternative, the position of the extension device relatively to the first device can also be derived by a third-partydevice/network server. In that case, the extension device provides an information about the movement direction to this third-party device/network server, which derives the extension device position relatively to the first device, then sends an information about this position to the first device for the purpose of content transmission and screen extension.

Afterwards, the first device 1, which is for example a host device for the content to be displayed (i.e. the device storing or receiving this content, for instance a streamed video) , extends its display content on its screen (i.e. the first screen) to the second screen and/or the third screen of the second device 2 and the third device 3. Since the extension position of the second device (e.g. left extension) and the third device (e.g. the right extension) is already determined as mentioned above, there is no need to configure the extension position manually by the user. The following content transmission between the first device 1 and the

extension devices

2, 3 can be proceeded based on this automatically determined extension position configuration by means of any display content transmission technologies in the art. As these technologies already exist and are known to the skilled person in the art, they will not be discussed.

Moreover, it is known to those skilled in the art, the aforementioned examples according to the present invention, can be implemented in many ways, such as program instructions for execution by a processor, as software modules, microcode, as computer program product on computer readable media, as logic circuits, as application specific integrated circuits, as firmware, etc. The embodiments of the invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

Furthermore, the embodiments of the invention can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer, processing device, or any instruction execution system. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can contain, store, communicate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be electronic, magnetic, optical, or a semiconductor system (or apparatus or device) . Examples of a computer-readable medium include, but are not limited to, a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a RAM, a read-only memory (ROM) , a rigid magnetic disk, an optical disk, etc. Current examples of optical disks include compact disk-read-only memory (CD-ROM) , compact disk-read/write (CD-R/W) and DVD.

The embodiments described hereinabove are illustrations of this invention. Various modifications can be made to them without leaving the scope of the invention which stems from the annexed claims. In particular, a specific example has been illustrated where the screen of a first device is extended to the screens of two other devices, but the invention is not limited to this specific example and can be applied to extend the screen of a first device to the screen (s) of a single or any number of secondary device (s) .

Claims

A method for extending the display of a content on a first screen of a first device to a second screen of a second device, comprising:

- detecting a movement of the second device;

- determining a position of the second device relatively to the first device based on the detected movement; and

- extending the display content on the first screen of the first device to the second screen of the second device based on the determined position.
The method according to claim 1, wherein the second device comprises an on-device camera, wherein the step of detecting the movement of the second device relatively to the first device comprises:

- capturing a plurality of first images by the on-device camera of the second device;

- comparing the captured first images in order to detect the movement of the second device.
The method according to claim 2, wherein the step of comparing the captured a plurality of images comprises:

- detecting a reference object in the captured images;

- detecting a displacement of the reference object in the captured images; and

- determining the movement of the second device relative to the first device based on the detected displacement of the reference object in the captured images.
The method according to claim 3, wherein the displacement of the reference object is a translation and/or a rotation thereof.
The method according to claim 3 or 4, wherein the reference object is a user face in front of the on-device cameras.
The method according to any one of claim 2 to 5, wherein the first device comprises an on-device camera, the method further comprising:

- capturing a second image by the on-device camera of the first device; and

- verifying the determined position of the second device relatively to the first device, by comparing the second image with one of the first images;

the step of extending display content being performed only if the verification of the determined position of the second device relatively to the first device outputs a positive result.
The method according to any one of claims 1 to 6, wherein the position of the second device is on the left, right, top or bottom side of the first device.
The method according to any one of claims 1 to 7, wherein it comprises, after the step of detecting a position of the second device based on the detected movement, connecting the second device to the first device.
The method according to claim 6, wherein it further comprises a step of connecting the second device to the first device comprises:

- calculating a first hash code of an image captured by the first device;

- calculating a second hash code of one of the images captured by the second device;

- determining whether said first and second hash codes match each other;

- when said first and second hash codes are matching, connecting the second device with the first deviceand transmitting, from the second device to the first device, an information about the detected movement and/or the determined position of the second device.
The method according to claim 8 or 9, wherein the second device is connected to the first device using a short-range wireless communication technology.
The method according to any one of claims 1 to 10, wherein the step of extending display content on the first screen of the first device to the second screen of the second device based on the detected position comprises transmitting extended content corresponding to the detected position to second device for displaying said extended content on the second screen.
A device for enabling screen extension, comprising:

- a screen;

- a movement detection module for detecting a movement of said device relatively to another device;

- a position detection module for determining a position of said device relatively to said other device based on the detected movement; and

- display content extension module for extending display content on the screen of the other device to the screen of said device for enabling screen extension, based on the detected position.
The device for enabling screen extension according to claim 12, wherein the device comprises an on-device camera and wherein the movement detection module comprises:

- a module for capturing a plurality of images by the on-device camera;

- a module for comparing the captured first images in order to detect the movement of the device for enabling screen extension.
A screen extension system comprising a first device with a first screen and a second device according to one of claims 12 and 13, wherein the display content extension module of the second device is for extending display content on the first screen of the first device to the screen of the second device.
A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 11.