GB2406455A - Ensuring alignment of user image in a mobile video conferencing arrangement - Google Patents

Abstract

A mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and indication means, the terminal being adapted to (i) analyse video data captured by the camera to determine whether the image is misaligned; and (ii) if the terminal detects in step i) that the image is misaligned, indicate via the indication means, to the user the misalignment of the image.

Description

Mobile Communications This invention relates to mobile communications

terminals and methods of performing mobile communications. More particularly, but not exclusively, the method relates to a terminal and method for aligning a video image in a video call.

Background Art

Video capability is one of the advantages that mobile phones of the third generation (3G) can offer. Some terminals of the third generation are designed to allow a full two-way video phone call.

Teleconferencing systems often provide a so-called picture-in-picture function feature that allows monitoring of the pictures or video images captured by the camera together with the images transmitted via the network.

The monitoring image is usually provided as an inset in one corner of the display, simultaneously with displaying the images transmitted via the network. In this way the user can view the transmitted images in a larger format, whereas the monitoring image is small compared to the main display.

Figure 5 A illustrates a display of a mobile communication terminal in the picture-in-picture mode. The display26 includes a main display portion 12 and a picture-in-picture portion 14. The main display 26 displays images transmitted via the communications network, and the picture-in picture 14 shows images captured by the camera of the mobile terminal.

However, this picture-in-picture function may be disadvantageous for some mobile terminals or some applications, for example for terminals with a relatively small display. In these terminals, a picture inlet may be too small to be helpful in monitoring and properly aligning the images received via the camera.

Also, depending on light conditions, a "picture-in-picture like" inset to monitor the images might be difficult to see because of reflections or other interfering light conditions.

Also, the user might not want to monitor the picture-in-picture image simultaneously to the images received via the communications.

It is therefore an object of the present invention to alleviate at least some of the disadvantages described above.

It is another object of the present invention to provide an indication to a user of a mobile terminal engaged in a video call to indicate if the image as presented to the remote party is aligned on the display.

According to one aspect of the present invention, there is provided a mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and indication means, the terminal being adapted to (i) analyse video data captured by the camera to determine whether the image is misaligned; and (ii) if the terminal detects in step i) that the image is misaligned, indicate via the indication means, to the user the misalignment of the image.

In this way it is ensured that the user is notified if he or she is holding the mobile terminal incorrectly. Accordingly, the use of a terminal including a camera is simplified. Also, the user does not need to monitor his camera's image in a "picture-in-picture" function or something similar.

Preferably, the terminal is adapted to indicate to the user how alignment of the image is to be achieved, for example by indicating the direction of a required movement of the camera. Again, in this way the user does not need to monitor his camera's image in a "picture-in-picture" function or something similar, but the user is ensured that he or she is notified if the terminal including the camera is incorrectly held.

According to another aspect of the present invention, there is provided a mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and a display for displaying video images, the terminal being adapted to conduct a video call with a second party's terminal and, upon receiving indications that an image received from the second party's terminal during a video call is misaligned, to automatically send a message to the second party's terminal including information relating to the misalignment of the image.

In this way the user of the terminal conducting a video call can notify the other party in the call when the user or the other party holds the terminal incorrectly such that the camera is not properly aligned.

The user may for example provide the information that the camera is not properly aligned via the transfer of data between the two mobile terminals during the call. This may be achieved via the use of messages such as the Unstructured Supplementary Service Data (USSD) or a short message service, such as the SMS of the GSM systems.

According to another aspect of the present invention, there is provided a first mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and a display for displaying video images, the terminal being adapted to conduct a video call with a second party's terminal and, upon receiving a message from the second party's terminal including information relating to the alignment or misalignment of the image captured by the first mobile terminal, to provide indications to its user relating to the alignment or misalignment of the image.

In this way the terminal of a user conducting a video call is adapted to indicate to the user that the image captured by his or her camera is misaligned, if the terminal has been received according information from the other party of the video call.

According to another aspect of the present invention, there is provided a method of operating a mobile video phone, wherein the method comprises the steps of: (i) during a video call, between a first mobile terminal and a second mobile terminal, obtaining information on whether the image captured by the terminal's camera is aligned, and (ii) displaying graphical indications indicating to the terminal's user whether the image captured by the camera is aligned.

The invention will now be described, by way of example only, with reference to the accompanying drawings in which Figure 1 is a schematic front view of a videophone in which the present invention can be implemented; Figure 2 is a schematic block diagram of the functional elements of a videophone in accordance with the invention, Figure 3A to 3D are illustrations of the video displays of two videophones during a video call according to one embodiment of the present invention; s Figure 4 is a block diagram illustrating the software architecture according to one embodiment of the present invention.

Figure 5 A is a schematic view of a videophone display according to

the prior art;

Figure 5 B is a schematic view of a videophone display according to another embodiment of the present invention; and Figure6A to 6D are illustrations of the video displays of two videophones during a video call according to another embodiment of the present invention.

Figure I is a schematic illustration of a mobile communication terminal 10. The terminal 10 includes a display 26, a camera 24, a microphone 16, speakers 18, a keypad 21 and navigation keys 23 Figure 2 is a schematic block diagram of the main functional elements which may be included commonly to the different embodiments of the invention, which elements are each individually known and will not be described in detail herein. A main processor 36 may be a conventional programmable microprocessor (for example an Intel 80386, 80486, etc.), or a special purpose or specially configured unit (e.g. a digital signal processor) could alternatively be used. A read-only memory (ROM) 38 is connected to the processor 36 for the storage of control programs, data and images. The ROM 38 can be implemented by any appropriate technology, for example, by a flash PROM. A random-access memory (RAM) 40 is connected to the processor 36 via a bus 42, is used as a working storage and for the storage of data and images captured using a CCD video camera 24.

Signals relating to the data captured by the camera are passed via a camera interface 44 to the processor 36 to be processed. The camera interface 44 also provides the video codec 46 with a digital representation of the captured data from the camera 24, where it can be suitably processed for display and/or transmission to the mobile communications system.

The camera interface 44 carries out all the necessary signal conditioning as required on receiving images from the camera 24. Signal conditioning will depend on the exact configuration of the camera but preferably comprises signal conditioning to enable accurate analogue to digital conversion with sufficient buffering of the captured data. The camera 24 will include all the necessary support circuitry to produce a fully functional camera delivering a fully formatted video signal. The camera 24 may also include circuitry to regulate the voltage for power supply control and a suitable output buffer to directly drive a standard VDU should the videophone be connected to an external device.

A display interface 52 connects the display 26 via the bus 42 to the processor 36. The display interface 52 responds to instructions from the processor 136 to drive the built-in display 26 in a conventional manner.

The display 26 is provided with a touch-screen 56. A touch-screen interface couples the touch-sensitive display 26 to the processor 36 via the bus 42. The touch-screen is a mechanism independent of the video display 26, for example, a transparent touch-screen membrane which is placed over the display 26 and connected appropriately.

The processor 36 can be arranged to transmit to the display 26 a menu of user selectable items, and to be responsive to a location at which the screen is touched for input of the user selection of an item. The touchsensitive screen can then thus be used as a dynamic and reconfigurable user interface.

Touch-screen entry can be used in place of or in addition to the entry's commands from an external keyboard or voice command if appropriate.

Additionally, the touch-screen area can be configured as a general purpose area to allow entry of data and written commands.

An audio interface 56 connects the audio receiver means, consisting of a microphone 18 and audio transmitter means such as an ear-piece and/or speaker 16 to the processor 36 and carries out all the necessary signal conditioning as required to output audio signals and to receive audio signals.

A radio-frequency (RF) interface 62 is also connected via the bus 42 to convert any data to be transmitted into signals for driving an RF transmitter 64, and converts signals from an RF receiver 66 into data to be passed via the bus to the relevant interfaces. The RF transmitter 64 and the RF receiver 66 are connected to a radio antenna 28. This RF interface 62 consequently enables wireless communications between the videophone and the mobile communications system.

The processor 36 is programmed by means of control programs and data stored in the ROM 38 and in use, the RAM 40, to receive signals from the camera 24 via camera interface 44, to interpret those signals and to derive data therefrom which are displayed on display 26 and which can be stored in the RAM 40 or any other suitable memory device.

Depending on the refresh rate used and the number of pixels used in the images, video image data transmitted and received by the videophone may require compression for transfer via a low data rate radio channel, such as those currently available in known cellular radio networks. The video data S may be compressed using the MPEG-4 standard. Alternatively, the video images captured may be compressed into a different format suitable for transmitting the data derived across the mobile communications system, such as that disclosed in WO95/20296.

In the following embodiment of the present invention will be described with reference to Figure 3. In one example, two users, Homer and Marge, conduct a video call. Figure 3A and B depict the video display on Marge's and Homer's mobile terminal, respectively. In Figure 3A, Marge can only see the top of Homer's head, as Homer is not holding his mobile terminal correctly. Homer holds the terminal in a way such that the camera in Homer's mobile terminal picks up only the top of his head. Homer's phone, on the other hand, displays a complete image of Marge's head.

However, Homer's mobile terminal analyses the images receive by the camera and detects that the camera is held in a way such that the image displayed on Marge's phone is not properly aligned.

Subsequently, Homer's mobile terminal displays an indication in the top right hand corner of the display indicating to Homer that he does not hold this terminal correctly (see Figure 3B). An arrow pointing downwards is displayed and indicates to Homer that he needs to tilt his mobile terminal downwards in order to properly align the image captured by his terminal's camera.

In response to this indication, Homer tilts the terminal downwards.

The camera is now properly aligned and Marge sees the image correctly in the display of her mobile terminal (see Figure 3C).

Homer's mobile terminal keeps analysing the image captured by the camera while Homer corrects the direction of the camera in his mobile terminal. When Homer holds the terminal correctly such that Homer's image is correctly aligned, Homer's terminal indicates this fact by displaying a circle in the top right hand corner (see Fig. 3D). Thus Homer knows that he has now correctly aligned the image and that he no longer needs to tilt or adjust the terminal.

With reference to Figure 4, the implementation of an embodiment of the present invention will now be described. Figure 4 is a block diagram showing the software elements as implemented in processor36.

Alternatively, more than one processor can be used. In this case several digital signal processors (DSP) can be used to handle different signals, such as the audio signals or video signals.

Video call application software70 communicates with the display driver software76, the audio codec 78 and the video codec in order to produce the audio/video stream72 and the formatted video stream74 for performing a video call.

An additional software element 80 is provided for controlling the alignment of the video images captured by the camera. Element 80 includes three different components, which will be described in more detail below.

The frame capture module 82 provides the input data for picture analysis by receiving frames from the video stream and making the frames available for processing. Module 82 may also perform some data compression in order to reduce the processing requirements of the picture analysis module 84 or other preprocessing steps. For example, the image may be saved at a lower resolution and/or colour depth.

The picture analysis module 84 performs image processing in order to determine the position of a person's face within the image. The step(s) of analysing the picture will be described in more detail below.

The indication generation module 86 receives from the picture analysis module 84 an indication of the position of the user's face on the screen. From the face position, the indication generation module 86 then determines in which direction the camera has to be moved in order to centre the user's face in the image displayed.

The indication generation module 86 then generates commands which arc communicated to the display driver software 76. The commands include details of graphical indications to be displayed to the user, for example an arrow on the top right hand side corner of the display as shown in Figure 3B.

The graphical indication are then generated by the display driver software 76 and displayed on display 26.

For example, if the user's image includes only the top of the user's head as shown in Figure 3A, the indication generation module determines that the user's face on the image can be centred by moving the phone including the camera downwards or tilting the phone downwards.

In a similar manner the direction in which the camera and/or phone needs to be moved or tilted can be determined if the image shows only the bottom, left or right side of the user's head.

In order to analyse the image data and to determine the position of a person's face within the received image in module 84, known techniques of head or face detection are applied. Such techniques are described in more detail for example in US patent US 6,301,370 or in European patent application EP 806739.

The position of the user's face may for example be determined with a head detection and tracking process. In such a method the head in an image is tracked by "visual pathways" using motion, colour, site, topology or pattern.

In addition, the face may be tracked in a landmark finding process, looking for "landmark" features like the eyes, the nose and/or the mouth.

Other methods are based on a feature detection in a first step. Here facial parts such as the eyes, nose and mouth are located by segmentation of the input image data to isolate the facial parts. In addition, the geometrical relationship between the facial parts may be used.

Other methods known in the art include the Karhunen-Loeve decomposition as used in statistical recognition and detection techniques, see for example C.W. Helstrom, "Statistical Theory of Signal Detection'", Pergamon Press, 1968, face recognition techniques using Eigenfaces, wherein a person's face is essentially represented as a vector of weights applied to a predefined set of Eigenfaces, see for example M.A. Turk and A.P. Pentland, "Face Recognition using Eigenfaces", Proc. Int. Conf. on Pattern Recognition, pp. 586-59], 1991, or face recognition techniques using neural networks, see for example R. Chellapppa, C.L. Wilson and S. Sirohey, "Human and Machine Recognition of Faces: A Survey", Proceedings of the IEEE, May 1995.

Alternatively, edge detection methods may be used to extract the edges or contours of a face from the image data received. Fast and simple edge detection can be achieved by filters such as the Sobel operator (see for example M.D. Levine, "Vision in Man and Machine", McGraw-Hill, 1985.

Still another method is to compare the captured image data to a reference image, determine correlations between the images and locate the user's head of face using these correlations.

The reference image of a user's face may be prerecorded and stored before a video call is performed.

It is appreciated that two or more of the methods described may be combined in order to facilitate or improve the detection of the user's face.

It is appreciated that alternatively other graphical signs may be used.

With reference to Figure SB an alternative display for indicating directions according to another embodiment of the present invention is described. The display contains an array 90 of arrows 92, wherein each of the arrows points in different direction, such as up, down, left and right. The terminal performs the analysis of the images captured by the camera in the same way as described above. If the terminal detects that the user holds the terminal such that the image captured by the camera is not properly aligned in the image of the other party's display, the terminal then highlights one or two of the arrows 92 to indicate in which direction the user has to move or tilt the terminal in order to align the image.

For example, the arrow pointing upwards will be highlighted if the user needs to move the terminal upwards to align the image, or the arrow pointing upwards and to the right will be highlighted if the user needs to move the terminal upwards and to the right hand side to align the image. The array l 5 may additionally include a circle 94, which will be highlighted if the user has successfully aligned the camera by moving the terminal in the direction indicated by the array of arrows.

Although graphical indications provided on the display of the terminal have been described in the embodiments above, it is appreciated that the indications for indicating the required movement of the camera to align the image may be provided in other ways.

The array described above with reference to Figure SB may for example be provided by the navigation keys 23, i.e. the up, down left and right keys on the keypad of the terminal, and the according key may be highlighted to indicate a direction. When the terminal has detected that the camera is properly aligned, all of the up, down left and right keys on the keypad may be highlighted for a short period of time to indicate that the camera has been successfully aligned.

A second embodiment of the present invention will now be described with reference to Figure 6.

Similarly to the example described above with reference to Figure 3, two users, Homer and Marge, conduct a video call. Figure 6A and B depict the video display on Marge's and Homer's mobile terminal, respectively. In Figure 6A, Marge can only see the top of Homer's head, as Homer is not holding his mobile terminal correctly. Homer holds the terminal in a way such that the camera in Homer's mobile terminal picks up only the top of his head. However, Marge's phone displays an additional key 222 on the left side of the display, and an array 224 of arrows 226 on the bottom of the display.

The displayed elements 222 and 226 are keys on the touch sensitive screen of the display.

According to the second embodiment of the invention, Marge can notify Homer when Marge sees that Homer's image transmitted in the video call and displayed on her terminal is misaligned. By activating key 222, Marge's phone automatically sets up a short message to Homer's phone notifying him of the fact that Homer's image captured by Homer's terminal is misaligned, and subsequently transmits the message to Homer's terminal. In addition, Marge can help Homer with the alignment of his phone by sending information in which direction Homer has to move or tilt his terminal in order to align the image.

When Marge is presented with Homer's image as depicted in Figure 3A, she may press the upwards pointing key 226 to let Homer know that he needs to tilt the terminal upwards to align the image. Marge's terminal then automatically transmits a short message to Homer's terminal containing the information given by Marge.

On reception of the short message, Homer's terminal indicates to Homer by displaying an arrow pointing downwards in the top right hand corner of the diplay (see Figure 6B) that he needs to tilt the terminal upwards in order to align the image.

In response to this indication, Homer tilts the terminal downwards.

The camera is now properly aligned and Marge sees the image correctly in the display of her mobile terminal (see Figure 6C).

After Marge's mobile terminal has sent the short message to Homer's terminal, the terminal displays a circle 228 instead of the key 222 on the left side of the display. When Marge presses this key, another short message is automatically sent to Homer's terminal, notifying Homer that his image is now properly aligned.

When Homer's terminal receives the short message, it indicates that the image is now properly aligned by displaying a circle in the top right hand corner (see Figure 6D). Thus Homer knows that he has now correctly aligned the image and that he no longer needs to tilt or adjust the terminal.

Again, although graphical indications provided on the display of the terminal have been described in the embodiments above, it is appreciated that the indications for indicating the required movement of the camera to align the image may be provided in other ways.

The array described above with reference to Figure 6B may for example be provided by the navigation keys 23 of the terminal, i.e. the up, down left and right keys on the keypad of the terminal, and the corresponding key may be highlighted to indicate a direction. When the terminal notifies the user that the camera is properly aligned, all of the up, down left and right keys on the keypad may be highlighted for a short period of time to indicate that the camera has been successfully aligned.

It is appreciated that the different embodiments of the present invention described above may be combined in any way. For example, the terminal may be adapted to be able to analyse the image received by the camera to determine whether the image is properly aligned, and, in addition, may be adapted to receive information, from the other party in a video call, including information relating to the alignment of the user's camera. In this way the other party may notify the user if he or she does not hold the terminal correctly, for example because the user does not pay attention to the indications indicating misalignment of the images captured by the user's 1 S camera as provided by the terminal.

Although graphical indications have been described in the embodiments above, it is appreciated that the indications for indicating the required movement of the camera to align the image may be provided in other ways, for example by audio or tactile indications. The terminal may for example vibrate if the image captured by the camera is misaligned, or the terminal may indicate a misaligned image by one or more predetermined tones, melodies or voice generated by the terminal.

Although the present invention has been described by way of example only and with reference to the possible embodiments thereof, it is to be appreciated that improvements and/or modifications may be made thereto without departing from the scope of the invention as set out in the appended claims.

Claims (27)

1. A mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and indication means, the terminal being adapted to: (i) analyse video data captured by the camera to determine whether the image is misaligned; and (ii) if the terminal detects in step i) that the image is misaligned, indicate via the indication means, to the user the misalignment of the image.

2. A terminal according to claim 1, wherein in step ii) the terminal is adapted to indicate to the user how alignment of the image can be achieved.

3. A terminal according to claim2, wherein the terminal is adapted to indicate the direction of movement of the camera required to align the image.

4. A terminal according to claims 1, wherein the terminal is adapted to indicate when the image is aligned.

5. A terminal according to claim 1, wherein the terminal is adapted to analyse the image in step i) using one of the following techniques: head detection, face detection, face recognition or correlation techniques.

6. A terminal according to claim 1, wherein the terminal is adapted to indicate misalignment with graphical indications.

7. A terminal according to claim 6, wherein the graphical indications are provided on a display. ]O

8. A terminal according to any of claim 1, wherein the terminal is adapted to indicate misalignment with audio indications.

9. A terminal according to any of claim 1, wherein the terminal is adapted to indicate misalignment with tactile indications.

10. A mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and a display for displaying video images, the terminal being adapted to conduct a video call with a second party's terminal and, upon receiving indications that an image received from the second party's terminal during a video call is misaligned, to automatically send a message to the second party's terminal including information relating to the misalignment of the image.

A first mobile terminal for use in a cellular telecommunications network, the terminal comprising a video camera for receiving video images and a display for displaying video images, the terminal being adapted to conduct a video call with a second party's terminal and, upon receiving a message from the second party's terminal including information relating to the alignment or misalignment of the image captured by the first mobile terminal, to provide indications to the user of the first mobile terminal relating to the alignment or misalignment of the image.

12. A mobile terminal according to claim 10 or 11, wherein the information transmitted via the message includes indications how alignment of the image can be achieved.

13. A mobile terminal according to claim to or 11, wherein the information includes directions of movement of the camera required to align the image.

14. A mobile terminal according to claim 10, wherein the terminal is further being adapted, upon receiving indications that the image from the second party's terminal received during the video call is aligned, to automatically send a message to the second party's terminal including information relating to the alignment of the image.

15. A terminal according to claims 10 or 11, wherein the terminal is adapted to indicate misalignment with graphical indications.

16. A terminal according to claims 10 or 11, wherein the graphical indications are provided on a display.

17. A terminal according to claim 10 or 11, wherein the terminal is adapted to indicate misalignment with audio indications.

18. A terminal according to claim 10 or 11, wherein the terminal is adapted to indicate misalignment with tactile indications.

19. A terminal according to claim 10 or 11, wherein said message is of the Short Message Service (SMS) or the Unstructured Supplementary Service Data (USSD) type.

20. A method of operating a mobile video phone, wherein the method comprises the steps of: i) during a video call, between a first mobile terminal and a second mobile terminal, obtaining information on whether the image captured by the terminal's camera is aligned; and ii) displaying graphical indications indicating to the terminal's user whether the image captured by the camera is aligned.

21. A method according to claim 2O, wherein in step i) the information is received from the second mobile terminal.

22. A method according to claim 21, wherein in step i) the information is obtained by the first terminal by analysing a frame of the image received from the first terminal's camera.

23. A terminal according to claim 22, wherein the terminal is adapted to analyse the image in step ii) using one of the following techniques: head detection, face detection, face recognition or correlation techniques.

24. A terminal according to claim 2O, wherein the terminal is adapted to indicate misalignment with graphical indications.

25. A terminal according to claim 2O, wherein the graphical indications are provided on a display.

26. A terminal according to any of claim 2O, wherein the terminal is adapted to indicate misalignment with audio indications.

27. A terminal according to any of claim 20, wherein the terminal is adapted to indicate misalignment with tactile indications.