WO2006021444A2

WO2006021444A2 - Electronic equipment for a wireless communication system to transmit and receive information content during ongoing communication

Info

Publication number: WO2006021444A2
Application number: PCT/EP2005/009198
Authority: WO
Inventors: Fredrik Ramsten; Emil Hansson
Original assignee: Sony Ericsson Mobile Communications Ab
Priority date: 2004-08-25
Filing date: 2005-08-25
Publication date: 2006-03-02
Also published as: WO2006021444A3

Abstract

The present invention relates to an electronic equipment for a wireless communication system enabling communication within the wireless communication system by transmitting a first audio signal and/or a first video signal, and enabling transmission of information content within the wireless communication system by transmitting a second video signal and/or a second audio signal, comprising an encoder (5) for encoding audio and/or video signals to be transmitted within the wireless communication system and an audio mixing means (13) for interleaving said second audio signal with said first audio signal and submitting the interleaved audio signal to the encoder (5). The present invention further relates to a method for executing the steps on the electronic equipment.

Description

SONY ERICSSON MOBILE COMMUNICATIONS AB

PS04 0153EP

P30365WO

TITLE

Electronic equipment for a wireless communication system

TECHNICAL FIELD OF THE INVENTION

The present invention relates to an electronic equipment for a wireless communication system. Specifically, the present invention relates to the problem of keeping up a communication and at the same time transmitting and receiving information content.

DESCRIPTION OF RELATED ART

At present, most electronic equipments for a wireless communication system, such as mobile telephones, pagers, personal digital assistants, electronic organisers and so forth, either enable the user to set up a communication to another person, i.e. a call, a video conference or the like, or to transmit and receive information content such as images, videos, sound, drawings, sketches, text, documents and the like. Thereby, it is not possible to keep up the communication and at the same time transmit and/or receive information content in order to share and discuss information content in real¬ time like in a normal face to face conversation. Therefore, it is not possible to get instant feedback on the information content, since it is not possible to send information content to the other party during an ongoing communication.

SUMMARY

The general problem with the state of the art electronic equipment for a wireless communication system as outlined above, is, that for sending or receiving information content the communication would have to be interrupted. Known electronic equipments do not provide the functionality of sending information content during an ongoing communication. The object of the present invention is therefore to provide an electronic equipment for a wireless communication system as well as a method for an electronic equipment for a wireless communication system, which enable the user to transmit and receive information content during an ongoing communication thereby getting an instant feedback from the person he is communicating with.

The above-mentioned object is achieved by an electronic equipment for a wireless communication system according to claim 1.

The present invention relates to an electronic equipment for a wireless communication system enabling communication within the wireless communication system by transmitting a first audio signal and/or a first video signal, and enabling transmission of information content within the wireless communication system by transmitting a second audio signal and/or a second video signal and comprising an encoder for encoding audio and/or video signals to be transmitted within the wireless communication system and an audio mixing means for interleaving said second audio signal with said first audio signal and submitting the interleaved audio signal to the encoder.

The above-mentioned object is further achieved by a method for an electronic equipment for a wireless communication system according to claim 8.

The present invention further relates to a method for transmitting information in a wireless communication enabling communication within the wireless communication system by transmitting a first audio signal and/or a first video signal, and enabling transmission of information content within the wireless communication system by transmitting a second audio signal and/or a second video signal and comprising the steps of encoding audio and/or video signals to be transmitted within the wireless communication system by an encoder and, in case of transmitting said first and second audio signal interleaving said second audio signal with said first audio signal and submitting the interleaved audio signal to the encoder.

Preferably a file system contains a file consisting of said second audio signal and/or said second video signal.

Further, preferably an audio converter converts said second audio signal to a predefined format. Advantageously, a separation means separates the second video signal from the second audio signal.

Further, advantageously a selecting means selects said second video signal or said first video signal and submits the selected video signal to the encoder.

Preferably, the selecting means selects the first video signal only when no second video signal is present.

Further, preferably a video converter converts said second video signal to a predefined format.

The above-mentioned object is further achieved by an electronic eqipment for a wireless communication system according to claim 15.

The present invention further relates to an electronic equipment for a wireless communication system enabling communication within the wireless communication system by transmitting a first signal comprising an audio and/or video signal and enabling transmission of information content within the wireless communication system by transmitting a second signal, comprising an encoder for encoding the first signal to be transmitted within the wireless communication system and a multiplexer for multiplexing the first signal received from the encoder with the second signal.

The above-mentioned object is further achieved by a method for an electronic equipment for a wireless communication system according to claim 18.

The present invention further relates to a method for transmitting information in a wireless communication system enabling communication within the wireless communication system by transmitting a first signal comprising an audio and/or video signal and enabling transmission of information content within the wireless communication system by transmitting a second signal, comprising the steps of encoding the first signal to be transmitted within the wireless communication system by an encoder and multiplexing the first signal received from the encoder with the second signal by a multiplexer.

Preferably, the second signal is transmitted on a client level.

Further, preferably the second signals are text strings. It should be emphasised that the term "comprises/comprising" when used in the specification is taken to specify the presence of stated features, integers, steps or components but does not preclude the presence or edition of one or more other features, integers, steps, components or groups thereof.

In context of the present application and the present invention, the term "electronic equipment for a wireless communication system" includes any kind of portable radio communication equipment. The term "portable radio communication equipment" includes all equipments such as mobile telephones, mobile self-phones, pagers, personal digital assistants, communicators, i.e. electronic organisers, smart phones or the like. The term "wireless communication system" relates to any kind of communication or tele-communication system which enables the wireless transfer of information.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example only, which reference to the accompanying drawings in which:

Fig. 1 is a block diagram showing schematically the elements of the system,

Fig. 2a and 2b are flow charts showing schematically the elements of an image transmitting system,

Fig. 3a and 3b are a block diagram showing schematically the elements of an audio transmitting system,

Fig. 4 is a block diagram showing schematically the elements of a video transmitting system,

Fig. 5 is a flow chart showing schematically the process steps according to the method of the present invention,

Fig. 6a to 6d are schematic use of different display images during the process and Fig. 7 is. a block diagram showing schematically the elements of an electronic equipment according to a second embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENT

Fig. 1 shows a schematic block diagram of the system according to the present invention. An electronic equipment 1 comprises an audio input means 2, which may be a microphone or, a voice recognition means or the like, for accepting or receiving voice, sounds, noise or other audio signals from the environment of the electronic equipment 1. A video input means 4, which may be a camera or the like, serves for accepting or receiving videos, still images or other video signals. In addition, a file system 3 contains information content, that is files consisting of audio and/or video components including images, videos, sound, drawings, sketches, text, documents and the like. Hereby, "audio" according to the present invention relates to acoustic data, like speech, music, noises or the like and "video" relates to moving or still optical data like images, texts and the like. Combined audio and video data may be music videos, a video conference or the like.

The electronic equipment is able to transmit and receive information in a wireless communication system, such as e.g. the UMTS, GPRS, GSM or any other available or future wireless telecommunication system. The electronic equipment 1 further comprises an encoder 5 for encoding signals which have to be transmitted within the wireless communication system. Hereby, the encoder 5 is able to encode an audio signal and/or a video signal which then are multiplexed and transmitted within the wireless communication system.

When a communication between two electronic equipments is set up, in a second phase negotiation has to be accomplished as the electronic equipments have to agree in which directions data and/or information will be exchanged, what picture resolution to use and what type of encoding/decoding for video and audio that will be used during the video call has to be provided.

During a normal communication the voice signals, e.g. speech, from the user are received by the audio input means 2 and this first audio signal is submitted to the encoder 5. If a video conference has to be established, then also the image or video signals, e.g. the face of the user, the ambience or the like, are received by the video input means 4 and this first video signal is submitted to the decoder. Hereby, the first audio and/or the first video signal serve for keeping up a vocal, visual or combined vocal and visual communication.

During said wireless communication the user can select a file from the file system 3 he wants to transmit to the person he is communication with. The file is then submitted to a separation means 6 where the file is split into a second audio signal and a second video signal.

In an audio mixing means 13 the second audio signal is then interleaved with said first audio signal coming from the audio input means 2 and the interleaved audio signal is then submitted to the encoder 5.

The second video signal coming from the file and the first video signal coming from the video input means 4 are submitted to a selecting means 8. The selecting means 8 then selects which of the video signals is submitted to the encoder 5. If no second video signal from the file is present, then the selecting means 8 will select the first video signal deriving from the video input means. Otherwise if a second video signal from the file is submitted to the selecting means 8, then the selecting means 8 will select the second video signal and submit it to the encoder 5.

Thereby, according to the present invention, a separate audio or a separate video (without audio) or a combined audio and video (e.g. a video conference) communication can be kept up and at the same time information content can be transmitted. The information content hereby can consist of audio and/or video components including images, videos, sound, drawings, sketches, text, documents and the like. Thus, the information content may either be a pure audio file, a pure video file or a combined audio and video file. For example, during a pure audio communication, additional pure audio information can be transmitted. Alternatively, during a pure video communication, additional pure video information can be transmitted. Of course, all possible combinations of above possibilities can be realised. For example, during a video and audio communication (e.g. video conference), pure audio information or pure video information or combined additional audio and video information can be transmitted.

Fig. 2a shows a block diagram of parts of the electronic equipment 1 used during an ordinary video call. Hereby, the video input means 4 serves for taking pictures or videos from objects persons or the like. During an ordinary video call the images are already in the negotiated video transmit format and a buffer 9 can be filled with frames containing the images. The encoder 5 then reads from the buffer 9 and encodes the frames in order to submit them to the transceiver (not shown). Fig. 2b shows a block diagram of parts of the electronic equipment 1 used if a still image is replacing the camera images. When the user decides to transmit an image contained in the file system 3 then the file is loaded from the file system 3 and afterwards converted by a video converter 10. The video converter 10 for example converts the file for example into JPEG decompression. If the file has a different resolution then the transmitted video, i.e. QCIF 176x144 it has to be resized or cropped to the outgoing format. The video signal is then sent to the selecting means 8, which then will select the second video signal from the file to be transmitted to the buffer 9. The buffer 9 is then filled with frames containing the converted image. The encoder 5 then reads from the buffer 9 and encodes the frame to the negotiated format, i.e. H.263. The encoded image can then be multiplexed along with the audio signal and transmitted to the other electronic equipment 1 just like during a normal video call, that is the audio from the microphone is still transmitted along with the video containing the still image.

Fig. 3a shows a schematic block diagram of parts of the electronic equipment used for transmitting a simple audio signal received by the audio input means 2 of the electronic equipment 1. For example during a normal call or a video call the speech has to be transmitted. Therefore, the audio input means 2 integrated in the electronic equipment 1 submits the received voice signals to an AD converter 12 which then converts the analogue signal to digital signals. Afterwards ^"the signal is encoded by the encoder 5 and submitted to the transceiver (not shown).

Fig. 3b shows a schematic block diagram of parts of the electronic equipment 1 used for sharing a sound file during an ongoing communication. If the user wants to transmit an audio file then the audio file is fetched from the file system 3 and afterwards an audio converter 14 converts to file to an appropriate format. Afterwards the converted file is then mixed with the audio data received from the microphone 11 in the audio mixing means 13 by using audio channel mixing. The combined audio data is then encoded to the negotiated format by the encoder 5, i.e. AMR, multiplexed with the video and transmitted as during a standard video call.

Fig. 4 shows a schematic block diagram of parts of the electronic equipment used for transmitting a video clip. If the user decides to transmit a video clip then video is fetched from the file system 3 and a separation device 6 separates the audio signal from the video signal. As explained above the second audio signal derived from the file in the audio converter 14 is then converted to the appropriate format and afterwards is mixed in the audio mixing means 13 with the sampled first audio signal from the audio input means 2. After separating the second audio signal from the second video signal in the separation device 6 the second video signal is converted by a video converter 10 and afterwards submitted to the selecting means 8. The selecting means then selects the second or the first video signals and stores the selected signal in the buffer 9. The encoder 5 then encodes the video signal from the buffer 9 and the audio signal from the audio mixing means and submits them to the transceiver (not shown).

Fig. 5 shows a flow chart of the steps according to the process of the present invention. In step Sl, the user selects a file from the file system 3 he wants to transmit and the selection is accepted by the electronic equipment 1. In the next step S2 it is checked, whether the selected file contains an audio signal. If the file does not contain an audio signal then in step S3 the video signal is converted to the needed format. Then in step S4 it is checked, whether the converted video signal has the correct resolution. If the video signal does not have the correct resolution then the video is resized in step S 12. The process then goes back to step S4 where it is checked again whether the video has the correct resolution. If in step S4 it is decided, that the video has the correct resolution then video submitted to the selecting means in step S 5 and the selected video signal is buffered in the buffer 9 in step S6. In the next step S7 the buffered signals are encoded and in step S 8 they are transmitted by the transceiver.

If in step S2 it is determined that the file contains an audio signal then in step S9 the audio signal is separated from the video or image signal. The video or image signal is then processed according to the step S3 and following as explained above. The audio signal in step SlO is converted to the appropriate format and in step SIl it is mixed with the audio signal from the microphone in the audio mixing means 13. The audio signals are then encoded by the encoder 5 in step S6 and afterwards also transmitted in step S7 by the transceiver.

If during a voice call when only a first audio signal is submitted in order to keep up the communication a video content or a combined audio/video content is to be transmitted or submitted, then the electronic equipment 1 will usually not switch into the video mode for communication but continue the voice call. In the same way if during a video conference when a combined first audio and video signal is submitted to keep up the communication an audio content is submitted, then the electronic equipment will keep up the video conference and will usually not switch into a voice call. As an advantageous additional feature, an option for the user can be provided enabling to switch between different communication modes, e.g. between voice call and video conference in the above cases. Also, in an alternative or additional solution the electronic equipment can provide the option that a user can select the setting so that the equipment 1 switches automatically between the voice call and video conference mode i the above cases.

Fig. 6a to 6d show different display images during the process serving as an user interface. During a video call on the main display part 17 the face of the other person to which the user of the electronic equipment 1 is communicating can be shown. Further a control display part 18 may be provided where the user can see what at present the other person to which he is communicating is seeing, e.g. that is his own face during a video call or images or the like he wants to share or he is transmitting. The display further comprises a main menu 16 through which the user can scroll and the actual selectable menu is high-lighted in a form for example enlarged or the colour is changed. In the present example the user can select between menus A, B, C, D, or E. In the present example the menu B actually can be selected. The display may further comprise information lines 19A and 19B in order to inform the user on the actual selected menu or file. Selectable menus may be the feature of a video call or a normal call, sharing pictures or text, sharing videos, sharing sounds or playing interactive video games with other persons. If, for example, the user has selected as shown in Fig. 6b the menu for sharing pictures then in a sub-menu 20a the pictures contained in the file system 3 are shown. By scrolling through the pictures the actual selectable picture again is high-lighted, which in the present example is picture 4. Fig. 6c shows the situation where the picture 4 is selected and has been transmitted to the other person. In the control display part 18 the user can see what the other person actually is viewing, that is in the present example the picture 4, and in the main display part 17 at the same time he can see the face of the other person and the reaction of the other person on the picture he has just received.

Another example is shown in Fig. 6d, where instead of sharing pictures the user has selected the main menu E, which in the present example is the option for playing games. After selecting the main menu E the sub-menu 20b again is shown with different games the user can select. In the present example the user has selected the game chess and therefore a chess board 21 is shown in the main display part 17. In this case it may be possible to show the face of the other person in the control display part 18 in order to play the game chess and at the same time see the reactions of the person he is playing with.

A possibility of scrolling through the menu may be a joy-stick or a special key pad having direction keys and a selection key for scrolling through the menus and selecting the actual high-lighted menu. In a second embodiment of the present invention it is possible for the user to keep up communication via the wireless communication system and at the same time to play a game like in a face to face environment. Fig. 7 shows a block diagram showing schematically the elements of the system of the electronic equipment according to the second embodiment. Hereby a first signal, which may either comprises audio and/or video data and allows communication within the wireless communication system is generated by the first signal control unit 25. The first signal control unit hereby may comprise alle the components as shown in Fig. 4 which are necessary for generating an audio and/or video signal. This first signal is then submitted to an encoder 5. A second signal control unit 26 generates a second signal which comprises all the information needed for playing a game. The first signal, which has been encoded by the encoder 5, and the second signal are then submitted to a multiplexer 27, which multiplexes both signals on a data stream to be sent via the wireless communication system.

Thereby it is possible to keep up communication, e.g. a video call, and to see and/or hear the other party and at the same time to play a game. This system enables the user of an electronic equipment to play games even over long distances and the user is not constricted to use short distance transmissions such as e.g. Bluetooth.

The second signal used for the transmission of information relating to the game hereby is exchanged on a client level as text strings as will be explained in the following. To be able to play a game with the other party during a video call game data need to be exchanged. The standard for video telephony in electronic equipments is 3GPP's 3G324m standard which inlcudes the control protocol H.245. In this control protocol there is room for the two devices participating in the video call to communicate on a client level. This is achieved by usind the Userlnputlndication filed, which allows the clients to exchange text strings. It is to be noted, that the present invention is not limited to the h.245 protocol but can be adapted by any protocol that allows the exchange of text strings or other type of data and/or information during an ongoing audio and/or video communication via the wireless communication system.

What type of data that has to be communicated is illustrated by the following Tic-Tac- Toe example.

The condition is an ordinary ongoing audio and/or video communication via the wireless communication system with transmission in both directions. To be able to play a game with the other party both user have to have the same game installed. In order to detect all playable games a synchronisation process is started after the communication via the wireless communication system has been established. When successfully performed the user can select one of the games found during the game synchronisation.

The synchronisation data are illustrated by the following example: The master client, which has been determined during the h.245 negotiation, initiates the synchronisation.

It transmits the following message: <F2F_GameSynch> <gameID>G876H2</gameID>

The slave client receives this message and parses out the important game ID, these numbers are compared to the game DDs for the locally stored games.

In this example four of the fives games are available on both sides. The client replies. <F2F_GameSynchReply>

<F2F_GameSynchReply>

If no coexistence games are found and empty reply is sent back to the master. <F2F_GameSynchReply> <F2F_GameSynchReply>

If the master does not receive any reply at all, the client makes the conclusion that gaming is not supported by the other client.

The client can now show the playable games to the user and he can select one of them. After a game is selected an invitation is transmitted to the other party. <F2F_GameInvitation>

<gameID>BQ836A</gameID> <F2F Gamelnvitation> When the other party has accepted the invitation the initiating client is informed. <F2F_GameInvitation_Accept>

If the user on the remote side does not want to play the game he rejects the invitation and a reject reply is transmitted. <F2F_GameInvitation_Reject>

The game session is ended and the communication via the wireless communication system continues as normal.

When the initiating client receives the accept reply it starts the game and when the game is started a message to inform the other party is transmitted. <F2F_GameStarted> <gameID>BQ836A</gameID>

<F2F_GameStarted>

When the game is started at the remote side the client informs the master that the game is started. <F2F_GameStarted_OK>

Now the game is up and running on both clients.

The initator of the game is the master of the session. The master's game application is the one which controls the game. Different games need to exchange different game data. The following data can for instance be exchanged during a Tic-Tac-Toe game. When the user of the master client makes his move, the data is transmitted to the slave client.

<F2F_TicTacToeData> <gameBoard>

</gameBoard> <turn>slave</turn> <gameθver>false</gameθver> <F2F TicTacToeData> In this example the master's user has made his move in the third row in the third space with an X.

When the slave has received and drawn the new board it replies. <F2F_TicTacToeData_Received>

The master waits for the slave's user to make his move. <F2F_TicTacToeData>

<turn>master</turn> <gameθver>false</gameθver> <F2F_TicTacToeData>

In this example the slave's user has made his move in the second row in the fourth space with an O.

The game then continues until it is ended by any of the users or until one user wins.

With the present invention the users of mobile equipments can either create a situation like in a conference where while keeping eye contact with each other they can discuss on sketches, diagrams or the like or the users can play games and share pictures at the same time having the impression of a normal face to face conversation. This solution gives the user the possibility to talk, look at pictures, play games and experience each other's face expressions in real time. The user can spontaneously select content while still having eye contact. It is very easy to select content in real time and at the same time have control over what the user is actually showing to the other party. The solution further is comfortable with existing video conferencing, applications, and infrastructure.

Claims

SONY ERICSSON MOBILE COMMUNICATIONS ABPS04 0153EPP30365WOCLAIMS

1. Electronic equipment for a wireless communication system enabling communication within the wireless communication system by transmitting a first audio signal and/or a first video signal, and enabling transmission of information content within the wireless communication system by transmitting a second audio signal and/or a second video signal, comprising an encoder (5) for encoding audio and/or video signals to be transmitted within the wireless communication system and an audio mixing means (13) for interleaving said second audio signal with said first audio signal and submitting the interleaved audio signal to the encoder (5).

2. Electronic equipment according to claim 1, characterised in that a file system (3) contains a file consisting of said second audio signal and/or said second video signal.

3. Electronic equipment according to any of claims 1 or 2, characterised in that an audio converter (14) converts said second audio signal to a predefined format.

4. Electronic equipment according to any of claims 1 to 3, characterised by a separation means (6) for separating the second video signal from the second audio signal.

5. Electronic equipment according to any of claims 1 to 4, characterised by a selecting means (8) for selecting said second video signal or said first video signal and submitting the selected video signal to the encoder (5).

6. Electronic equipment according to claim 5, characterised in that the selecting means (8) selects the first video signal only when no second video signal is present.

7. Electronic equipment according to any of claims 1 to 6, characterised in that a video converter (10) converts said second video signal to a predefined format.

8. Method for transmitting information in a wireless communication system enabling communication within the wireless communication system by transmitting a first audio signal and/or a first video signal, and enabling transmission of information content within the wireless communication system by transmitting a second audio signal and/or a second video signal comprising the steps of encoding audio and/or video signals to be transmitted within the wireless communication system by an encoder (5) and, in case of transmitting said first and second audio signal, interleaving said second audio signal with said first audio signal and submitting the interleaved audio signal to the encoder (5).

9: Method according to claim 8, characterised by storing a file consisting of said second audio signal and/or said second video signal in a file system (3).

10. Method according to claim 8 or 9, characterised in converting said second audio signal to a predefined format.

11. Method according to any of claims 8 to 10, characterised by separating the second video signal from the second audio signal.

12. Method according to any of claims 8 to 12, characterised in selecting said second video signal or said first video signal and submitting the selected video signal to the encoder (5).

13. Method according to claim 12, characterised in selecting the first video signal only when no second video signal is present.

14. Method according to any of claims 8 to 13, characterised in converting said second video signal to a predefined format.

15. Electronic equipment for a wireless communication system enabling communication within the wireless communication system by transmitting a first signal comprising an audio and/or video signal and enabling transmission of information content within the wireless communication system by transmitting a second signal, comprising an encoder (5) for encoding the first signal to be transmitted within the wireless communication system and a multiplexer (27) for multiplexing the first signal received from the encoder (5) with the second signal.

16. Electronic equipment according to claim 15, characterised in that the second signal is transmitted on a client level.

17. Electronic equipment according to claim 15 or 16, characterised in that the second signal are text strings.

18. Method for transmitting information in a wireless communication system enabling communication within the wireless communication system by transmitting a first signal comprising an audio and/or video signal and enabling transmission of information content within the wireless communication system by transmitting a second signal, comprising the steps of encoding the first signal to be transmitted within the wireless communication system by an encoder (5) and multiplexing the first signal received from the encoder (5) with the second signal by a multiplexer (27).

19. Method according to claim 18, characterised in transmitting the second signal on a client level.

20. Method according to claim 18 or 19, characterised in providing text strings as second signal.