JP2018515979A - Communication processing method and electronic apparatus - Google Patents

Abstract

The present invention discloses a communication processing method and an electronic device, wherein the method controls the first electronic device to be in a first communication state, and in the first communication state, the first electronic device has at least one In response to the first operation, the second electronic device is controlled to switch from the first communication state to the second communication state in response to the first operation, and in the second communication state, the at least one first Holding an interaction of audio data with two electronic devices, sharing a video frame with at least one second electronic device, and switching from the second communication state to the first communication state in response to a second operation Controlling. [Selection] Figure 1

Description

  The present invention relates to a communication management technique in the field of information processing, and more particularly to a communication processing method and an electronic apparatus.

  With the development of technology, in the current voice interaction products based on the VoIP (Voice over Internet Protocol), all participants in the call simply make a voice call or just a video call, and the basics In addition to closing the video call, this call is also closed. Such a video communication method has the following two drawbacks. First, one side cannot share video unilaterally. For example, if one of the two invites the other party to a video call, as long as the other party answers the video call, their camera will be automatically activated and begin collecting video frames. At that time, if the other party is not convenient to respond, the invitation is usually rejected directly and the call is terminated. But in such a case, the inviting party's intention may not want to watch the other party's video at the same time, but simply want to share their video with the other party. Second, there is no initiative to control whether or not to implement video during a call. For example, you cannot control when to open and close your video at any time, that is, when you start the video, call participants can watch their video, but if not, Call participants cannot see their video when they end the video.

  For this reason, in the prior art, call participants cannot actively control the timing of sharing video frames, cannot reduce the use of server processing resources, and smooth both calls (fluency). Degree) can not be kept.

  In view of this, an embodiment of the present invention is to provide a communication processing method and an electronic device, and to solve at least the above-described problems in the prior art.

An embodiment of the present invention provides a communication processing method applied to a first electronic device, the method comprising:
Controlling the first electronic device to be in a first communication state, and in the first communication state, the first electronic device interacts with audio data with at least one second electronic device;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and Sharing a video frame with at least one second electronic device;
In response to a second operation, controlling to switch from the second communication state to the first communication state;
including.

Embodiments of the present invention further provide an electronic device,
The first electronic device is controlled to enter a first communication state, and in the first communication state, the first electronic device interacts with voice data with at least one second electronic device and responds to a second operation. A first processing unit configured to control to switch from the second communication state to the first communication state;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and A second processing unit configured to share a video frame with at least one second electronic device;
including.

  According to the communication processing method and the electronic device provided in the embodiment of the present invention, when the user is in the first communication state, only the voice data can be interacted, and the electronic device is operated in response to the first operation. Control is made to switch to the two communication state, and not only the transmission of audio data, but also the currently collected video frame is shared, and in response to the second operation, the electronic device is again switched to the first communication state. . In this way, in the process of voice communication interaction, the user can freely control the timing of video frame interaction, and the server used to continuously transfer the video frame and voice data during communication. Processing resources can be reduced and the smoothness of both calls can be maintained.

It is a figure which shows the communication processing method flow in the Example of this invention. It is a figure which shows the connection scene in the Example of this invention. It is a figure which shows the interface of the video frame transmission in the Example of this invention. It is a figure which shows the transmission and analysis flow of a video frame in the Example of this invention. It is a figure which shows the flow of interaction of the video frame in the Example of this invention. It is a figure which shows the flow which complete | finishes the interaction of the video frame in the Example of this invention. It is a figure which shows the operation interface which shares a video frame in the Example of this invention with two people. It is a figure which shows the flow which shares the video frame in the Example of this invention with several people. It is a figure which shows the operation interface which shares the video frame in the Example of this invention with several people. It is a 1st figure which shows the structure of the electronic device in the Example of this invention. It is a 2nd figure which shows the structure of the electronic device in the Example of this invention. It is a 3rd figure which shows the structure of the electronic device in the Example of this invention.

  In the following, embodiments of the present invention will be described in more detail with reference to the drawings and specific embodiments.

Example 1
An embodiment of the present invention provides a communication processing method, which is applied to a first electronic device, and the method includes steps 11 to 13 as shown in FIG.

  In step 11, the first electronic device is controlled to enter a first communication state, and in the first communication state, the first electronic device interacts with at least one second electronic device in audio data.

  In step 12, in response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, voice data interaction with the at least one second electronic device is performed. Hold and share the video frame with at least one second electronic device.

  In step 13, control is performed to switch from the second communication state to the first communication state in response to the second operation.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  Preferably, before performing step 11, the first electronic device can establish a communication connection with at least one second electronic device, for example, as shown in FIG. Establishes a communication connection to the second electronic device 221 and the second electronic device 222.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

In the above step 12, in response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, audio data with the at least one second electronic device And sharing a video frame with at least one second electronic device,
Controlling to switch from the first communication state to the second communication state in response to the first operation, and maintaining the interaction of the audio data with the at least one second electronic device in the second communication state; ,
Collecting at least one video frame and transmitting the at least one video frame to the at least one second electronic device;
including.

  Here, the at least one video frame may be at least one video frame obtained by combining at least one image information based on the collection time.

  Here, the collected at least one video frame may be a frame that is periodically collected via the camera when the camera is activated. Here, the period may be set according to the performance of the camera. For example, the image information may be collected once every 2 ms and set as one video frame.

  Preferably, in the present embodiment, the first operation or the second operation is also detected before responding to the first operation or the second operation, and is specifically as follows.

  Prior to responding to the first operation, the method further determines that the first operation has been detected if it is detected that the length of time to operate on the first touch operation area is greater than a first threshold. Including that.

  Prior to responding to the second operation, the method further includes determining that the second operation has been detected if it is detected that the touch operation on the first touch operation area has been canceled.

  Here, the first touch operation area may be a touch point for detecting a touch operation set in accordance with an actual situation or a preset area for detecting a touch operation.

  The touch point may include at least the abscissa and ordinate of the touch point, and the preset area includes at least the abscissa of the lower left corner and the abscissa of the upper right corner of the area.

  The first touch operation area may be expressed as a virtual button 31 displayed on the first electronic device as shown in FIG.

  The length of the operation time may be the length of time that the user presses the first touch operation area. In other words, when the user presses the first touch operation area, the camera is activated. Until at least one video frame is collected through the camera and transmitted to at least one second electronic device until it is detected that the user has canceled the pressing operation of the first touch operation area.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the video frames of the currently collected image file, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, at the time of voice communication interaction, the user can freely control the timing of the video frame interaction of the image file, so that the video frame and voice data of the image file at the time of communication can be maintained continuously. The processing resources of the server used to transfer to the server can be reduced, and the smoothness of both calls can be maintained.

Example 2
An embodiment of the present invention provides a communication processing method and applies a first electronic device, and the method includes steps 11 to 13 as shown in FIG.

  In step 11, the first electronic device is controlled to enter a first communication state, and in the first communication state, the first electronic device interacts with at least one second electronic device in audio data.

  In step 12, in response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, voice data interaction with the at least one second electronic device is performed. Hold and share the video frame with at least one second electronic device.

  In step 13, control is performed to switch from the second communication state to the first communication state in response to the second operation.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  Preferably, before performing step 11, the first electronic device can establish a communication connection with at least one second electronic device, for example, as shown in FIG. Establishes a communication connection to the second electronic device 221 and the second electronic device 222.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

The difference between the present embodiment and the first embodiment is that, in step 12, the control is performed to switch from the first communication state to the second communication state in response to the first operation, and in the second communication state, Holding an audio data interaction with the at least one second electronic device and sharing a video frame with the at least one second electronic device,
Receiving at least one video frame transmitted from at least one second electronic device;
Analyzing the at least one video frame transmitted from at least one second electronic device;
including.

  Here, analyzing the video frame may be that the first electronic device analyzes the video frame by a video analysis function to obtain an image corresponding to one or a plurality of video frames.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the currently collected video frame, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, the user can freely control the timing of video frame interaction at the time of voice communication interaction, thereby being used to continuously transfer video frame and voice data during communication. Server processing resources can be reduced, and the smoothness of both calls can be maintained.

Example 3
An embodiment of the present invention provides a communication processing method and applies a first electronic device, and the method includes steps 11 to 13 as shown in FIG.

  In step 11, the first electronic device is controlled to enter a first communication state, and in the first communication state, the first electronic device interacts with at least one second electronic device in audio data.

  In step 12, in response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, voice data interaction with the at least one second electronic device is performed. Hold and share the video frame with at least one second electronic device.

  In step 13, control is performed to switch from the second communication state to the first communication state in response to the second operation.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  Preferably, before performing step 11, the first electronic device can establish a communication connection with at least one second electronic device, for example, as shown in FIG. Establishes a communication connection to the second electronic device 221 and the second electronic device 222.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

  The difference from the above two embodiments is that this embodiment provides a scene in which the first electronic device not only collects and transmits video frames but also receives video frames. Can also be displayed. Specifically, it is as follows.

In step 12, in response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, audio data with the at least one second electronic device is Holding the interaction and sharing the video frame with at least one second electronic device,
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and Collecting at least one video frame and transmitting the at least one video frame to the at least one second electronic device;
Receiving at least one video frame transmitted from at least one second electronic device and analyzing at least one video frame transmitted from the second electronic device;
The video frame can be rendered and displayed after analyzing the video frame transmitted from the second electronic device.

Preferably, in this embodiment, transmitting the video frame comprises
Sending a video frame share request to the signaling server;
Sending the video frame share request to at least one second electronic device by the signaling server;
Activating the camera, collecting at least one image information, coding the image information to obtain a video frame;
Sending a video frame to a data server;
Transferring the video frame from the data server to the at least one second electronic device;
Wherein the signaling server may be a VOIP signaling server or a mobile cellular network signaling server,
The data server may be a VOIP data server or a data server in a mobile cellular network.

  Preferably, in the present embodiment, the corresponding time stamp is added to the video frame or the audio data, respectively, thereby ensuring the synchronization at the receiving end of the transmitted video frame and audio data. When a video frame is collected, a time stamp is added to the audio data, and when a video frame is collected, a time stamp corresponding to the collected video frame is added. Here, the time stamp may be a time when audio data or a video frame is collected.

Analyzing the at least one video frame transmitted from the at least one second electronic device;
Analyzing audio and time stamps in audio data, analyzing video frames and corresponding time stamps,
Controlling to output audio data and video frames based on the parsed timestamps;
including.

  In the present embodiment, the method of controlling to output audio data and video frames based on the analyzed time stamp may be to output audio data and video frames having the same time stamp at the same time. Good.

  Preferably, in the present embodiment, the first operation or the second operation is also detected before responding to the first operation or the second operation, and is specifically as follows.

  Prior to responding to the first operation, the method further determines that the first operation has been detected if it is detected that the length of time to operate on the first touch operation area is greater than a first threshold. Including that.

  Prior to responding to the second operation, the method further includes determining that the second operation has been detected if it is detected that the touch operation on the first touch operation area has been canceled.

  Here, the first touch operation area may be a touch point for detecting a touch operation set in accordance with an actual situation or a preset area for detecting a touch operation. The touch point can include at least the abscissa and ordinate of the touch point, and the preset area includes at least the abscissa of the lower left corner and the abscissa of the upper right corner of the area.

  The first touch operation area may be expressed as a virtual button 31 displayed on the first electronic device as shown in FIG. In this embodiment, a function triggered by pressing the first touch operation area may be set as a “glance” operation.

  The length of the operation time may be the length of time that the user presses the first touch operation area. In other words, when the user presses the first touch operation area, the camera is activated. And at least one video frame is collected through the camera until it is detected that the user cancels the pressing operation of the first touch operation area.

Furthermore, in this embodiment, sharing a video frame with the at least one second electronic device includes:
Transmitting a video frame to all second electronic devices of the at least one second electronic device;
Or
Selecting a target second electronic device from among the at least one second electronic device and transmitting the video frame to the target second electronic device;
including.

  In other words, the first electronic device switches to the second communication state after the first communication state in which the voice data interacts with a plurality of electronic devices, and transmits the collected images to all the second electronic devices. Alternatively, one target device may be selected from a plurality of electronic devices, and an image may be transmitted to the selected target device.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the currently collected video frame, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, the user can freely control the timing of video frame interaction at the time of voice communication interaction, thereby being used to continuously transfer video frame and voice data during communication. Server processing resources can be reduced, and the smoothness of both calls can be maintained.

Example 4
In this embodiment, the video frame transmission operation will be described by taking the case where the signaling server and the data server are the VOIP signaling server and the VOIP data server as an example.

  As shown in FIG. 4, a simple flow chart of real-time video is shown, in which the VOIP signaling server mainly controls and synchronizes the state in the call process, and the data server mainly A flow for performing transmission (transparent transmission) on audio data and performing real-time video between the two devices M1 and M2 will be described below, where the device M1 is a first electronic device and the device M2 is a first device. 2 electronic devices.

  1. M1 activates the camera device and collects video frames.

  2. The video codec engine performs an encoding process on the video frames collected from M1.

  3. After encoding the video frame, M1 packetizes the network and sends it to the VOIP data server using the UDP method.

  4). The VOIP data server transparently transmits the video frame to the device M2.

  5. The device M2 receives and decompresses the video frame.

  6). The video codec engine performs a decoding process on the video frame.

  7). Device M2 renders and displays the decoded video frame.

  Preferably, in this embodiment, an example of an interaction operation between audio data and video frame between two electronic devices and an example of an interaction operation between audio data and video frame between one-to-many electronic devices are respectively provided. ing.

  Example 1 “Glance” interaction between two people is a way of playing based on real-time video technology, where one button (not limited to a similar format) is provided and the button is continuously pressed to the other party Real-time video transmission starts, and when the button is released, the real-time video transmission to the other party is completed. As an example, the flow of “glance” interaction between user A and user B includes four types of interaction cases. As shown in FIG. 5, “glance” between two people is based on real-time video technology. When one button is provided, and the user A continuously presses the first touch operation area provided by the display interface provided by the apparatus, that is, one virtual button, The operation is triggered, in other words, the user B, i.e. the second electronic device, is triggered to initiate the operation of transmitting the collected video frames in real time. On the other hand, when the button is released, the “glance” operation is completed and the transmission of the real-time video to the other party is terminated. Specifically, the following cases are included.

  Case 1 is a one-way video share.

  1) -2) User A continuously presses the button, sends a video share request to the VOIP signaling server, activates the camera, and starts collecting video. Here, the button may be a button set according to an actual situation.

  3) Notify user B from the VOIP signaling server that user A has requested video sharing.

  4) to 5) User A transmits the collected real-time video frame to the VOIP data server, and the VOIP data server transmits the video frame to user B transparently.

  6) After user B receives the real-time video frame from user A, it first checks whether the notification of 3) is received, and if received, renders and displays the video frame.

  Case 2 is an interactive video share.

  7) -8) On Case 1, User B also sends a video share request to the VOIP signaling server, activates the camera, and collects video frames.

  9) The user A is notified from the VOIP signaling server that the user B has started “glance”.

  10) to 11) User B transmits the collected real-time video frame to the VOIP data server, and the VOIP data server transmits the video frame to user A transparently.

  12) After user A receives the real-time video frame from user B, it first checks whether the notification of 9) is received, and if received, renders and displays the video frame. In this way, the flow of simultaneously sharing the real-time video between the users A and B is completed.

  Case 3 is that one end of video sharing.

  13) -14) User A releases the button, sends a video share end request to the VOIP signaling server, stops collecting video frames of the camera, and at the same time, the VOIP data server receives the video frame from user A. No more receiving.

  15) Notify user B from the VOIP signaling server that user A has finished sharing the video.

  16) User B does not receive any more video frames from user A, and after receiving the notification of 15), switches to one-way video sharing.

  Case 4 is that both sides end video sharing.

  17) -18) On case 3, user B also releases the button, sends a video share end request to the VOIP signaling server, stops collecting video frames from the camera, and switches to normal voice call interaction. At the same time, the VOIP data server does not receive any more video frames from user B.

  19) The user VO notifies the user A that the video sharing is completed from the VOIP signaling server.

  20) User A will not receive any more video frames from user B, and will switch to normal voice call interaction after receiving the notification of 19).

  On the premise of interaction of audio data between two electronic devices, the interaction of audio data and video frame provided by this embodiment is as shown in FIGS.

  The interface 61 in FIG. 6 is a scene in which the user A interacts with the second electronic device of the user B via the first electronic device in the first communication state, and the user presses the “shoot” button 611 for a long time. This is a virtual button of “Request video sharing”.

  Then, transmission of one video frame is started, and the operation interface of the first electronic device of the user A enters the interface 62, and the currently collected video frame is displayed on the interface 62. A character “shared” is also displayed, and the virtual button 611 is also moved in accordance with the movement of the finger on the screen.

  When the user B on the receiving side receives the frame video from the user A, it is possible to make one simple ring tone, and the video frame screen transmitted from the transmitting side is displayed on the display interface 63 of the user B. The screen is displayed, and at the same time, the length of the current call time and the text “the other party is sharing in real time” are displayed. The user B on the receiving side activates the “video sharing” function of the second electronic device by pressing the button 631 in the corresponding interface 63 using the preset button 631 as a touch operation area, and interactively. You can enter video sharing.

  As the sharing of both videos, after entering into the sharing of interactive video, both interfaces are as shown by the interface 64 and the interface 65 in FIG. In the interface 64 and the interface 65, the circular area 641 and the circular area 651 display the contents photographed by the own camera, and at the same time, the circular area is moved as the finger moves on the screen, and the full screen area is displayed. Displays the other party's video frame screen. In this scene, if one of the buttons is released first, it is possible to return to the interface 63 if it is released and return to the interface 62 if it is not released. When both are released, both return to the interface 61.

Example 2
This example further provides a “glance” interaction between multiple people (three and more) as shown in FIG. 8, which is also a way of playing based on real-time video technology, but between two people Unlike the format, only one person can start sharing video at the same time, and when multiple people make a video sharing request at the same time, the VOIP signaling server makes a determination and the video sharing request is first sent to the VOIP signaling. If the user has reached the server, the success of the video sharing request is sent back, the video sharing request failure is sent back to the other person, and the other person wants to share the video successfully. You can wait for the sharing user to finish before you can start sharing successfully (where multiple videos share contention It may be understood as). The following two interaction cases are included, taking as an example the video share interaction flow between a plurality of users A1 to An.

  Case 1 is the start of video sharing.

  1) Some of the users A1 to An simultaneously transmit a video sharing request to the multi-person VOIP signaling server.

  2) The multi-person VOIP signaling server makes a determination based on the order of front and back of the received video sharing request, returns the success of video sharing to the user A1 who has received the request first, and sends video to other users. Returns the share request failure.

  3) The multi-person VOIP signaling server notifies other users that the user A1 has started “glance”.

  4) User A1 activates the camera and starts collecting video frames.

  5-6) After the user A1 transmits the collected real-time video frame to the multi-person VOIP data server, the VOIP data server transparently transmits the video frame to other users.

  7) After receiving the real-time video frame from user A1, the other users first check whether the notification of 3) has been received and, if already received, render and display the video frame. .

  Case 2 is the end of video sharing.

  8-9) On case 1, user A1 releases the button, sends a share termination request to the multi-person VOIP signaling server, and at the same time, the camera stops collecting video frames and enters a normal multi-person voice call state. At the same time, the VOIP data server will not receive any more video frames from user A1.

  10) The VOIP signaling server notifies the other users that the user A1 has finished “glance”.

  11) The other users do not receive any more video frames from user A1, and after receiving the notification of 10), the user switches to the normal multi-person voice call interaction.

  As shown in FIG. 9, the explanation regarding each effect diagram is as follows.

  The multi-person call interface 91 is an entrance for sharing the video of a plurality of persons. In the interface 91, as an operation for triggering the first touch operation area, a long press of the “shoot” 911 button, that is, the video sharing function is activated. However, it is possible to make one simple ringtone at the same time. By activating the video sharing function, the interface 92 is entered, and after entering the interface 92, there is a process of “fight”. When the “fight” is successful, the process is held in the interface 92. Return to or display the interface 93 to receive “glance” data of other people.

  In the interface 92, that is, after entering the video sharing function, the content captured by the camera itself is displayed on the full screen, and the length of the current call time and the text “Real time sharing” are displayed. The button is moved as the finger moves on the screen, and when the button is released, the interface 91 is returned to.

  In the interface 93, when a plurality of video frames share a video frame, a device that does not share an image can also make a simple ringtone when it sends a video frame from the invitee, and invites The video frame screen sent from the side is displayed on the full screen, and at the same time the length of the current call time and the text “who is sharing in real time” are displayed.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the video frames of the currently collected image file, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, at the time of voice communication interaction, the user can freely control the timing of the video frame interaction of the image file, so that the video frame and voice data of the image file at the time of communication can be maintained continuously. The processing resources of the server used to transfer to the server can be reduced, and the smoothness of both calls can be maintained.

Example 4
An embodiment of the present invention provides an electronic device, as shown in FIG.
The first electronic device is controlled to be in a first communication state. In the first communication state, the first electronic device interacts with audio data with at least one second electronic device, and performs a second operation. In response, a first processing unit 1001 configured to control to switch from the second communication state to the first communication state;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and A second processing unit 1002 configured to share a video frame with at least one second electronic device;
including.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  Preferably, the first electronic device can establish a communication connection with at least one second electronic device. For example, as shown in FIG. 2, the first electronic device 21 is connected to the second electronic device 221 and the second electronic device 221. 2 Establish communication connection to electronic device 222.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

  The electronic device further includes an image collection unit 1003 for collecting at least one image information.

  Accordingly, the second processing unit 1002 controls to switch from the first communication state to the second communication state specifically in response to the first operation, and in the second communication state, the at least one An audio data interaction with a second electronic device is maintained, a first video frame is generated using the collected at least one image information, and the first video frame is transmitted to the at least one second electronic device. Used to transmit.

  Here, collecting the at least one image information may be to obtain at least one image information by starting the camera and periodically collecting the image information by the camera. Here, the period can be set based on the performance of the camera. For example, the period may be set so as to collect image information once every 10 ms.

  Alternatively, collecting the at least one image information may be to activate a camera to capture a picture and collect only one image, and the image is a video frame.

  Preferably, the electronic device further includes a touch detection unit 1004, and when it is detected that the length of time for operating the first touch operation area is larger than the first threshold, it is determined that the first operation is detected. When it is detected that the touch operation on the first touch operation area is cancelled, it is used to determine that the second operation has been detected.

  Here, the first touch operation area may be a touch point for detecting a touch operation set in accordance with an actual situation or a preset area for detecting a touch operation. Here, the touch point may include at least the abscissa and ordinate of the touch point, and the preset area includes at least the abscissa of the lower left corner and the abscissa of the upper right corner of the area. including.

  The first touch operation area may be expressed as a virtual button 31 displayed on the first electronic device as shown in FIG.

  The length of the operation time may be the length of time that the user presses the first touch operation area. In other words, when the user presses the first touch operation area, the camera is activated. Until at least one piece of image information is collected through the camera until the user detects that the pressing operation of the first touch operation area is canceled, and the at least one piece of image information is used as a video frame.

  As shown in FIG. 4, a simple implementation flowchart for transmitting a video frame in the present embodiment is shown. Here, the VOIP signaling server mainly controls and synchronizes the state in the call process, and the signaling server , The interaction method for controlling and synchronizing the call process is to use TCP, and the data server mainly transfers (transparent transmission) to video or audio data, and transfers the data according to the UDP protocol. .

  Hereinafter, a process of performing real-time video between two devices, the first electronic device and the second electronic device, will be described.

  The first electronic device activates the camera device, collects image information, and the video codec engine performs encoding processing on the image information, packetizes the encoded video frame into a network, and uses a UDP data server. And the data server may be a VOIP data server.

  Transparent transmission from the VOIP data server to the second electronic device.

  The second electronic device receives and decompresses the video frame, and the video codec engine performs a decoding process on the video frame, and renders and displays the decoded video frame.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the video frames of the currently collected image file, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, at the time of voice communication interaction, the user can freely control the timing of the video frame interaction of the image file, so that the video frame and voice data of the image file at the time of communication can be maintained continuously. The processing resources of the server used to transfer to the server can be reduced, and the smoothness of both calls can be maintained.

Example 5
An embodiment of the present invention provides an electronic device, as shown in FIG.
The first electronic device is controlled to be in a first communication state. In the first communication state, the first electronic device interacts with audio data with at least one second electronic device, and performs a second operation. In response, a first processing unit 1101 configured to control to switch from the second communication state to the first communication state;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and A second processing unit 1102 configured to share a video frame with at least one second electronic device;
including.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

  The difference between the present embodiment and the first embodiment is that the electronic device further analyzes the second video frame to obtain at least one image information and displays the at least one image information, respectively. And an analysis unit 1103.

  The second processing unit 1102 is further used to receive a second video frame transmitted from at least one second electronic device.

  Here, in analyzing the second video frame, the first electronic device obtains one or more pieces of image information by analyzing the video frame using an image analysis function.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the currently collected video frame, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, the user can freely control the timing of video frame interaction at the time of voice communication interaction, thereby being used to continuously transfer video frame and voice data during communication. Server processing resources can be reduced, and the smoothness of both calls can be maintained.

Example 6
An embodiment of the present invention provides an electronic device, as shown in FIG.
The first electronic device is controlled to be in a first communication state. In the first communication state, the first electronic device interacts with audio data with at least one second electronic device, and performs a second operation. In response to the first processing unit 1201 configured to control to switch from the second communication state to the first communication state, and in response to the first operation, switch from the first communication state to the second communication state. A second control unit configured to hold an interaction of audio data with the at least one second electronic device and share a video frame with the at least one second electronic device in the second communication state. A processing unit 1202;
including.

  Here, the first electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  The second electronic device may be a device such as a smartphone, a tablet PC, or a laptop computer.

  In this embodiment, the communication connection establishment method between the first electronic device and the second electronic device may be established via a mobile cellular network or established via an IP network. Good.

  In this embodiment, before the first electronic device interacts with audio data with at least one second electronic device, the first electronic device can further activate an audio collecting unit to collect audio. The voice data is obtained by coding using the collected voice. Here, the voice collecting unit may be a microphone.

  This embodiment provides a scene, in which the first electronic device can not only collect and transmit video frames, but can also receive and display video frames. Specifically, the electronic device further includes an image collection unit 1203 for collecting at least one image information, and accordingly, the second processing unit 1202 specifically responds to the first operation, Control is performed so as to switch from the first communication state to the second communication state, and in the second communication state, it is used to hold an interaction of audio data with the at least one second electronic device. A first video frame is generated using the collected at least one image information, and the first video frame is transmitted to the at least one second electronic device.

  The electronic apparatus further includes an analysis unit 1205 for analyzing the second video frame to obtain at least one image information and displaying the at least one image information.

  The second processing unit 1202 is further used to receive a second video frame transmitted from at least one second electronic device.

Preferably, in the present embodiment, transmitting the video frame means that the second processing unit 1202 transmits a video frame share request to the signaling server, and at least one video frame share request by the signaling server. Transmitting to a second electronic device, activating a camera, collecting at least one image information, encoding the image information to obtain a video frame, and transmitting the video frame to a data server; ,
Transferring the video frame from the data server to the at least one second electronic device;
Wherein the signaling server may be a VOIP signaling server or a mobile cellular network signaling server,
The data server may be a VOIP data server or a data server in a mobile cellular network.

  Here, collecting the at least one image information may be to obtain at least one image information by starting the camera and periodically collecting the image information by the camera. Here, the period can be set based on the performance of the camera. For example, the period may be set so as to collect image information once every 10 ms.

  Alternatively, collecting the at least one image information may be to activate a camera to capture a picture and collect only one image, and the image is a video frame.

  The electronic device further includes a touch detection unit 1204, and when it is detected that the length of time for operating the first touch operation area is greater than a first threshold, it is determined that the first operation has been detected, When it is detected that the touch operation on the first touch operation area is canceled, it is used to confirm that the second operation is detected.

  Here, the first touch operation area may be a touch point for detecting a touch operation set in accordance with an actual situation or a preset area for detecting a touch operation. Here, the touch point may include at least the abscissa and ordinate of the touch point, and the preset area includes at least the abscissa of the lower left corner and the abscissa of the upper right corner of the area. including.

  The first touch operation area may be expressed as a virtual button 31 displayed on the first electronic device as shown in FIG.

  The length of the operation time may be the length of time that the user presses the first touch operation area. In other words, when the user presses the first touch operation area, the camera is activated. Until at least one piece of image information is collected through the camera until the user detects that the pressing operation of the first touch operation area is canceled, and the at least one piece of image information is used as a video frame.

Further, the second processing unit 1202 is specifically used to transmit a video frame to all the second electronic devices of the at least one second electronic device,
Or
The second processing unit 1202 is specifically used to select a target second electronic device from the at least one second electronic device and transmit the video frame to the target second electronic device.

  In other words, the first electronic device switches to the second communication state after the first communication state in which the voice data interacts with a plurality of electronic devices, and transmits the collected images to all the second electronic devices. Alternatively, one target device may be selected from a plurality of electronic devices, and an image may be transmitted to the selected target device.

The electronic device further includes an audio collection unit 1206 for collecting audio data and adding a time stamp to the audio data,
Specifically, the image collection unit 1203 is used to add a time stamp corresponding to the collected video frame when the video frame is collected.

  The first processing unit 1201 is used to specifically analyze the voice and time stamp in the voice data and to control to output the voice data based on the analyzed time stamp. The analysis unit 1205 In particular, it is used to analyze a video frame and a time stamp corresponding to the video frame and to output a video frame according to the analyzed time stamp.

  Therefore, according to the above technical solution, when the user is in the first communication state, only the voice data is interacted, and the electronic device is controlled to switch to the second communication state in response to the first operation. In addition to transmission, it also shares the video frames of the currently collected image file, and in response to the second operation, the electronic device can be switched back to the first communication state. In this way, at the time of voice communication interaction, the user can freely control the timing of the video frame interaction of the image file, so that the video frame and voice data of the image file at the time of communication can be maintained continuously. The processing resources of the server used to transfer to the server can be reduced, and the smoothness of both calls can be maintained.

  It should be understood that in some embodiments provided herein, the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely exemplary, for example, the unit division is merely logical function division, and may actually be another division when realized, For example, multiple units or components may be combined or assembled into another system, or some technical features may be omitted or not implemented. Also, the coupling or direct coupling, or communication connection between each component that is explicitly or described is connected by indirect coupling or communication of several interfaces, devices, or units. It may be electrical, mechanical, or other types.

  The unit described above as the separation component may be physically separated or may not be physically separated. The component as a unit may be a physical unit or may not be a physical unit. It may be placed in one place and distributed to multiple network units. Depending on actual needs, some or all of the units may be selected to realize the object of the technical solution of this embodiment.

  In addition, each functional unit of each embodiment of the present invention may be assembled into one processing unit, each unit may be a single unit, or two or more units may be assembled into one unit. Alternatively, the assembled unit may be realized by a hardware method, or may be realized by a method of adding a software function unit to hardware.

  Those skilled in the art may implement all or part of the steps of the above-described method embodiments by instructing related hardware by a program, and the program is a computer-readable storage medium. When the program is executed, the steps including the embodiments of the method are executed, and the storage medium includes a mobile storage medium, a read-only memory (ROM), It should be understood to include media capable of storing various program codes, such as magnetic disks or compact disks.

  This embodiment provides specific hardware based on the above device embodiment, and the electronic device includes at least a processor, a memory, a microphone, and a camera.

  Here, the processor controls the electronic device to be in a first communication state, and in the first communication state, the first electronic device performs audio data interaction with at least one second electronic device. , In response to the first operation, control to switch from the first communication state to the second communication state, and in the second communication state, holds the interaction of audio data with the at least one second electronic device, The video frame is shared with at least one second electronic device, and is used to control to switch from the second communication state to the first communication state in response to a second operation.

  What has been described above are merely specific embodiments of the present invention, and the present invention is not limited thereto, and modifications that can be easily conceived by those skilled in the art within the scope disclosed in the present invention. Or substitutions should be included within the scope of the invention. Therefore, the scope of the present invention should be subject to the appended claims.

Claims (15)

A communication processing method applied to a first electronic device,
Controlling the first electronic device to be in a first communication state, and in the first communication state, the first electronic device interacts with audio data with at least one second electronic device;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and Sharing a video frame with at least one second electronic device;
In response to a second operation, controlling to switch from the second communication state to the first communication state;
Including the communication processing method. In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the interaction of audio data with the at least one second electronic device is held, And sharing a video frame with at least one second electronic device,
Controlling to switch from the first communication state to the second communication state in response to the first operation, and maintaining the interaction of the audio data with the at least one second electronic device in the second communication state; ,
Collecting at least one video frame and transmitting the at least one video frame to the at least one second electronic device;
The communication processing method according to claim 1, comprising: Prior to responding to the first operation, the communication processing method further detects the first operation when it is detected that the length of time for operating the first touch operation area is greater than a first threshold. Including confirming,
Prior to responding to the second operation, the communication processing method further includes determining that the second operation has been detected when it is detected that the touch operation on the first touch operation area has been canceled.
The communication processing method according to claim 2. Sharing a video frame with the at least one second electronic device;
Receiving at least one video frame transmitted from at least one second electronic device;
Analyzing the at least one video frame transmitted from at least one second electronic device;
The communication processing method according to claim 1 or 2, comprising: Sharing a video frame with the at least one second electronic device;
Transmitting at least one video frame to all second electronic devices of the at least one second electronic device;
Or
Selecting a target second electronic device from among the at least one second electronic device and transmitting the video frame to the target second electronic device;
The communication processing method according to claim 1, comprising: Adding a time stamp to the audio data when the audio data is collected,
Adding a time stamp corresponding to the collected video frame when collecting the video frame;
The communication processing method according to claim 4, further comprising: Analyzing the at least one video frame transmitted from the at least one second electronic device;
Analyzing audio and time stamps in audio data, analyzing video frames and corresponding time stamps,
Controlling to output audio data and video frames based on the parsed timestamps;
The communication processing method according to claim 6, comprising: An electronic device,
The first electronic device is controlled to be in the first communication state, and in the first communication state, the voice data interacts with at least one second electronic device, and in response to the second operation, the second communication state A first processing unit configured to control to switch from to a first communication state;
In response to the first operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained; and A second processing unit configured to share a video frame with at least one second electronic device;
Including electronic devices. Further comprising an image collection unit configured to collect at least one image information;
The second processing unit controls to switch from the first communication state to the second communication state in response to the first operation, and in the second communication state, audio data with the at least one second electronic device Configured to transmit the video frame to the at least one second electronic device.
The electronic device according to claim 8. When it is detected that the length of time for operating the first touch operation area is longer than the first threshold, it is determined that the first operation is detected, and it is detected that the touch operation for the first touch operation area is canceled. A touch detection unit configured to confirm that the second operation has been detected,
The electronic device according to claim 9, further comprising: An analysis unit configured to analyze at least one video frame transmitted from the at least one second electronic device;
The second processing unit is configured to receive a second video frame transmitted from at least one second electronic device;
The electronic device according to claim 9 or 10. The second processing unit is configured to transmit at least one video frame to all second electronic devices of the at least one second electronic device;
Or
The second processing unit is configured to select a target second electronic device from the at least one second electronic device and to transmit the at least one video frame to the target second electronic device;
The electronic device according to claim 9. Further comprising an audio collection unit configured to collect audio data and add a timestamp to the audio data;
The image collection unit is configured to add a time stamp corresponding to the collected video frame when collecting the video frame.
The electronic device according to claim 11. The first processing unit is configured to analyze a voice and a time stamp in the voice data and to control to output the voice data according to the analyzed time stamp,
The analysis unit is configured to analyze the video frame and a corresponding time stamp and control to output the video frame according to the analyzed time stamp.
The electronic device according to claim 13. An electronic device,
Including processor, memory, microphone and camera,
The memory is configured to store a set of instructions that, when executed, cause at least one processor to perform an operation;
The processor controls the first electronic device to be in a first communication state, and in the first communication state, the first electronic device interacts with at least one second electronic device for voice data, and In response to one operation, control is performed to switch from the first communication state to the second communication state, and in the second communication state, the voice data interaction with the at least one second electronic device is maintained, and at least Configured to share a video frame with one second electronic device and control to switch from the second communication state to the first communication state in response to a second operation;
The electronic device.

Images