CN102215217B

CN102215217B - Establishing a video conference during a phone call

Info

Publication number: CN102215217B
Application number: CN201010600623.4A
Authority: CN
Inventors: E·C·克兰菲尔; S·O·勒梅; J·S·阿布安; 吴锡荣; 周小松; 小R·加西亚
Original assignee: Apple Computer Inc
Current assignee: Apple Inc
Priority date: 2010-04-07
Filing date: 2010-09-25
Publication date: 2014-09-17
Anticipated expiration: 2030-09-25
Also published as: CN102215217A

Abstract

Some embodiments provide a method for initiating a video conference using a first mobile device. The method presents, during an audio call through a wireless communication network with a second device, a selectable user-interface (UI) item on the first mobile device for switching from the audio call to the video conference. The method receives a selection of the selectable UI item. The method initiates the video conference without terminating the audio call. The method terminates the audio call before allowing the first and second devices to present audio and video data exchanged through the video conference.

Description

During conversing, set up video conference

When the model machine of the iPhone 4 of Apple is in the time that on March 25th, 2010 is stolen from an engineer of Apple, the invention that will disclose and advocate in the application by advance and without Apple authorize to public.At this obviously before theft, not yet submit to the application based on U.S.'s priority application.

Background technology

Many current portable sets, such as intelligent telephone set possesses video capture function.By the camera on telephone set, the user of portable set can take rest image and video.But, after completing video capture, for the video of taking is sent to the opposing party, user must directly send to described the opposing party video conventionally, or video is uploaded to another location (for example, internet video trustship website (hosting site)).Unfortunately, this does not allow described the opposing party in portable set capture video, checks live video stream.

In addition, the portable set of standard is only equipped with a camera, and processes the quite difficulty of information from this camera.Desirable equipment ought to have multi-section camera, and can send real-time video, and described real-time video is synthesizing from the video of at least two cameras.Just process the equipment of the video flowing of multiple shootings, and with described equipment connection, be responsible for processing the network of transmission of live video stream, in view of the resource-constrained that can utilize for portable set, this is an especially problem for difficulty.

Summary of the invention

Some embodiments of the present invention provide to be had two and can take pictures and the mobile device of the camera of video.The mobile device of some embodiment has the photograph image of demonstration shooting and the display screen of video image.It also comprises preserves the image of taking to send after a while the memory of another equipment to.Described equipment also has network interface, and described network interface allows this equipment during the real time communication session between the user of multiple equipment, and the image of taking is sent to one or more equipment.Described equipment also comprises encoder, and it can use described encoder to the Image Coding of taking, and stores or send to another equipment so that local.Described mobile device also comprises decoder, and described decoder allows this equipment to decode to the diagram of being taken by another equipment during real time communication session, or the image decoding that this locality is preserved.

An example that relates to the real time communication session of the transmission of the video image of shooting is video conference.In certain embodiments, any specific time of mobile device during video conference, be merely able to transmit the video image of a camera.But, in other embodiments, mobile device can, during video conference or other real time communication session, send the video image from its two camera simultaneously.

With the video conference of another equipment during, the mobile device of some embodiment, by together with the video by one portion or two camera, transmits the content of other type.The camera that an example of this other content is included in this equipment is during for the video of video conference, by low resolution or the high resolution picture image of another camera of equipment.Other example of this other content comprises that (1) is kept at file and other content on equipment, (2) screen display of equipment (, be presented at the content on the screen of equipment), (3) during video conference or other real time communication session, the content receiving from another equipment, etc.

The mobile device of some embodiment adopts (in-conference) adjustment technology in novel meeting, during video conference, adjusts.For example, only transmit the video of a camera during video conference time, the mobile device of some embodiment can dynamically switch to the video transmitting by its another camera.In this case, the mobile device of some embodiment participates in this switching notice any miscellaneous equipment of video conference, and described miscellaneous equipment can be provided by the level and smooth conversion between the video of these two camera in its that one end.

In certain embodiments, switch the request of camera and not only can originate from during video conference, " this locality " equipment switching between its camera, but also can originate from another " long-range " equipment that is just receiving the video that local device takes.In addition, allowing another equipment of a device directive to switch camera is an example of the remote control ability of the equipment of some embodiment.In certain embodiments, example from other operation to device directive that can long-range comprises that exposure adjusts operation (for example, automatic exposure), Focussing operation (for example, automatic focus) etc.Another example of adjusting in the novel meeting of can Local or Remote specifying is the area-of-interest (ROI) in identification capture video, with the behavior that uses described ROI identification amendment to take camera, amendment is with the image processing operations of the equipment of shooting camera, or amendment is with the encoding operation of the equipment of shooting camera.

Another example of adjusting in the novel meeting of some embodiment relates to the real time modifying of the synthetic video demonstration of equipment generation.Specifically, in certain embodiments, mobile device generation shows the compound display by multiple videos of the multi-section camera of one or more equipment simultaneously.In some cases, compound display is placed on video in adjacent viewing area and (for example, is placed in adjacent window).In other cases, compound display is that picture-in-picture (PIP) shows, described PIP demonstration comprises that at least two viewing areas that show two different videos, one of them viewing area are the main viewing areas of background, and another viewing area is that the prospect overlapping on the main viewing area of background is inserted viewing area.

The real time modifying that synthetic video in some embodiment shows relates to the selection to viewing area and movement in response to user, mobile one or more viewing areas in compound display.In the time that the screen rotation of equipment of compound display is provided, some embodiment also rotate this compound display during video conference.In addition, video during user's exchange (swap) PIP of the mobile device permission equipment of some embodiment shows (, the video that prospect is inserted in showing appears in the main demonstration of background, and the video in the main demonstration of background is appeared in prospect insertion demonstration).

Summary of the invention intention above is simply introduced some embodiments of the present invention.Do not mean it is introduction or the summary of disclosed all subject matters in the literature.Embodiment below and the accompanying drawing relating in embodiment will further illustrate the embodiment describing in described summary of the invention, and other embodiment.Therefore, be appreciated that all embodiment that the literature is described, need to intactly investigate summary of the invention, embodiment and accompanying drawing.

Brief description of the drawings

In accessory claim, state novel feature of the present invention.But, in order to illustrate, in following accompanying drawing, several embodiments of the present invention have been represented.

The compound display of some embodiment of Fig. 1 graphic extension.

The another kind of compound display of some embodiment of Fig. 2 graphic extension.

Fig. 3 is the Video processing of two camera movement equipment and the software architecture of coding module of some embodiment of graphic extension conceptually.

Fig. 4 is the photographic images processing unit of some embodiment of graphic extension conceptually.

Fig. 5 concept map explains orally Benq in the example of the different frame rates of different vertical blanking interval (VBI).

Fig. 6 concept map explains orally Benq in the example of the different interlacing scan frame rate of different VBI.

Fig. 7 is another photographic images processing unit of some embodiment of graphic extension conceptually.

Fig. 8 is another photographic images processing unit of some embodiment of graphic extension conceptually.

Fig. 9 is the video conference of two camera movement equipment and the software architecture of processing module of some embodiment of graphic extension conceptually.

Figure 10 is the example video conference request sending and receiving sequence of some embodiment of graphic extension conceptually.

The user interface of the video conference setting operation of some embodiment of Figure 11 graphic extension.

The user interface of accepting video conference invitation of some embodiment of Figure 12 graphic extension.

Another user interface of accepting video conference invitation of some embodiment of Figure 13 graphic extension.

Another user interface of the video conference setting operation of some embodiment of Figure 14 graphic extension.

Figure 15 is the processing of the bit rate of the setting video conference of some embodiment of graphic extension conceptually.

Figure 16 is the video conference of two camera movement equipment of some embodiment of graphic extension and another software architecture of processing module conceptually.

Figure 17 is another software architecture of two camera movement equipment of some embodiment of graphic extension conceptually.

Figure 18 processing that graphic extension is carried out by the video conference manager as some embodiment of graphic extension in Figure 16 conceptually.

Figure 19 is the software architecture of the time reducing noise module of some embodiment of graphic extension conceptually.

Figure 20 is the processing of the time noise of the reduction video image of some embodiment of graphic extension conceptually.

Figure 21 processing that graphic extension is carried out by the image processing manager as some embodiment of graphic extension in Fig. 9 conceptually.

The user interface of the exposure adjustment operation of some embodiment of Figure 22 graphic extension.

The user interface of the Focussing operation of some embodiment of Figure 23 graphic extension.

Figure 24 perspective correction processing that graphic extension is carried out by the image processing manager as some embodiment of graphic extension in Figure 16 conceptually.

Figure 25 is the example perspective correction operation of some embodiment of graphic extension conceptually.

Figure 26 conceptually graphic extension as the software architecture of the encoder-driven device of some embodiment of graphic extension in Figure 16.

Figure 27 image scaling processing that graphic extension is carried out by the encoder-driven device as some embodiment of graphic extension in Figure 26 conceptually.

Figure 28 conceptually graphic extension by as the software architecture of the decoder driver of some embodiment of graphic extension in Figure 16.

Figure 29 image extraction process that graphic extension is carried out by the decoder driver as some embodiment of graphic extension in Figure 28 conceptually.

The encoder-driven device that comprises two rate controllers of some embodiment of Figure 30 graphic extension.

Figure 31 conceptually graphic extension as the software architecture of the networking manager of some embodiment of graphic extension in Figure 16.

The user interface that locks onto corner (snap to corner) operation of some embodiment of Figure 32 graphic extension.

Another user interface that locks onto corner operation of some embodiment of Figure 33 graphic extension.

The PIP of some embodiment of Figure 34 graphic extension shows the user interface of rotation operation.

The PIP of some embodiment of Figure 35 graphic extension shows another user interface of rotation operation.

The PIP of some embodiment of Figure 36 graphic extension shows another user interface of rotation operation.

The PIP of some embodiment of Figure 37 graphic extension shows another user interface of rotation operation.

Prospect during the convergent-divergent PIP of some embodiment of Figure 38 graphic extension shows is inserted the user interface of viewing area.

Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 39 graphic extension shows.

Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 40 graphic extension shows.

Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 41 graphic extension shows.

The user interface of the area-of-interest in the identification display frame of some embodiment of Figure 42 graphic extension.

Another user interface of area-of-interest in the identification display frame of some embodiment of Figure 43 graphic extension.

Another user interface of area-of-interest in the identification display frame of some embodiment of Figure 44 graphic extension.

The processing of carrying out local switching camera operation on two camera movement equipment of some embodiment of Figure 45 graphic extension.

The user interface of the switching camera operation of some embodiment of Figure 46 graphic extension.

Another user interface of the switching camera operation of some embodiment of Figure 47 graphic extension.

Another user interface of the switching camera operation of some embodiment of Figure 48 graphic extension.

Another user interface of the switching camera operation of some embodiment of Figure 49 graphic extension.

The processing of carrying out long-range switching camera operation on two camera movement equipment of some embodiment of Figure 50 graphic extension.

The Long-distance Control of some embodiment of Figure 51 graphic extension is switched the user interface of camera operation.

The Long-distance Control of some embodiment of Figure 52 graphic extension is switched another user interface of camera operation.

The Long-distance Control of some embodiment of Figure 53 graphic extension is switched another user interface of camera operation.

The Long-distance Control of some embodiment of Figure 54 graphic extension is switched another user interface of camera operation.

Figure 55 conceptually the carrying out exposure of some embodiment of graphic extension adjusts the processing operating.

The user interface of operation is adjusted in the carrying out exposure of some embodiment of Figure 56 graphic extension.

Another user interface of operation is adjusted in the carrying out exposure of some embodiment of Figure 57 graphic extension.

Another user interface of operation is adjusted in the carrying out exposure of some embodiment of Figure 58 graphic extension.

Figure 59 exposure adjustment processing that graphic extension is carried out by the image processing manager as some embodiment of graphic extension in Figure 16 conceptually.

Figure 60 is the exposure adjustment operation of some embodiment of graphic extension conceptually.

Figure 61 is the processing of the execution Focussing operation of some embodiment of graphic extension conceptually.

The user interface of the Focussing operation of some embodiment of Figure 62 graphic extension.

Another user interface of the Focussing operation of some embodiment of Figure 63 graphic extension.

Another user interface of the Focussing operation of some embodiment of Figure 64 graphic extension.

The difference about the video of taking from one or more pairs of camera movement equipment of some embodiment of Figure 65 graphic extension shows arrangement.

The prospect inserting video of some embodiment of Figure 66 graphic extension is superimposed upon the user interface on the background video in PIP demonstration.

The technology of the prospect of definite video image of some embodiment of Figure 67 graphic extension.

Some embodiment of Figure 68 graphic extension during video conference, exchange PIP show in insertion demonstration and background show user interface.

The user interface that locks onto corner operation of some embodiment of Figure 69 graphic extension.

The user interface that locks onto corner and push operation of some embodiment of Figure 70 graphic extension.

The PIP of some embodiment of Figure 71 graphic extension shows the user interface of rotation operation.

The PIP of some embodiment of Figure 72 graphic extension shows another user interface of rotation operation.

Some embodiment of Figure 73 graphic extension during video conference, that from two long-distance videos, selects a video uses corpse interface.

Some embodiment of Figure 74 graphic extension during video conference, from two local videos, select the user interface of a video.

Before the meeting of some embodiment of Figure 75 graphic extension, select the user interface for the video of video conference.

The example of the allocated bandwidth between two videos being taken by two camera movement equipment of some embodiment of Figure 76 graphic extension.

Figure 77 is the arbiter module of the rate controller of the two camera movement equipment of management of some embodiment of graphic extension conceptually.

Figure 78 method that the image to the camera by two camera movement equipment of some embodiment of graphic extension is encoded conceptually.

Figure 79 other method that the image to the camera by two camera movement equipment of some embodiment of graphic extension is encoded conceptually.

Figure 80 is illustrated in the example image of the method for graphic extension in Figure 79 and synthesizes.

Figure 81 other method that the image to the camera by two camera movement equipment of some embodiment of graphic extension is encoded conceptually.

Figure 82 method that the image to the camera by two camera movement equipment of some embodiment of graphic extension is decoded conceptually.

Figure 83 other method that the image to the camera by two camera movement equipment of some embodiment of graphic extension is decoded conceptually.

Figure 84 is the video conference of two camera movement equipment of some embodiment of graphic extension and another software architecture of processing module conceptually.

The user interface of the multipart video-meeting of some embodiment of Figure 85 graphic extension.

Another user interface of the multipart video-meeting of some embodiment of Figure 86 graphic extension.

Another user interface of the multipart video-meeting of some embodiment of Figure 87 graphic extension.

Figure 88 is the API of some embodiment of graphic extension (API) architecture conceptually.

The architecture of two camera movement computing equipments of some embodiment of Figure 89 graphic extension.

Figure 90 is touch I/O (I/O) equipment of some embodiment of graphic extension conceptually.

Figure 91 is the example communication system of some embodiment of graphic extension conceptually.

Figure 92 is another example communication system of some embodiment of graphic extension conceptually.

Embodiment

In the following description, for purpose of explanation, stated numerous details.But, those of ordinary skill in the art will appreciate that and can in the situation that not utilizing these details, put into practice the present invention.In other cases, for because unnecessary details makes description of the invention smudgy, known structure and equipment have been represented with the form of block diagram.

Some embodiments of the present invention provide a kind of to be had two and can take pictures and the mobile device of the camera of video.The example of mobile device comprises the mobile computing device of mobile phone, intelligent telephone set, personal digital assistant (PDA), laptop computer, tablet personal computer or any other type.Photo used herein refers to by the each ground of individual screening-mode, or presses the picture image of each several the ground of quick screening-mode by camera.On the other hand, video refers to a series of video images that camera is taken with special speed (being commonly referred to frame rate).The typical frame speed of capture video is 25 frame/seconds (fps), 30fps and 60fps.The camera of the mobile device of some embodiment can be with these and other frame rate capture video image (, frame of video).

The mobile device (1) of some embodiment can show photograph image and the video image of shooting, (2) can preserve the image of shooting, to send after a while another equipment to, (3) can be during the real time communication session between the multidigit user of multiple equipment, the image of taking is sent to one or more equipment, (4) can, to the Image Coding of taking, store or send to another equipment so that local.

An example that relates to the real time communication session of the transmission of the video image of shooting is video conference.In certain embodiments, any specific time during video conference, mobile device is merely able to transmit the video image of a camera.But, in other embodiments, during video conference or other real time communication session, mobile device can transmit the video image of its two camera simultaneously.

The mobile device of some embodiment produces compound display, and described compound display shows when comprising by multiple video of the multi-section camera of one or more equipment.In some cases, compound display is placed on video in adjacent viewing area and (for example, is placed in adjacent window).A this example of Fig. 1 graphic extension compound display 100, compound display 100 comprises two adjacent viewing areas 105 and 110, these two viewing areas 105 and 110 show by two camera of an equipment simultaneously, or by two videos of two camera of two distinct devices of participating in video conference.

In other cases, compound display is that the PIP that comprises at least two viewing areas that show two different videos shows, one of them viewing area is the main viewing area of background, and another viewing area is that the prospect overlapping on the main viewing area of background is inserted viewing area.The synthetic PIP of Fig. 2 graphic extension shows a this example of 200.Synthetic PIP shows that 200 comprise the main viewing area 205 of background and overlap the prospect insertion viewing area 210 on the main viewing area of background.These two viewing areas 205 and 210 show by two camera of an equipment simultaneously, or by two videos of two camera of two distinct devices of participating in video conference.Although showing, the synthetic PIP of the example illustrating in this article and discuss is similar to the synthetic PIP demonstration 200 of inserting viewing area 210 in the whole prospect of the background interior demonstration in main viewing area 205, but have on the main viewing area 205 of the background of overlapping, but also other synthetic PIP of incomplete prospect insertion viewing area 210 in the main viewing area 205 of background shows it is also possible.

Except with the video conference of another equipment during transmit video content, the mobile device of some embodiment can transmit together in company with the video content of meeting the content of other type.An example of this other content is included in one of camera of equipment while taking the video for video conference, by low resolution or the high resolution picture image of another camera of this equipment.Other example of this other content comprises that (1) is kept at file and other content on equipment, (2) screen display of equipment (, be presented at the content on the screen of equipment), (3) during video conference or other real time communication session, the content receiving from another equipment, etc.

The mobile device of some embodiment adopts adjustment technology in novel meeting, during video conference, adjusts.For example, when only transmit the video of a camera during video conference time, the mobile device of some embodiment can dynamically switch to the video transmitting by its another camera.In this case, the mobile device of some embodiment is participated in this switching notice any another equipment of video conference, makes described another equipment to provide the level and smooth conversion between the video of these two camera in its that one end.

Another example of adjusting in the novel meeting of some embodiment relates to the real time modifying of the synthetic video demonstration of equipment generation.Specifically, in certain embodiments, the real time modifying that synthetic video shows relates to the selection to viewing area and movement in response to user, mobile one or more viewing areas in compound display.In the time that the screen rotation of equipment of compound display is provided, some embodiment also rotate this compound display during video conference.In addition, the order of the video during user's upset (flip) PIP of the mobile device permission equipment of some embodiment shows (, the video that prospect is inserted in showing appears in the main demonstration of background, and the video in the main demonstration of background is appeared in prospect insertion demonstration).

The following describes several more detailed embodiment.I joint provides the explanation of the video processing architecture of some embodiment.Subsequently, II joint is described the photographic images processing unit of some embodiment.In certain embodiments, photographic images processing unit is that the responsible processing of equipment is by the assembly of the original image of the camera of equipment.

Next, III joint is described the video conference architecture of some embodiment.III joint is also described the video conference module of some embodiment, and several modes of single camera video conference are set.Afterwards, in the meeting of some embodiment of IV joint description, adjust and control operation.V joint describe subsequently some embodiment during video conference, transmit and show the video conference feature from multiple videos of each equipment.Next, during VI joint is described in video conference, in company with the transmission of non real-time content real-time video together.Finally, VII joint is described the hardware architecture of two camera apparatus of some embodiment.

I. video capture and processing

Fig. 3 is Video processing and the coding module 300 of two camera movement equipment of some embodiment of graphic extension conceptually.In certain embodiments, module 300 is processed the image of the camera by two camera movement equipment, and video is encoded.As shown in Figure 3, module 300 comprises photographic images processing unit (CIPU) driver 305, exchange of media module 310, encoder-driven device 320 and video processing module 325.

In certain embodiments, program (user of media content and producer) the switched-media content on exchange of media module 310 permission equipment and the instruction about the processing of media content.In Video processing and coding module 300, the exchange of media module 310 of some embodiment between video processing module 325 and CIPU driver 305, and between video processing module 325 and encoder-driven device 320 these instructions of route and media content.In order to make the route of described instruction and media content easier, the exchange of media module 310 of some embodiment provides the one group of API (API) for the user of media content and producer.In some such embodiment, exchange of media module 310 is as the one group of one or more framework of a part that run on the operating system on two camera movement equipment.An example of this exchange of media module 310 is Core Media frameworks that Apple provides.

Image and/or the video of video processing module 325 to the camera by equipment carries out image processing.The example of this operation comprises that exposure adjustment operation, Focussing operation, perspective correction, dynamic range adjustment, image scaling, image synthesize etc.In certain embodiments, some image processing operations also can be carried out by exchange of media module 310.For example, as shown in Figure 3, the exchange of media module 310 of some embodiment is carried out time reducing noise (TNR) operation (for example,, with TNR 315) of reduction by the noise in the video image of the camera of equipment.The other example of this image processing operations of video processing module 325 and exchange of media module 310 is provided below.

By exchange of media module 310, video processing module 325 and CIPU driver 305 and encoder-driven device 320 interfaces, as mentioned above.CIPU driver 305 serves as the communication interface between photographic images processing unit (CIPU) 330 and exchange of media module 310.As described further below, CIPU 330 is assemblies of two camera apparatus, is responsible for processing the image taking of camera or the image of video capture operating period shooting at equipment.CIPU driver 305, by exchange of media module 310, receives the image of one or two camera from equipment and/or the request of video from video processing module 325.CIPU driver 305 passes to CIPU 330 described request, as response, receive image and/or the video of request from CIPU 330, CIPU driver 305, subsequently by exchange of media module 310, sends to video processing module 325 the image and/or the video that receive.By CIPU driver 305 and exchange of media module 310, the video processing module 325 of some embodiment also sends instruction to CIPU 330, for example, to revise its certain operations (, operation, Focussing operation etc. are adjusted in the frame rate of amendment camera, exposure).

Encoder-driven device 320 serves as for example, communication interface between exchange of media module 310 and encoder hardware 335 (, encoding pack on encoder chip, chip system etc.).In certain embodiments, encoder-driven device 320, by exchange of media module 310, receives image and the request to Image Coding from video processing module 325.Encoder-driven device 320 sends to encoder 335 the image that will encode, and encoder 335 carries out photo coding or Video coding to image subsequently.In the time that encoder-driven device 320 receives coded image from encoder 335, encoder-driven device 320 is by exchange of media module 310, coded image loopback to video processing module 325.

In certain embodiments, the coded image that video processing module 325 can receive from encoder it carries out different operations.The example of described operation comprises coded image is kept in the memory of equipment, transmits coded image by the network interface of equipment in video conference, etc.

In certain embodiments, some or all modules of Video processing and coding module 300 are realized as a part for operating system.For example, some embodiment are embodied as all four assemblies 305,310,320 and 325 of Video processing and coding module 300 part for the operating system of equipment.Other embodiment is embodied as exchange of media module 310, CIPU driver 305 and encoder-driven device 320 part for the operating system of equipment, and using video processing module 325 as the application moving in operating system.In addition, other realization of module 300 is also possible.

The operation of Video processing and coding module 300 is described now during video capture session.In order to start video capture session, video processing module 325 is initialized as the required several assemblies of video capture session.In certain embodiments, these assemblies comprise (1) CIPU 330, (2) proportional zoom of video processing module 325 and synthesis module (not shown), (3) the image processing module (not shown) of video processing module 325, and (4) encoder 335.In addition, in the time participating in video conference, the video processing module 325 initialization network manager (not shown) of some embodiment.

By exchange of media module 310 and CIPU driver 305, video processing module sends to CIPU 330 its initialization request, so that one of equipment or two cameras start video capture.In certain embodiments, described request is specified particular frame speed, exposure and the scaling size of every the camera that needs capture video.Response described request, frame rate, exposure and scaling size that CIPU 330 starts to specify, return to the video image from asked camera.These video images are returned to video processing module 325 by CIPU driver 305 and exchange of media module 310, and as mentioned above, before video image is offered to video processing module 325, exchange of media module 310 is carried out TNR operation to video image.At video processing module 325, video image is stored in buffer (not shown), to carry out other image processing.

The image processing module of video processing module 325 is fetched the video image being kept in buffer, to carry out other Video processing.Proportional zoom and synthesis module are fetched video image after treatment subsequently, so that scaled video image (if the necessary words of real-time demonstration on the display screen of equipment).In certain embodiments, the image of this module by two camera of equipment, or be used in during video conference the image creation composograph by the camera of equipment and the camera of another equipment, to the real-time demonstration of the video image of shooting is provided on equipment, or create the composite video image for coding.

Process and/or synthetic after video image be provided for encoder 335 by encoder-driven device 320 and exchange of media module 310.Encoder 335 is subsequently to encoding video pictures.Coded image is returned to video processing module 325 (again by encoder-driven device 320 and exchange of media module 310) subsequently, to be kept on equipment, or transmits during video conference.In the time that equipment is participated in video conference, network manager (by video processing module 325 initialization) is fetched these coded images subsequently, to coded image subpackage, and by the network interface (not shown) of equipment, coded image is sent to one or more miscellaneous equipments.

II. photographic images processing

A. single lead

Image by the camera of two camera movement equipment of some embodiment is original raw image.Can be used to other operation at these images, for example, before image being sent to another equipment (, during video conference), preserve image or showing image, these images need to transform to the specific color space.In addition, the image of camera needs processed, with error recovery and/or distortion, and adjusts color, size of image etc.Therefore, some embodiment, before preserving, transmit and showing this image, carry out several processing operations to image.The section processes of described image is carried out by CIPU 330.

In Fig. 4, illustrate an example of described CIPU.Specifically, Fig. 4 summarizes the photographic images processing unit (CIPU) 400 of some embodiment of ground graphic extension.CIPU 400 comprises or only processes at every turn the image of one of camera from equipment, or process the single processing pipeline (single processing pipeline) 485 from the image of two cameras of equipment according to time division multiplexing mode (, according to time-interleaved mode) simultaneously.Can differently configure the processing pipeline 485 of CIPU 400, to process different qualities and/or the operation setting of different cameras.The example of the different camera characters in some embodiment comprises different resolution, noise transducer, lens type (fixed lens or zoom lens), etc.In addition, in certain embodiments, equipment can comprise image resolution ratio size, frame rate, zoom rank, exposure etc. according to the example of the different operating setting of its operate camera.

As shown in Figure 4, CIPU 400 comprises sensor assembly 415, OK/frame buffer 417, bad pixel correction (BPC) module 420, lens shade (LS) module 425, demosaicing module 430, white balance (WB) module 435, γ module 440, color space transformation (CSC) module 445, tone, saturation and contrast (HSC) module 450, scaler module 455, filter module 460, statistics engine 465, two groups of registers 470, and controller module 475.In certain embodiments, whole modules of CIPU 400 are all for example, to realize with hardware (, ASIC, FPGA, with the SOC of microcontroller, etc.), and in other embodiments, some or all modules of CIPU 400 realize with software.

As shown in Figure 4, two pel array 410a of two cameras of sensor assembly 415 and equipment and 410b, and two sensors 405a and 405b coupled in communication.In certain embodiments, by the mobile industry processor interface (MIPI) of each camera sensor, make described coupled in communication easier.

By described coupled in communication, sensor assembly 415 can forward instruction to camera, to control the various aspects of operation of every camera, such as its power stage, zoom rank, focal length, exposure etc.In certain embodiments, every camera has four kinds of operating power patterns.Under the first operating power pattern, camera is de-energized.With regard to the second operating power pattern, camera is energized, but camera is not also configured.Under the third operating power pattern, camera is energized, and the transducer of camera is configured, and the pixel of camera sensor collection photon, and a photon of collecting is converted to digital value.But, camera sensor does not also send image to sensor assembly 415.Finally, under the 4th kind of operating power pattern, camera is under the operating power pattern identical with the third power mode, except camera sends image to sensor assembly 415 now.

In the operating period of equipment, camera can repeatedly be switched to another kind of operating power pattern from a kind of operating power pattern.In the time of switch operating power mode, some embodiment require camera according to said sequence switch operating power mode.So in these embodiments, the camera under the first operating power pattern can only be switched to the second operating power pattern.When camera is during in the second operating power pattern, it can be switched to the first operating power pattern or the third operating power pattern.Similarly, camera can be switched to the second operating power pattern or the 4th kind of operating power pattern from the third operating power pattern.When camera is during in the 4th kind of operating power pattern, it is merely able to switch back the third operating power pattern.

In addition, be switched to one or front a kind of operating power pattern a certain amount of time of needs from a kind of operating power pattern.Thereby, switch two or three operating power pattern and be slower than a kind of operating power pattern of switching.Different operating power patterns also consumes the power of different amounts.For example, the 4th kind of operating power pattern consumes the power of maximum, and the power that the third operating power pattern consumes is higher than the first and the second operating power pattern, and the power that the second operating power pattern consumes is higher than the first operating power pattern.In certain embodiments, the first operating power pattern does not consume any power.

When camera is not when the 4th kind of operating power pattern in photographic images, can make camera remain under one of other operating power pattern.Make no camera remain on definite under which kind of mode of operation and depend on the request that allows camera to consume how much power and camera needs how soon to respond beginning photographic images.For example, be configured to for example, be configured to the more power of camera consumption in the first operating power pattern (, power-off) according to the camera ratio of the third operating power pattern (, standby mode) work.But, in the time of instruction camera image, can be than be switched to quickly the 4th kind of operating power pattern according to the camera of the first operating power work pattern according to the camera of the third operating power work pattern.Thereby for example, according to different requirements (, the response time of the request to photographic images, power consumption), in the time of photographic images not, camera can be configured to according to different operating power work patterns.

As described further below, ask one or two cameras when video processing module 325 and start photographic images, and when sensor assembly 415 receives this request by controller module 475, by the coupled in communication of it and every camera, sensor assembly 415 can one group or two groups camera sensor of instruction start photographic images.Baeyer filter is superimposed on each camera sensor, thus each camera sensor output Baeyer format-pattern, and described Baeyer format-pattern is stored in the pel array being associated with each camera sensor.Baeyer format-pattern is that wherein each pixel is only preserved a kind of color value: red, blue or green image.

By coupling of it and pel array 410a and 410b, sensor assembly 415 is fetched the primitive Bayer format-pattern being kept in camera image pixel array 410a and 410b.The speed of fetching image from the pel array of camera by controlling sensor assembly 415, sensor assembly 415 can be controlled the frame rate of the video image of being taken by particular camera.The speed of fetching by controlling its image, sensor assembly 415 can also interweave by the reading of the image of different camera, to interweave the image processing of CIPU processing pipeline 485 to the photographic images from different cameras.In below in the son joint of II.A.1 and II.A.2, further illustrate the control that sensor assembly 415 is fetched its image.

Sensor assembly 415 is fetched it image line (, the several rows of pixel of image) from pel array 410a and 410b is preserved in be expert at/frame buffer 417.Each image line of processing in row/frame buffer 417 by CIPU processing pipeline 485.As shown in Figure 4, CIPU processing pipeline 485 is made up of BPC module 420, LS module 425, demosaicing module 430, WB module 43, γ module 440, CSC module 445, HSC module 450, scaler module 455 and filter module 460.In certain embodiments, CIPU processing pipeline 485 (that is, is processed the image from row/frame buffer 417, and in other embodiments, the whole image of CIPU processing pipeline 485 come voluntarily/frame buffers of frame by frame processing 417 line by line by row).

In the example pipeline of graphic extension in Fig. 4, BPC module 420 is the modules of fetching image from row/frame buffer 417.BPC module 420 is carried out bad pixel and is eliminated operation, this operation (for example manages to proofread and correct the bad pixel that may be caused by one or more defective camera sensor in the image fetched, not sensor light of defective photon sensor, sensor light etc. mistakenly).In certain embodiments, BPC module 420, by the specific pixel in movement images and the one or more neighbors in image, detects bad pixel.If the difference between the value of described specific pixel and the value of described neighbor is greater than threshold quantity, use so the mean value of the value of several neighbors that color (, red, green and blue) is identical with described specific pixel to replace the value of described specific pixel.

The operation part of BPC module 420 is by the value control being kept at for this module in two groups of registers 470 of CIPU 400.Specifically, in order to process the image by two different camera of equipment, some embodiment differently configure CIPU processing pipeline 485 about every camera, as mentioned above.By in two groups of CIPU 400 different register 470a (Ra) and two groups of different values of preservation in 470b (Rb), be two different camera arrangement CIPU processing pipeline 485.Every group of register 470 comprises the register (Ra or Rb) for the each module 420-460 in CIPU processing pipeline 485.Each register in each register group is preserved the value of the operation of a processing pipeline module of one group of definition.Therefore, as shown in Figure 4, register group 470a is used to indicate the mode of operation of each processing pipeline module of a camera (camera A) of two camera movement equipment, and register group 470b is used to indicate the mode of operation of each processing pipeline module of another camera (camera B) of two camera movement equipment.

An example that differently configures CIPU processing pipeline 485 for every camera is that the block configuration of CIPU processing pipeline 485 is become to process the image varying in size.For example, if camera sensor 405a is 640 × 480 pixels, camera sensor 405b is 2048 × 1536 pixels, register group 470a be configured to the to hold instruction modules of CIPU processing pipeline 485 is processed the value of 640 × 480 pixel images so, and register group 470b be configured to the to hold instruction modules of CIPU processing pipeline 485 is processed the value of 2048 × 1536 pixel images.

In certain embodiments, different processing pipeline configuration (, register value) is stored in during different profiles arranges.In some such embodiment, allow the user of mobile device to select profile that (for example, by being presented at the user interface on mobile device) one of is set, so that the operation of or multi-section camera to be set.For example, user can select the profile setting that camera arrangement is become to take high-resolution video, the profile setting that identical camera arrangement is become to take low-resolution video, or the profile setting that two cameras is all configured to take the static image of high-resolution.Various configuration is all possible, can be stored in during many different profiles arrange.In other such embodiment, replace and allow user to select profile setting, application or the activity selected according to user, select profile setting automatically.For example, if user selects video conference application, the so automatic profile of selecting two camera arrangement to become capture video, if user selects the application of taking a picture, the profile that selection is configured to take rest image one of camera so automatically, etc.

After BPS module 420, LS module 425 receives the image after bad pixel correction.LS module 425 is carried out correcting lens shadow operation, to proofread and correct the image deflects that caused by the camera lens that produces optical attenuation effect (,, towards the edge of camera sensor, light reduces gradually).This effect causes image illuminance inhomogeneous (for example, in corner and/or edge darker).In order to proofread and correct these image deflects, the LS module 425 of some embodiment is estimated the Mathematical Modeling of the illuminance decay of lens.The model of estimating is used to the lens decay of compensating images subsequently, with illumination image equably not by the part of Uniform Illumination.For example, if the brightness in the corner of image is the half of the brightness of picture centre, the LS module 425 of some embodiment is multiplied by 2 corner pixels value so, to produce uniform image.

Demosaicing module 430 is carried out demosaicing operation, to produce full-colour image by the image of sample color.As mentioned above, camera sensor output Baeyer format-pattern, because each pixel of Baeyer format-pattern is only preserved a kind of colour, therefore Baeyer format-pattern is incomplete.Demosaicing module 430 by inserting the colour of every group of color, by Baeyer format-pattern reconstruct red, green, blue (RGB) image in Baeyer format-pattern.

WB module 435 is carried out white balance operation to the RGB image that receives from demosaicing module 430, makes the color similarity of the described content that in the color of picture material and actual life, human eye perceives arrives.WB module 435 is adjusted white balance by adjusting the color of image, for example, correctly to present neutral color (, grey, white etc.).For example, the image of a blank sheet of paper under incandescent lamp may be revealed as yellow, and described in human eye perceives, a piece of paper is white.In order to solve the difference between the color of image and the color of the image of human eye perceives that transducer takes, WB module 435 is adjusted the colour of image, makes the image of shooting correctly reflect the color of human eye perceives.

Statistics engine 465 is collected in the view data in each stage of CIPU processing pipeline 485.For example, Fig. 4 represents that statistics engine 465 is collected in LS module 425, demosaicing module 430 and WB module 435 view data afterwards.Different embodiment collects data from the different phase of the arbitrary number of CIPU processing pipeline 485.Statistics engine 465 is processed the data of collecting, and according to data after treatment, adjusts the operation of camera sensor 405a and 405b by controller module 475 and sensor assembly 415.The example of this operation comprises exposure and focuses on.Although Fig. 4 shows the statistics engine 465 of controlling camera sensor 405a and 405b by controller module 475, but, other embodiment of statistics engine 465 controls camera sensor by sensor assembly 415.

Data after treatment also can be used for the operation of the modules of adjusting CIPU 400.For example, the statistics engine 465 of some embodiment, according to the data of collecting after WB module 435, is adjusted the operation of WB module 435.In some such embodiment, statistics engine 465 is by utilizing data after treatment that Automatic white balance (AWB) function is provided, to adjust the white balance operation of WB module 435.Other embodiment can use data the processing of collecting from stage of the arbitrary number of CIPU processing pipeline 485 to adjust the operation of the arbitrary number module in CIPU processing pipeline 485.In addition, statistics engine 465 can also receive instruction from controller module 475, to adjust the operation of one or more modules of CIPU processing pipeline 485.

After WB module 435 receives image, γ module 440 is carried out γ correct operation to image, with brightness or the tristimulus values of Code And Decode camera arrangement.The γ module 440 of some embodiment, by the linear signal of 10-12 bit being converted to the non-uniform encoding of 8 bits, with the γ value of correcting image, is carried out γ value and is proofreaied and correct.Some embodiment utilize look-up table to proofread and correct γ value.

445 images that receive from γ module 440 of CSC module are from a color space transformation to another color space.Specifically, CSC module 445 image from RGB color space transformation to brightness with colourity (YUV) color space.But, other embodiment of CSC module 445 can be to and from the color space transformation image of arbitrary number.

Tone, saturation, contrast or their combination in any of the image that HSC module 450 capable of regulatings receive from CSC module 445.For example, HSC module 450 can be adjusted these character, to reduce noise or to strengthen image.For example, can increase the saturation of the image of being taken by low noise camera sensor, so that image seems more bright-coloured.On the contrary, can reduce the saturation of the image of being taken by strong noise camera sensor, to reduce the coloured noise of this image.

After HSC module 450, the scalable image of scaler module 455, to adjust the pixel resolution of image, or the size of data of adjustment image.For example, scaler module 455 also can reduce the size of image, to be applicable to less display.For example, scaler module 455 can be with many different mode proportional zoom images.For example, scaler module 455 can scale up (, amplifying) and reduce in proportion (, dwindling) image.Scaler module 455 can also bi-directional scaling image, or shifting ground zoomed image.

Filter module 460 is to one or more filtering operations of the image applications receiving from scaler module 455, to change one or more attributes of some or all pixels of image.The example of filter comprises low pass filter, high pass filter, band pass filter, two-sided filter, Gaussian filter, etc.Thereby filter module 460 can be to image applications multiple different filtering arbitrarily.

The controller module 475 of some embodiment is microcontrollers of controlling the operation of CIPU 400.In certain embodiments, controller module 475 (1) is by sensor assembly 41, (for example control the operation of camera sensor, exposure), (2) operation of control CIPU processing pipeline 485, (3) timing of control CIPU processing pipeline 485 (for example, when switch camera sensor, when switch register, etc.), (4) control photoflash lamp/stroboscope light (not shown), described photoflash lamp/stroboscope light is a part for two camera movement equipment of some embodiment.

Some embodiment of controller module 475 process the instruction receiving from statistics engine 465 and CIPU driver 480.In certain embodiments, from two camera movement equipment (the instruction receiving from CIPU driver 480 is, be received from local device) instruction, and in other embodiments, the instruction receiving from CIPU driver 480 is the instruction (for example, the Long-distance Control during video conference) from another equipment.According to instruction after treatment, controller module 475 can, by the value of planning register 470, be adjusted the operation of CIPU 400.In addition, controller module 475 can, in the operating period of CIPU 400, dynamically be planned the value of register 470 again.

As shown in Figure 4, CIPU 400 comprises the number of modules in CIPU processing pipeline 485.But, those of ordinary skill in the art will appreciate that can be only by the module of some graphic extensions, or realize CIPU 400 with other disparate modules.In addition, the processing of being carried out by disparate modules can be according to the be applied sequentially to image different from the order of graphic extension in Fig. 4.

With reference now to Fig. 4,, an exemplary operations of CIPU 400 is described.For the purpose of illustrating, one group of register Ra is used to process the image of being taken by the camera sensor 405a of two camera movement equipment, and one group of register Rb is used to process the image of being taken by the camera sensor 405b of two camera movement equipment.Controller module 475 receives the instruction of the image that produces the shooting of one of camera by two camera movement equipment from CIPU driver 480.

Controller module 475 is the modules of initialization CIPU processing pipeline 485 subsequently, the image of taking to process one of camera by two camera movement equipment.In certain embodiments, this comprises that controller module 475 checks the one group of correct register whether having used in register 470.If for example CIPU driver 480 instruction control unit modules 475 produce the image of being taken by camera sensor 405a, controller module 475 checks that whether one group of register Ra is one group of register that the module of CIPU 400 reads so.If not, controller module 475 switches between two groups of registers so, and making this group register Ra is the one group of register being read by the module of CIPU 400.

Each module in CIPU processing pipeline 485, mode of operation is indicated by the value being kept in described one group of register Ra.As previously mentioned, the value in one group of register 470 can be in the operating period of CIPU 400 by dynamically again regular.Thereby the processing of an image is different from the processing of next image.Although the value of the mode of operation of each module reading and saving in CIPU 400 instruction modules in register is described in the discussion of this exemplary operations of CIPU 400, in the embodiment but realizing at some softwares, change the modules that parameter is sent to CIPU400 into.

In certain embodiments, controller module 475 postpone special time amount after pel array 410a fetches image, carrys out initialization sensor assembly 415 by instruction sensor module 415.In other words, controller module 475 instruction sensor modules 415 are fetched image with special speed from pel array 410a.

Subsequently, controller module 475 is by sensor assembly 415 instruction camera sensor 405a photographic images.In certain embodiments, controller module 475 also provides exposure parameter and other camera operation parameter to camera sensor 405a.In other embodiments, camera sensor 405a uses the default value of camera sensor operating parameter.According to described parameter, camera sensor 405a takes original image, and described original image is stored in pel array 410a.Sensor assembly 415 is fetched described original image from pel array 410a, and image is sent to row/frame buffer 417 to preserve, and CIPU processing pipeline 485 is processed this image afterwards.

In some cases, image may be abandoned by row/frame buffer 417.When camera sensor 405a and/or 405b are during with two-forty photographic images, the comparable BPC module 420 of sensor assembly 415 can be fetched image from row/frame buffer 417 and be received quickly image, and Image Saving in be expert at/frame buffer 417 (for example, while taking high frame rate video), OK/frame buffer 417 can become completely and be full of.In the time that this thing happens, row/frame buffer 417 of some embodiment abandons image (, frame) according to first in first out.,, in the time that row/frame buffer 417 abandons a two field picture, OK/frame buffer 417 abandons that two field picture receiving before all other images in be expert at/frame buffer 417.

The image of CIPU processing pipeline 485 is processed from BPC module 420 and is fetched voluntarily/image of frame buffer 417, starts with any bad pixel in correcting image.BPC module 420 sends to LS module 425 image subsequently, with any non-uniform illumination degree in correcting image.After the illuminance of correcting image, LS module 425 sends to demosaicing module 430 image, and demosaicing module 430 is processed original image, produces RGB image by original image.Subsequently, WB module 435 receives RGB image from demosaicing module 430, and adjusts the white balance of RGB image.

As mentioned above, statistics engine 465 may have been collected some data at each point of CIPU processing pipeline 485.For example, as shown in diagram in Fig. 4, statistics engine 465 is collected data after LS module 425, demosaicing module 430 and WB module 435.According to the data of collecting, the operation of the one or more modules in operation and/or the CIPU processing pipeline 485 of statistics engine 465 capable of regulating camera sensor 405a, to adjust from the shooting of the successive image of camera sensor 405a.For example, according to the data of collecting, statistics engine 465 can determine that the exposure of present image is too low, thereby increases the exposure of follow-up clapped image by sensor assembly 415 instruction camera sensor 405a.Thereby the statistics engine 465 of some embodiment plays some and processes the feedback loop operating.

After WB module 435 is adjusted the white balance of image, it sends to γ module 440 image, for example, to carry out γ correction (, adjusting the gamma curve of image).CSC module 445 receives the image γ proofreaies and correct, the circumstances in which people get things ready for a trip of going forward side by side spatial alternation from γ module 440.In this example, CSC module 445 becomes YUV image RGB image conversion.In other words, CSC module 445 is becoming the image with yuv color space performance with the image conversion of RGB color space performance.HSC module 450 receives YUV image from CSC module 445, and adjusts tone, saturation and the contrast properties of each pixel in image.After HSC module 450, scaler module 455 zoomed images (for example, zooming in or out image).After scaler module 455 receives image, filter module 460 is to one or more filtering of image applications.Finally, filter module 460 sends to CIPU driver 480 image after treatment.

In this example of operation of above-described CIPU 400, the each module in CIPU processing pipeline 485 is processed image in some way.But, other image that CIPU 400 processes may not need the processing of all modules of CIPU processing pipeline 485.For example, image may not need blank level adjustment, γ correction, proportional zoom or filtering.Thereby CIPU 400 can be according to the various inputs that receive, such as the instruction from CIPU driver 480, or the data of being collected by statistics engine 465, with any treated in various ways image.

Different embodiment differently controls the speed (, frame rate) of processing image.A kind of mode of control frame speed is by the manipulation of vertical blanking interval (VBI).For fetching image line to process line by line some embodiment of image, VBI fetches the last column by an image of the video of the camera of two camera movement equipment from pel array, and fetches from pel array the time difference between the first row of next image of video.In other embodiments, VBI fetches an image by the video of the camera of two camera movement equipment from pel array, and fetches from pel array the time difference between the next image of video.

Wherein can use an example of VBI between sensor assembly 415 and pel array 410a and 410b.For example, some embodiment of sensor assembly 415 fetch image from pel array 410a and 410b line by line, and other embodiment of sensor assembly 415 fetches image from pel array 410a and 410b one by one image.Thereby, by adjusting the VBI of sensor assembly 415, can control frame speed: increase VBI and can reduce frame rate, can improve frame rate and reduce VBI.

The application of 1.VBI to single camera: frame rate control

Fig. 5 concept map explains orally Benq in the example of the different frame rates 505,510 and 515 of different VBI.Each sequence is presented at each moment 525-555 along timeline 520, holds the personage's of guitar image, and described image is taken by one of camera of two camera movement equipment.In addition, the time between each moment 525-555 is identical, is called a time quantum.For the purpose of illustrating, referring now to sensor assembly 415 and the pel array 410a of Fig. 4, key diagram 5.Thereby each image represents the moment of fetching image from pel array 410a along the sensor assembly 415 of timeline 520.

In the frame rate 505 of example, sensor assembly 415 is set to 3 time quantums (for example, being set by controller module 475) about the VBI of pel array 410a., sensor assembly 415, along timeline 520, is fetched a two field picture every two moment from pel array 410a.As shown in the frame rate 505 of example, sensor assembly 415 is fetched image in the moment 525,540 and 555.Thereby the frame rate 505 of example has the frame rate of every three time quantum one two field pictures.

Except VBI is set to 2 time quantums, the frame rate 510 of example is similar to the frame rate 505 of example.Thereby sensor assembly 415, along timeline 520, is fetched a two field picture every a moment from pel array 410a.The frame rate 510 of example represents that sensor assembly 415 is in the moment 525,535,545 and 555, and from pel array, 410a fetches image.Because the VBI of the frame rate 510 of example is less than the VBI of the frame rate 505 of example, therefore the frame rate of the frame rate 510 of example is higher than the frame rate of the frame rate 505 of example.

Except sensor assembly 415 is set to 1 time quantum about the VBI of pel array 410a, the frame rate 515 of example is also similar to the frame rate 505 of example.So sensor assembly 415 along timeline 520, is fetched a two field picture in each moment from pel array 410a by instruction.As shown in the figure, sensor assembly 415 is at moment 525-555, and from pel array, 410a fetches image.The VBI of the frame rate 515 of example is less than the VBI of the frame rate 505 and 510 of example.So the frame rate of the frame rate 515 of example is greater than the frame rate 505 and 510 of example.

The application of 2.VBI to two cameras

Some embodiment may wish to operate two cameras (for example,, during video conference, transmitting video from two cameras) of two camera movement equipment simultaneously.The different embodiment that comprise two camera movement equipment of single processing pipeline provide the different mechanisms of two cameras of the two camera movement equipment of operation simultaneously.

Such mechanism is the VBI by controlling every camera, interweaves by the processing of the image of two camera.That is, during the VBI of a camera, take and process the one or more images by another camera, vice versa.Because CIPU 400 described above has single processing pipeline 485, therefore this mechanism can realize in the CIPU of some embodiment 400.In such embodiments, sensor assembly 415 is fetched image from one of pel array 410a and 410b, and the image of fetching is processed during the VBI about one other pixel array at sensor assembly 415 by CIPU 400.

Sensor assembly 415 can be set to particular value about the VBI of each pel array.But, in certain embodiments, VBI is not configured to fetch and process a little value of two field picture time used than CIPU 400.Some embodiment set identical value sensor assembly 415 for about the VBI of each pel array.For example, in the time that sensor assembly 415 is configured to identical value about the VBI of each pel array, sensor assembly 415 is alternately fetched image from pel array 410a and 410b.Other embodiment sets different values sensor assembly 415 for about the VBI of each pel array.In some such embodiment, sensor assembly 415 is configured to the multiple of sensor assembly 415 about the VBI of one other pixel array about the VBI of a pel array.For example, sensor assembly 415 is configured to 2 time quantums about the VBI of a pel array, and sensor assembly 415 is configured to 4 time quantums about the VBI of one other pixel array.In this example, each two field picture of fetching from described one other pixel array with respect to sensor assembly 415, sensor assembly 415 is fetched two two field pictures from a described pel array.

Fig. 6 concept map explains orally Benq in the example of the different interleaving frame rate 605,610 and 615 of different VBI.Except Fig. 6 comprises that Fig. 6 is similar to Fig. 5 along 13 moment 625-685 of timeline 620.In addition, hold the personage's of guitar image representative and fetches the moment of this image along timeline 620 from a pel array, and the image that has on the personage of trencher cap represents the moment of fetching image along timeline 620 from one other pixel array.

For the purpose of illustrating, hold the personage's of guitar image supposition and taken by the camera sensor 405a of two camera movement equipment, and the image supposition that has on the personage of trencher cap is taken by the camera sensor 405b of two camera movement equipment.In addition, referring now to sensor assembly 415 and pel array 410a and the 401b key diagram 6 of Fig. 4.

In the interlacing frames speed 605 of example, sensor assembly 415 is configured to 2 time quantums about the VBI of pel array 410a and pel array 401b.As shown in the interlacing frames speed 605 of example, sensor assembly 415 is in the moment 625,635,645,655,665,675 and 685 along timeline 620, from pel array, 410a fetches image, and sensor assembly 415 is in the moment 630,640,650,660,670 and 680 along timeline 620, from pel array, 410b fetches image.,, at each time quantum, sensor assembly 415 is alternately fetched image from pel array.

Except sensor assembly 415 is configured to 4 time quantums about the VBI of pel array 410a and pel array 401b, the interlacing frames speed 610 of example is similar to the interlacing frames speed 605 of example.The interlacing frames speed 610 of example shows that sensor assembly 415 is in the moment 625,645,665 and 685 along timeline 620, from pel array, 410a fetches image, and sensor assembly 415 is in the moment 635,655 and 675 along timeline 620, from pel array, 410b fetches image.Because the VBI of the interlacing frames speed 610 of example is greater than the VBI of the interlacing frames speed 605 of example, therefore the frame rate of the interlacing frames speed 610 of example is less than the frame rate of the interlacing frames speed 605 of example.

Except sensor assembly 415 is configured to 6 time quantums about the VBI of pel array 410a and pel array 401b, the interlacing frames speed 615 of example is also similar to the interlacing frames speed 605 of example.As shown in Figure 6, sensor assembly 415 is in the moment 625,655 and 685 along timeline 620, and from pel array, 410a fetches image, and sensor assembly 415 is in the moment 640 and 670 along timeline 620, and from pel array, 410b fetches image.The VBI of the interlacing frames speed 615 of example is greater than the VBI of the interlacing frames speed 605 and 610 of example.Thereby the frame rate of the interlacing frames speed 615 of example is less than the frame rate of the interlacing frames speed 605 and 610 of example.

B. multi-line

Fig. 7 is another photographic images processing unit (CIPU) 700 of some embodiment of graphic extension conceptually.Except CIPU 700 is by two front-end processing pipelines, a memory, and a back-end processing pipeline realization, instead of outside being realized by single processing pipeline, CIPU 700 realizes the function identical with CIPU described above 400.Therefore,, with reference to the module of CIPU 400, the description of the function of CIPU 700 is described.

As shown in the figure, CIPU 700 comprises for the front-end processing pipeline 715 of camera sensor 405a and pel array 410a, front-end processing pipeline 720, memory 725, controller module 730 and back-end processing pipeline 735 for camera sensor 405b and pel array 410b.The camera sensor 405a of some embodiment and 405b are the transducers of the camera of two camera movement equipment.

The front-end processing pipeline 715 and 720 of some embodiment is carried out the parts of images processing of CIPU 400.Thereby different embodiment can comprise the module of the different numbers of CIPU 400.For example, each front-end processing pipeline 715 and 720 of some embodiment comprises sensor assembly 415, BPC module 420, LS module 425, demosaicing module 430, WB module 435 and the statistics engine 465 of CIPU 400.

Although owing to thering is identical module, front-end processing pipeline 715 and 720 is carried out the image processing of same type, but, by the different register value as above illustrating about CIPU 400, can differently configure the each module in each front-end processing pipeline 715 and 720.In addition,, because each camera sensor 405a and 405b have its front-end processing pipeline, therefore front-end processing pipeline 715 and 720 can be processed image independently of each other.For example, front-end processing pipeline 715 and 720 concurrently (, in the identical time), in the different time with different rate processing images.

In certain embodiments, each front-end processing pipeline 715 and 720 can be fetched image from its corresponding camera sensor and pel array.For example, front-end processing pipeline 715 is fetched the image of being taken by camera sensor 405a from pel array 410a, and front-end processing pipeline 720 is fetched the image of being taken by camera sensor 405b from pel array 410b.In the time that one of front end processing pipeline 715 and 720 is fetched image from its corresponding camera sensor and pel array, this this image of front-end processing pipeline, and image after treatment is sent to memory 725.In addition, each front-end processing pipeline 715 and 720 communicate by letter with controller module 730 as mentioned above (for example,, by the statistics engine of each front-end processing pipeline).

Image after the memory 725 storage unit divisional processing of some embodiment, completes processing for back-end processing pipeline 735.In these embodiments, memory 725 is from front end processing pipeline 715 and 720 receiving unit image after treatment, and the image after section processes is sent to back-end processing pipeline 735.Some embodiment (are for example embodied as volatile memory memory 725, random access memory (RAM)), for example, and other embodiment is embodied as nonvolatile memory (, flash memory, hard disk, CD etc.) memory 725.In addition, the memory 725 of some embodiment is internal storage (for example, RAM), and the memory 725 of other embodiment be external memory storage (for example, compact flash (CF) card, secure digital (SD) card, etc.).

Some embodiment of back-end processing pipeline 735 carry out the parts of images processing of CIPU 700.In certain embodiments, back-end processing pipeline 735 comprises the module of front-end processing pipeline 715 and 720 CIPU 400 not to be covered.For example, referring to example above, back-end processing pipeline 735 should comprise CSC module 445, γ module 440, HSC module 450, scaler module 455 and the filter module 460 of CIPU 400.Thereby the back-end processing pipeline 735 of this embodiment is carried out the residual image processing of the CIPU 400 that front-end processing pipeline 715 and 720 do not carry out.Therefore, back-end processing pipeline 735 is fetched the image section processes from memory 725, and the image after section processes is carried out to remaining image processing.After processing image, back-end processing pipeline 735 sends to CIPU driver 480 image after treatment.

Controller module 730 is carried out the identical function above with reference to Fig. 4 explanation.As shown in Figure 7, controller module 730 and front-end processing pipeline 715 and 720 and back-end processing pipeline 735 mutual.In certain embodiments, controller module 730 is included in back-end processing pipeline 735, and in other embodiments, controller module 730 is included in one of front-end processing pipeline 715 and 720.

With reference now to camera sensor 405a and 405b, pel array 401a and 410b, front-end processing pipeline 715 and 720, memory 725 and the back-end processing pipeline 735 of graphic extension in Fig. 7,, the operation of CIPU 700 is described.In the time that one of front end processing pipeline 715 and 720 is fetched image from its corresponding camera sensor and pel array, this image of front-end processing pipeline, and the image after section processes is sent to memory 725.For example, front-end processing pipeline 715 can be fetched the image of being taken by camera sensor 405a from pel array 410a, or front-end processing pipeline 720 can be fetched the image of being taken by camera sensor 405b from pel array 410b.As mentioned above, front-end processing pipeline 715 and 720 can both parallel processing image.

Back-end processing pipeline 735 is fetched the image section processes from memory 725, and processing section image after treatment is to complete the image processing of image.In certain embodiments, the image being kept in memory 725 is fetched and processed to the principle of back-end processing pipeline 735 based on first in first out.In other words, specific image in memory 725 will be processed in the reception prior to this specific image with after being kept at all images in memory 725, but, this specific image will be processed before being later than the reception of this specific image and being kept at the image in memory 725.After back-end processing pipeline 735 is processed image, it sends to CIPU driver 480 image after treatment.

Fig. 8 is another photographic images processing unit (CIPU) 800 of some embodiment of graphic extension conceptually.Except CIPU 800 is realized by two independent processing pipeline, each camera sensor has outside its independent processing pipeline, and CIPU 800 carries out the function identical with CIPU described above 400.Thereby, with reference to the module of CIPU 400, the description of the function of CIPU 800 is described.

As shown in the figure, CIPU 800 comprises the processing pipeline 815 for camera sensor 405a and pel array 410a, and for the processing pipeline 820 of camera sensor 405b and pel array 410b.Each processing pipeline 815 and 820 of some embodiment comprises all modules that are included in CIPU 400.So each processing pipeline 815 of these embodiment is all identical with the operation of CIPU 400 with 820 operation.

Because each camera sensor 405a and 405b have its processing pipeline, therefore processing pipeline 815 and 820 can be processed image independently of each other.For example, processing pipeline 815 and 820 concurrently (, in the identical time), in the different time with different rate processing images.In addition, by the different register value illustrating above with reference to CIPU 400, can differently configure each processing pipeline 815 and 820 of some embodiment.

In certain embodiments, the number of modules of CIPU 400 comprises one or more row/frame buffers of some or all operations for carrying out this module.For example, the filtration module 460 of some embodiment is realized as and carries out 3 × 3 low-pass filtering.In such embodiments, 3 × 3 low pass filters are processed continuous three row in image, to row application 3 × 3 low-pass filtering in the middle of described continuous three row.Thereby the filtration module 460 of this embodiment at least needs three row/frame buffers, to realize 3 × 3 low-pass filtering.Other module in CIPU 400 also comprises one or more row/frame buffers, such as BPC module 420 and LS module 425.

The processing pipeline of CIPU 800 all can have different row/frame buffer sizes, so that according to the characteristic customization image processing of its corresponding camera.For example, if a camera of two camera movement equipment has 2048 × 1500 element sensors, the processing pipeline of this 2048 × 1500 element sensor can comprise row/frame buffer that 2048 pixels are wide so.Similarly, if another camera of two camera movement equipment has 640 × 480 element sensors, the processing pipeline of 640 × 480 element sensors can be included as the wide row/frame buffer of 640 pixels so.The large I that, is included in the row/frame buffer in the modules of a processing pipeline is different from the size of the row/frame buffer in the modules that is included in another processing pipeline.

III. video conference

A. video conference architecture

Fig. 9 is the video conference of two camera movement equipment of some embodiment of graphic extension and the software architecture of processing module 900 conceptually.Video conference and processing module 900 comprise and the respective modules and the similar CIPU driver 905 of driver 305,301 and 320, exchange of media module 910 and the encoder-driven device 920 that illustrate above with reference to Fig. 3.Video conference and processing module 900 also comprise video conference module 925, video conference client 945 and the network interface 950 for realizing various video conference functions.Be similar to Video processing and coding module 300, video conference and processing module 900 are processed and are encoded from the image of the camera of two camera movement equipment.

As referring to Figure 3 as described above, the user of the media content in exchange of media module 910 permission equipment and producer's switched-media content, with the instruction relevant with the processing of media content.CIPU driver 905 serves as the communication interface with photographic images processing unit (CIPU) 955, encoder-driven device 920 serve as with encoder hardware 960 (for example, encoding pack in encoder chip, SOC (system on a chip), etc.) communication interface.

The video conference module 925 of some embodiment is responsible for various video conference functions, such as image processing, video conference management and networking.As shown in the figure, video conference module 925 is mutual with exchange of media module 910, video conference client 945 and network interface 950.In certain embodiments, video conference module 925 receives instruction from video conference client 945, and sends instruction to video conference client 945.The video conference module 925 of some embodiment is also by network interface 950, send data and receive data from network to network, for example, described network is local area network (LAN) (LAN), WLAN (wireless local area network) (WLAN), wide area network (WAN), network of network (a network of networks), code division multiple access access (CDMA) network, GSM network, etc.

Video conference module 925 comprises image processing layer 930, management level 935 and network layer 940.In certain embodiments, image processing layer 930 carries out image processing operations to image, for use in video conference.For example, the image processing layer 930 of some embodiment expose adjustment, image scaling, perspective correction and dynamic range adjustment, following further in detail described in.The image processing layer 930 of some embodiment sends the request to the image from CIPU955 by exchange of media module 910.

The management level 935 of some embodiment are controlled the operation of video conference module 925.For example, in certain embodiments, one/two cameras of the two camera movement equipment of management level 935 initialization, process image and audio frequency, to send remote equipment to, and process the image and the audio frequency that receive from remote equipment.In certain embodiments, management level 935 for example produce, for synthetic (PIP) of this equipment and show.In addition, management level 935 can, according to the networking report receiving from network layer 940, change the operation of video conference module 925.

In certain embodiments, some or all network functions that network layer 940 realizes for video conference.For example, as described below, except other function, the network that the network layer 940 of some embodiment is set up between two camera movement equipment and the remote equipment of video conference connects (not shown), image is sent to remote equipment, and receive image from remote equipment.In addition, network layer 940 receives the networking data such as packet loss, unidirectional stand-by period and round trip delay time, and other various data, processes such data, and data report to management level 935.

The video conference client 945 of some embodiment is to utilize the application of the video conference function of video conference module 925, for example, such as video conference application, IP speech (VOIP) application (, Skype) or instant messaging application.In certain embodiments, video conference client 945 is independently to apply, and in other embodiments, video conference client 945 is integrated in Another application.

In certain embodiments, network interface 950 is the communication interfaces that allow video conference module 925 and video conference client 945 for example, to send data by network (, cellular network, local area network (LAN), wireless network, network of network, internet etc.) and receive data.For example, for example, if video conference module 925 to another equipment sending data on internet (is wanted, by the image of the camera of two camera movement equipment), video conference module so 925 sends to described another equipment by network interface 950 image.

B. video conference setting

Figure 10 is the video conference request message sending and receiving sequence 1000 of the example of some embodiment of graphic extension conceptually.Figure 10 represents the video conference client 1010 on the equipment that runs on 1005, videoconference server 1015, and run on the video conference request message sending and receiving sequence 1000 between the video conference client 1025 on equipment 1020.In certain embodiments, video conference client 1010 and 1025 is identical with the video conference client 945 shown in Fig. 9.As shown in Figure 10, an equipment (, equipment 1005) request video conference, and another equipment (, equipment 1020) is replied this request.Two camera movement equipment of describing in this application can be carried out these two operations (, the request of sending and response request).

The videoconference server 1015 of some embodiment route messages between video conference client.Although some embodiment realize videoconference server 1015 on a computing equipment, but other embodiment realizes videoconference server 1015 on multiple computing equipments.In certain embodiments, videoconference server is the server that can openly access, and it can process the message with the numerous meetings of route simultaneously.Each video conference client 1010 and 1025 of some embodiment is through network interface, such as network interface 950 described above, for example, communicate by letter with videoconference server 1015 by network (, cellular network, local area network (LAN), wireless network, network of network, internet etc.).

When video conference client 1010 receives the request of video conference of (in operation 1) beginning and equipment 1020 from the user of equipment 1005, the video conference request message sending and receiving sequence 1000 of some embodiment starts.In the time that the user of equipment 1005 selects to be presented at a certain user interface (UI) project of the user interface on equipment 1005, the video conference client 1010 of some embodiment receives the request that starts video conference.In Figure 11 of explanation and Figure 14, illustrate below the example of this user interface.

After video conference client 1010 receives request, video conference client 1010 sends (in operation 2) video conference request to videoconference server 1015, and this request, according to user's input, is designated as addressee equipment 1020.Videoconference server 1015 is transmitted to video conference request the video conference client 1025 of (in operation 3) equipment 1020.In certain embodiments, videoconference server 1015 is utilized push technology, and video conference request is transmitted to video conference client 1025.That is, when receive request from video conference client 1010, videoconference server 1015 starts to transmit video conference request to video conference client 1025, instead of waits for that client 1025 sends the request to any message.

In the time that the video conference client 1025 of some embodiment receives video conference request, on equipment 1020, show user interface, to point out to the user of equipment 1020, the user of equipment 1005 has sent the request that starts video conference, and the user of prompt facility 1020 accepts or refuse this video conference request.In Figure 12 of explanation, illustrate below the example of this user interface.In certain embodiments, in the time that video conference client 1025 receives (in operation 4) and accepts the request from the user's of equipment 1005 video conference request, video conference client 1025 sends (in operation 5) video conference to videoconference server 1015 and accepts.In the time that the user of equipment 1020 selects a certain user interface items of user interface as illustrated in Figure 12, the video conference client 1025 of some embodiment receives the request of accepting video request.

Receive video conference acceptance from video conference client 1025 in videoconference server 1015 after, videoconference server 1015 accepts to be transmitted to (in operation 6) video conference client 1010 video conference.Some embodiment of videoconference server 1015 utilize above-mentioned push technology, and video conference is accepted to be transmitted to video conference visitor corpse end 1010.

In the time receiving video conference and accept, some embodiment set up (in operation 7) video conference between equipment 1005 and equipment 1020.Different embodiment differently sets up video conference.For example, the video conference of some embodiment is set up and is comprised: connection between consulting device 1005 and equipment 1020, determine the bit rate to Video coding and exchange video at equipment 1005 and equipment 1020.

In the above example, the user of equipment 1020 accepts video conference request.In certain embodiments, equipment 1020 can be configured to (for example, by the preferential selection setting of equipment) and automatically accept the video conference request arriving, and does not show UI.In addition, the user of equipment 1020 also can refuse (in operation 4) video conference request (for example,, by being chosen in certain user interface items of the user interface showing on equipment 1020).Replace and send video conference acceptance, video conference client 1025 sends video conference refusal to videoconference server 1015, and videoconference server 1015 is transmitted to video conference client 1010 video conference refusal.Thereby do not set up video conference.

1. video conference arranges user interface

In certain embodiments, according to ongoing call, initiate video conference.,, in the user and second user's communication of mobile device, through the opposing party's license, this user can become video conference call.Concerning some embodiments of the present invention, the startup of this video conference that Figure 11 graphic extension is undertaken by two camera handheld mobile devices 1100.Figure 11 utilizes five operational phases 1110,1115,1120,1125 and 1130 of the user interface (" UI ") 1105 of equipment 1100, the startup of graphic extension video conference.

As shown in Figure 11, UI 1105 comprises name field 1135, choice menus 1140 and selectable UI project 1145.The user that name field 1135 is presented at the call other end is intended to request and carries out with it the personage's of video conference name.In this example, selectable UI project 1145 (it can be realized as selectable button) provides selectable end call (End Call) option to user, to finish call.Choice menus 1140 shows the menu of selectable UI project, such as speakerphone project 1142, quiet project 1144, numeric keypad project 1146, telephone directory project 1148, maintenance project 1152, video conference project 1154 etc.Different embodiment differently shows choice menus.Concerning the embodiment of Figure 11 graphic extension, choice menus 1140 comprises several onesize icons, and each icon represents a different operation.Other embodiment provides rotatable menu, or gives specific project with priority (for example,, by making these projects become larger).

Referring now to the state in the double teacher 1110,1115,1120,1125 and 1130 of UI 1105 graphic extension in Figure 11, the operation of UI 1105 is described.In the first stage 1110, between handheld mobile device user and Nancy Jones, set up call.Second stage 1115 be presented at user select selectable video conference option one 154 (for example, by point 1150 singly refer to dub), to activate the UI 1105 after video conference instrument.In this example, video conference option one 154 (it can be realized as selectable icon) allows user during conversing, to start video conference.In second stage, video conference option one 150 is highlighted, is activated with instruction video meeting instrument.Different embodiment can for example,, with different modes (, by highlighting frame or the text of project), indicate such selection.

Phase III 1120 is presented at according to the selection of selecting video conference option one 154, and equipment 1100 has started video conference and processed UI 1105 afterwards.Phase III is the transition maintenance stage of in the time that video conference is set up in equipment wait (for example,, in the time that equipment waits for that the equipment of the call other end is accepted or refuses video conference).In the phase III 1120, in setting up video conference connection, the user of equipment 1100 still can talk with the user of another equipment (, Nancy Jones).In addition, some embodiment allow the user by selecting of equipment 1100 be presented on UI 1105 for cancelling the optional UI project (not shown) of video conference request, in the phase III 1120, cancel video conference request.During this maintenance stage, different embodiment shows to indicate this pending state by the difference in UI 1105.

As shown in Figure 11, in certain embodiments, utilize the full screen display of the video taken by equipment 1100, and in the wait state of " Preview (preview) " symbol graphic extension phase III of the bottom of this video.Specifically, in Figure 11, the phase III 1120, demonstration was presented by the full frame of video of the camera of equipment, the beginning of graphic extension video conference processing by the viewing area 1160 of UI 1105.In certain embodiments, positive camera is in the time starting video conference, the acquiescence camera that equipment is selected.Conventionally,, in the time starting video conference, described front camera is facing to the user of equipment.Therefore, in Figure 11, in the example of graphic extension, the phase III 1120 is illustrated as equipment 1100 user's of display device 1100 full screen video.During being located at the phase III 1120, appear at the wait state of the further outstanding equipment of " Preview " instruction under the video in viewing area 1160.

In certain embodiments, can differently represent cambic the 3rd maintenance stage 1120.For example, some embodiment allow the user of equipment 1100 to select the camera at the back side as the camera that starts video conference.In order to allow this selection, some embodiment allow user (for example, preferentially select to arrange by menu) camera at the back side is appointed as and is started the acquiescence camera of video conference, and/or allow user from after selecting video conference option one 154 user, show the camera of selecting the back side in the menu of the back side and positive camera.In these situations any, UI 1105 (for example, viewing area 1160), during the 3rd maintenance stage 1120, shows the video by the camera at the back side.

In addition, other embodiment can be by the less version of display device 1100 video of taking the photograph, be kept at the rest image on equipment 1100 by demonstration, by provide the equipment of highlighting wait state message (for example, by showing " Conference Being Established " (setting up meeting), by not showing " Preview " instruction, etc., the activation of video conference instrument can be pointed out.In addition, in the phase III 1120, if the UI of some embodiment 1105 provide this stage (for example, in the time that user waits for that long-distance user answers his request), user determines not enter video conference, allow so user to cancel and enter video conference, and return to end (End) the button (not shown) of talking state.

Fourth stage 1125 is illustrated in long-distance user and has accepted video conference request, and after having set up video conference connection, UI 1105 in an interim state.Under this transition state, show that the size of the viewing area 1160 of local user's video (in this example, by the video of positive camera) reduces (that is, dwindling gradually) gradually, as shown in arrow 1175.Viewing area 1160 (, local user's video) dwindles, and makes the UI 1105 can be after viewing area 1160, shows the viewing area 1170 (for example, display window 1170) comprising from the video of the camera of remote equipment.In other words, the generation PIP that dwindles of local user's video 1160 shows that 1180, PIP shows that the 1180 prospects insertions with local user's video show 1160, and the main demonstration 1170 of long-distance user's background.In this example, the Ms that the front camera that the main demonstration 1170 of background presents remote equipment is being taken its video (for example, Nancy Jones, the user of remote equipment), or the back side camera of remote equipment is being taken Ms's's (for example, Nancy Jones is taking the Ms of its video) of its video video.Those of ordinary skill will appreciate that the cambic fourth stage shown in Figure 11 is a kind of exemplary method that some embodiment use, and other embodiment is the cambic fourth stage of flash demo differently.

Fourth stage 1125 is the optional UI project 1132 of graphic extension in viewing area 1155 below also.Optional UI project 1132 (it can be realized as selectable button) shows under 1180 selectable closing session (End Conference) option one 132 is provided at PIP.User can select closing session option one 132 to finish video conference (for example, by singly refer to dub).Different embodiment can allow user with different mode closing sessions, such as by the switch on conversion mobile device, and by sending voice commands, etc.In addition, different embodiment can allow closing session option one 132 to fade away during video conference, thereby allows PIP to show that 1180 occupy whole viewing area 1185.In the time singly referring to dub the bottom of viewing area 1185, closing session option one 132 can reappear subsequently, makes user can utilize closing session option one 132.In certain embodiments, the layout of viewing area 1155 and the viewing area 1155 that further describes are below identical.

Five-stage 1130 is illustrated in the UI 1105 after the animation that finishes the 4th transition state 1125.Specifically, during five-stage 1130 is illustrated in video conference, the PIP being presented by UI1105 shows 1180.As mentioned above, PIP shows that 1180 comprise that two videos show: the larger background from remote camera shows 1170, and inserts and show 1160 from the less prospect of local camera.

PIP shows that 1180 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Except this synthetic view, the equipment of some embodiment provides other synthetic view.For example, replace the larger background with long-distance user to show 1170, larger background shows that 1170 can be local user, and the insertion of less prospect shows that 1160 is long-distance users.As further described below, some embodiment allow user during video conference, between the local camera of the camera as providing PIP to show 1180 insertion view and front view and/or remote camera, switch.

In addition, some embodiment allow local video and long-distance video to appear in UI 1105 two viewing areas (for example, left and right display window, or upper and lower display window) side by side, or in the viewing area of arranging, two diagonal angles.In certain embodiments, as further described below, by the preferential selection setting of equipment, or by during video conference, the control that user can select, user can specify that PIP shows or the mode of acquiescence display mode.

In the time that the user of the equipment 1100 of Figure 11 invites long-distance user to carry out video conference, long-distance user can accept or refuse this invitation.Figure 12 is illustrated in six different stages 1210,1215,1220,1225,1230 and 1235, the UI 1205 of long-distance user's equipment 1200, and the equipment that described six phase table are shown in long-distance user presents and accepts the sequence of operation of video conference invitation.The description of UI 1205 below equipment 1200 (, receive the equipment of video conference request) user be called invitee (invite recipient), the user of equipment 1100 (, sending the equipment of video conference request) is called to inviter (invite requestor).In addition, in this example, the equipment 1200 of supposing invitee is two camera apparatus, the same with inviter's equipment.But, in other example, one or two in these equipment are all single camera apparatus.

First stage 1210 graphic extension is worked as invitee from inviter, the UI 1205 when John Smith receives video conference invitation.As shown in Figure 12, the UI 1205 of first stage comprises name field 1235, message hurdle 1240 and two optional UI projects 1245 and 1250.Name field 1235 shows the personage's who just asks video conference name.In certain embodiments, name field 1235 shows the personage's who just asks video conference telephone number, instead of personage's name.Message hurdle 1240 shows the invitation of inviter to invitee.In this example, the instruction of " Video Conference Invitation (the video conference invitation) " in message hurdle 1240 inviter is asking to carry out video conference with invitee.Optional UI project 1245 and 1250 (they can be realized as selectable button) provides the selectable invitee of confession " refusal request (Deny Requst) " and " (Accept Request) accepts request " option one 245 and 1250 for refusing or accepting the invitation.Different embodiment can differently show these options and/or show other option.

In the time seeing " the Video Conference Invitation " symbol being presented in message hurdle 1240, by selecting respectively " refusal request " option one 245 or " accepting request " option one 250 in UI, invitee can refuse or accept request.Second stage 1215 is illustrated in the example shown in Figure 12, and user selects " accepting request " option one 250.In this example, click gently " accepting request " option one 250 by user's finger, realize described selection, and by the highlighting of option one 250, point out described selection.Option one 245 and 1250 (for example to provide in certain embodiments other technology to select " acceptance " or " refusal request ", double click gently, etc.) for example, to point out described selection (, highlighting frame or the text of UI project).

Phase III 1220 is presented at invitee and adheres to video conference UI 1205 afterwards.In this stage, UI 1205 enters preview mode, and preview mode shows from the full frame of video of the front camera of remote equipment and presents in viewing area 1244.Front camera is in this case facing to the user (, the Nancy Jones in this example) of remote equipment.Therefore, show her image by described preview mode.This preview mode makes invitee can guarantee that her video is correctly shown, and for example, before video conference starts (, before starting actual transmissions video), she pleases oneself to its appearance.In certain embodiments, can under viewing area 1244, show the symbol such as " Preview " symbol, to point out that invitee is in preview mode.

Some embodiment allow invitee to select the camera at the back side as the acquiescence camera that starts video conference, or in the time starting video conference, select the camera at front or the back side, as further described below.In addition, other embodiment differently shows that invitee's preview shows (for example, being placed in the less image in corner of viewing area 1244).In addition other embodiment does not comprise this preview mode, but after invitee accepts request, starts immediately video conference.

In the phase III, UI 1205 shows two optional UI projects 1275 and 1246.One of them overlaps on viewing area 1244, and another is below viewing area 1244.Optional UI project 1275 is that user can select to start video conference " acceptance " button 1275.Optional UI project 1246 is if invitee determines not add video conference in this stage, so her selectable " end " button 1246.

Fourth stage 1225 is presented at invitee and selects " acceptance " button 1275 UI1205 afterwards.In this example, " acceptance " button 1275 is highlighted, to point out that invitee can start video conference at any time.Can point out such selection by different modes in other embodiments.

Five-stage 1230 is illustrated in after invitee accepts video conference request, UI 1205 in an interim state.In this transition stage, show that the size of the viewing area 1244 of invitee's video (in this example, described video is by positive camera) reduces (that is, dwindling gradually) gradually, as shown in arrow 1260.Invitee's video dwindles, and makes the UI 1205 can be after viewing area 1244, shows the viewing area 1265 (for example, display window 1265) comprising from the video of inviter's camera.In other words, the generation PIP that dwindles of invitee's video shows that 1280, PIP shows that 1280 have the prospects insertion viewing area 1244 of invitee's video, and the main demonstration 1265 of inviter's background.

In this example, the front camera that the main demonstration 1265 of background presents local device is being taken the man's (, John Smith, the user of local device 1100) of its video video.In another example, this video can be man's's (for example, being taken the man of its video by John Smith) of the back side its video of camera of local device video.The differently differently cambic five-stage of flash demo of embodiment.

During the UI of five-stage 1230 also shows and is included in video conference, make another user audio mute optional UI project 1285 (for example, mute button 1285), (for example finish the optional UI project 1287 of video conference, closing session button 1287), the viewing area 1155 (for example, tool bar or menu bar) of the optional UI project 1289 (for example, switching camera button 1289) of the switching camera further illustrating below.For example, thereby invitee can select any optional UI project 1285-1289 (, by singly refer to dub), to carry out the operation of wishing during video conference.Different embodiment allows invitee according to different modes, for example, switch the switch on mobile device, by providing voice commands etc., carries out operation arbitrarily.

Although Figure 12 represents the example layout of viewing area 1155, but, some embodiment provide the different layouts of viewing area 1155, and such as the layout of the viewing area 1155 of Figure 11, this layout only includes the optional closing session UI option one 132 for finishing video conference.Other layout of viewing area 1155 can comprise any multiple different optional UI project for carrying out difference in functionality.In addition, five-stage 1230 represents the viewing area 1155 of the bottom that is presented at UI 1205.The different embodiment of viewing area 1155 can be displayed on the diverse location in UI 1205, and/or are defined as different shapes.

Figure 12 is expressed as static viewing area (, viewing area 1155 is always shown) viewing area 1155.But, in certain embodiments, viewing area 1155 is dynamic display area.In some such embodiment, viewing area 1155 is usually not shown.On the contrary, for example only have, in the time receiving trigger event (, such as dubbing viewing area 1280 once, the user of voice commands and so on selects), just show described viewing area 1155.(for example select receiving user, select optional quiet UI project 985), or the time of ormal weight (for example, 3 seconds) afterwards, viewing area 1155 disappears, and the time of described ormal weight can arrange appointment by the preferential selection of mobile device or video conference application by user.In some such embodiment, viewing area 1155 is automatically displayed after video conference starts, and disappears according to above mentioned same way.

The 6th stage 1235 was illustrated in the UI1205 after the animation that finishes the 5th transition stage.Specifically, during the 6th stage was illustrated in video conference, the PIP being presented by UI 1205 shows 1280.As mentioned above, PIP shows that 1280 comprise that two videos show: the larger background from local camera shows 1265, and inserts and show 1244 from the less prospect of remote camera.PIP shows that 1280 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Except this synthetic view, the equipment of some embodiment provides other synthetic view.For example, replace the background with larger invitee to show, it can be inviter's video that larger background shows, less prospect inserts to show it can is invitee's video.As further described below, some embodiment allow user to control insertion view and the front view in PIP demonstration, switchably to show local camera and remote camera.In addition, some embodiment allow local video and long-distance video to appear in UI 1205 two viewing area (for example, left and right display window, or upper and lower display window) side by side, or in the viewing area of arranging, two diagonal angles.As further described below, by the preferential selection setting of equipment, or by during video conference, the control that user can select, mode or acquiescence display mode that user can specify PIP to show.

Although Figure 12 represented to present and accept the sequence of operation that video conference is invited according to six different operational phases, but, some embodiment can realize described operation with the less stage.For example, some such embodiment can omit and present phase III 1220 and fourth stage 1225, thereby after user selects " accepting request " option one 250, enter five-stage 1230 from second stage 1215.Realize described operation (with the less stage, present and accept video conference invite) other embodiment can omit first stage 1210 and second stage 1215, thereby in the time that invitee receives the invitation of video conference from inviter, present the phase III 1220 to user.

Figure 13 graphic extension by first and the phase III be combined into a stage, and second and fourth stage be combined into a stage, carry out the example of the operation of graphic extension in Figure 12 with less stage.Especially, Figure 13 is illustrated in five different stages 1390,1392,1394,1230 and 1235, the UI 1205 of long-distance user's equipment 1200.First stage 1390 is similar to the stage 1110, except name field 1295 display name " John Smith ", to point out outside personage's the name of the call other end., between the user of remote-moving apparatus and the user of local device (, the John Smith in this example), set up call.Second stage 1392 and phase III 1394 are similar to first stage 1210 and the second stage 1215 of Figure 12, except second stage 1392 and phase III 1394 also show outside user's (, the Nancy Jones in this example) the preview of remote-moving apparatus.Fourth stage 1230 and five-stage 1235 are identical with the 6th stage 1235 with the five-stage 1230 of Figure 12.

Except during conversing, activate outside video conference instrument by selectable option, some embodiment allow the user of two camera apparatus directly to initiate video conference, and needn't first make a phone call.The such alternative approach of another kind of video conference is initiated in Figure 14 graphic extension.Figure 14 is illustrated in the UI 1405 of seven different phases 1410,1415,1420,1425,1430,1435 and 1440, and described seven different phases show the alternative operation sequence that starts video conference.

In the first stage 1410, user browses the Affiliates List on mobile device, the personage who finds him and wish to carry out with it video conference, and the mode of searching contact person with him in order to converse is similar.In second stage 1415, the personage 1455 (for example, by singly referring to dub 1460 in the name 1455 this people) that user selects him to be ready to carry out with it video conference.This selection triggers information and the various user-selectable options that UI 1405 shows contact person.In this example, the name 1455 of Jason is highlighted, to point out that this is the people that user is intended to carry out with it video conference.Different embodiment can point out this selection by different modes.Although the user of second stage 1415 permission equipment 1400 passes through Affiliates List, select user to be intended to carry out with it the people of video conference, but some embodiment allow user to pass through " Recents (recently) " call history and select this people, and the user that described " Recents " call history is listed equipment 1400 carries out with it the personage's of video conference or call particular number or name recently.

In the phase III 1420, after having selected personage's name 1455, UI 1405 shows selected personage's information 1462, and various optional UI project 1468,1472 and 1470.In this example, one of each optional UI project 1472 (can be realized as optional icon or button) provides video conference instrument.Video conference option one 472 allows user to invite the personage who identifies according to contact person 1466 to add video conference.Different embodiment is (for example,, with different arrangements) demonstration information 1462 and optional UI project 1468,1472 and 1470 differently.

Fourth stage 1425 represents that user selects " video conference " option one 472 (for example, by singly refer to dub).In this example, " video conference " option one 472 is highlighted, to point out that video conference instrument 1472 has been activated.In different embodiment, can differently point out this selection (for example,, by highlighting text or the frame of selected icon).

Five, the 6th and the 7th stage 1430,1435 and 1440 was similar to the 3rd, the 4th and five-stage 1120,1125 and 1130 of graphic extension in Figure 11, can understand for the 5th, the 6th and the 7th stage 1430,1435 and 1440 by reference to the discussion in these stages.Briefly, five-stage 1430 graphic extensions wait for that long-distance user responds the cambic maintenance stage that video conference is invited.The 6th stage 1435 was illustrated in after long-distance user accepts video conference request, the size of (demonstration local user's video) viewing area 1480 reduces gradually, make the UI 1405 can be after viewing area 1480, show the viewing area 1492 comprising from the video of long-distance user's camera.In the 7th stage 1440, during video conference, UI 1405 presents PIP and shows 1447.In certain embodiments, the 6th stage 1435 is similar with the layout of the layout of viewing area 1155 in the 7th stage 1440 and the viewing area of Figure 12 described above 1155.

Figure 10,11,12,13 and 14 represents to set up several modes of video conference.In certain embodiments, during conversing, (for example transmit voice data by a communication port (by the communication network such as circuit exchanging communicating network or packet exchange communication network), speech), and during video conference, transmit voice data by another communication port.Thereby, in such embodiments, before setting up video conference, (for example transmit voice data by a communication port, speech), once and set up video conference, just by different communication port (instead of the communication port using during conversing) transmission audio frequency.

For example, in order to provide voice data (, to switch) from call to the seamless transitions of video conference, some embodiment before setting up video conference, not finished call.For example, some embodiment, in finished call and before starting to transmit audio/video data by peer-to-peer communications session, set up reciprocity video conference and connect (for example,, after completing the message sequence of graphic extension in Figure 10).On the other hand, other embodiment is in finished call with before starting to present the audio/video data of reception, set up reciprocity video conference and connect (for example, after completing the message sequence of graphic extension in Figure 10), and start to transmit audio/video data by this peer-to-peer communications session.

The reciprocity video conference of some embodiment connects the directly intercommunication mutually (instead of by for example central server communication) of mobile device allowing in video conference.Some embodiment of equity video conference allow the mutual shared resource of mobile device in video conference.For example, via the control communication port of video conference, by sending instruction from a mobile device to another mobile device, differently process image (to order described another mobile device, share its image and process resource), than the operation of exposure adjustment operation as will be described in further detail below, Focussing and/or switch camera operation, the operation of described another mobile device of a described mobile device in can Long-distance Control video conference.

2. dynamic bit rate setting

In general, mobile device in video conference is by dissimilar communication network, such as different special and public wireless communication network (for example, cellular network such as GSM, UMTS) communication port, mutually transmit data (for example, Voice & Video image).Below with reference to the example of Figure 91 and this cordless communication network of 92 explanation.

Because the number of the mobile device at special time access communications network constantly changes, therefore to can be used for carrying out the bandwidth of video conference be different at different time to communication network.Even during video conference, available bandwidth also can change.In addition, during video conference, flood communication network with high bit rate, or a large amount of signalings that attempt calculates best video meeting bit rate are worthless.

Due to these reasons, some embodiment adopt the new method of the initial optimal bit rate of designated meeting.In order to identify the initial optimal bit rate of video conference, described method starts video conference with specific bit rate, if these embodiment do not detect the network condition that can worsen Video Frequency Conference Quality, so described method incrementally increases bit rate with specified time interval.

This embodiment of graphic extension example in Figure 15.The processing 1500 of the bit rate of the setting video conference of Figure 15 graphic extension some embodiments of the present invention conceptually.The part arranging as video conference (for example, a part of setting up as the video conference of graphic extension in Figure 10) carry out and process 1500, with according to variety of network conditions, dynamically determine the bit rate that transmits data (for example, Voice & Video image).In certain embodiments, processing 1500 is carried out by the management level 935 of the video conference module 925 illustrating above with reference to Fig. 9.The more detailed form of this video conference module is described below with reference to Figure 16.

As shown in Figure 15, by bit rate being made as to (1505) initial bit rate, start to process 1500.In certain embodiments, initial bit rate is the acquiescence reference speed rate of equipment.But, some embodiment allow user to specify initial bit rate.1505, process 1500 also by by initial bit rate, by one or more communication ports, transmit data (for example, Voice & Video image) to remote equipment, start video conference.

Subsequently, process the group network conditional parameter that the remote equipment of 1500 identifications (1510) from video conference receives.In certain embodiments, local device, by real time transport protocol (RTP) communication session of setting up in the time starting video conference, receives a described group network conditional parameter from remote equipment.For example, some embodiment, by the extension feature of RTP, provide network condition parameter.In addition, by pointing out the existence of extension header in RTP packet header, and defining the extension header about additional information, the RTP extension feature of some embodiment can be used to transmit the information (for example, a described group network conditional parameter) of any type.

In different embodiment, each equipment in video conference transmits the different network condition/congestion parameter of many groups.In the embodiment of explanation, a described group network conditional parameter comprises unidirectional stand-by period and bandwidth estimation bit rate below.In other embodiments, a described group network conditional parameter comprise packet loss data and two-way time (RTT) delayed data.Thereby different embodiment can comprise any number of different network condition parameters at a described group network conditional parameter.

In certain embodiments, the group network conditional parameter receiving from the remote equipment of video conference is with the initial bit rate by setting in operation 1505, during video conference, from local mobile device (, carry out and process 1500 mobile device) data (for example, Voice & Video) that send remote equipment to be basic.For example, in certain embodiments, by utilizing the timestamp of audio packet, calculate audio packet and connect by network, propagate into the remote equipment time used from local mobile device, remote equipment can be determined the unidirectional stand-by period.Specifically, in certain embodiments, each audio packet is stamped timestamp.In the situation that there is no packetization delay, the setting-up time interval that remote equipment should equate with the difference of timestamp, audio reception grouping.But, in the time there is unidirectional latency delays, remote equipment divided into groups every audio reception with the time larger than the difference of timestamp.

In addition, in certain embodiments, remote equipment receives the time of video packets by inspection, receives the time of adjacent video grouping, and the size of described adjacent video grouping, determines bandwidth estimation bit rate., the time difference of two continuous videos groupings of reception is used to the size of second video packets the available bandwidth that estimation network is connected.Some embodiment, by checking multipair continuous videos grouping, determine bandwidth estimation bit rate.Example above utilizes the data (, for determining the voice data of unidirectional stand-by period, with for determining the video data of bandwidth estimation bit rate) of particular type.But, in certain embodiments, also can use the data that connect other type transmitting by the network between local mobile device and remote equipment.

After identification (1510) described group network condition, process 1500 and determine subsequently whether (1515) unidirectional stand-by period has deteriorated into the threshold quantity that exceedes definition.In certain embodiments, described threshold quantity is defined as specific stand-by period amount, if the difference between current unidirectional stand-by period and previous unidirectional stand-by period exceedes specific latency amount, determines so that the unidirectional stand-by period deteriorates into and exceedes threshold quantity.In other embodiments, threshold quantity is defined as the specific change rate of unidirectional stand-by period.For example, thereby the rate of change of (, current unidirectional stand-by period and previous unidirectional stand-by period) exceedes specific rate of change when one group of unidirectional stand-by period, determine that so the unidirectional stand-by period deteriorates into and exceedes threshold quantity.

Deteriorate into while exceeding threshold quantity when definite unidirectional stand-by period, process 1500 and finish.Otherwise, process 1500 and determine whether (1520) current bit rate has reached bandwidth estimation bit rate.In certain embodiments, the amount of bandwidth available (for example, 15kb/s (kbps)) that bandwidth estimation bit rate indication network connects.In the time that processing 1500 determines that current bit rate exceed bandwidth estimation bit rate, process 1500 and finish.In the time that processing 1500 determines that current bit rate do not exceed bandwidth estimation bit rate, process 1500 and enter operation 1525.

1525, process 1500 and determine whether current bit rate reaches the Maximum Bit Rate of definition.In the time that processing 1500 determines that current bit rate exceed the Maximum Bit Rate of definition, process 1500 and finish.Otherwise, process 1500 and enter operation 1530, current bit rate is increased to ormal weight.Different embodiment differently defines the amount that increases bit rate.The example that increases the ormal weight of current bit rate comprises the value of 32kbps, 64kpbs and any multiple other increase bit rate.

Whether subsequently, process (1535) the official hour amount of determining passes by.Official hour amount can be 1 second, 2 seconds, 5 seconds or any other possible time quantum, because different embodiment differently defines described time quantum.The 1500 wait official hour amounts of processing are gone over, so that the bit rate (in operation 1530) that remote equipment can be received newly to increase, the data that transmit from local mobile device (for example, Voice & Video image), and determine network condition parameter according to the bit rate of new increase.If the 1500 definite official hour amounts of processing are not also pass by, process so 1500 and return to operation 1535, until official hour amount in the past.Determine official hour amount past tense when processing 1500, process 1500 and return to operation 1510.Proceed as mentioned above to process 1500 since 1510 operation, finish until process 1500.

In the time processing 1500 end (, after operation 1515,1520 or 1525), complete the setting of the bit rate of video conference, and determined optimal bit rate.Because the available bandwidth of video conference may change during video conference, therefore, some embodiment continue, according to the group network conditional parameter (, unidirectional stand-by period and bandwidth estimation bit rate) receiving from remote equipment, to adjust bit rate.Can, by increasing bit rate, during video conference, adjust bit rate.For example, if process 1500 because the unidirectional stand-by period deteriorates into the threshold quantity that exceedes regulation and finishes, and during video conference, the unidirectional stand-by period improves, and some embodiment increase bit rate so.Similarly, finish because bit rate exceedes bandwidth estimation bit rate if process 1500, and during video conference, bandwidth estimation bit rate increases, some embodiment increase bit rate so.

On the contrary, can, by reducing bit rate, during video conference, adjust bit rate.For example, if during video conference, the unidirectional stand-by period continues to deteriorate into the threshold quantity that exceedes regulation, and some embodiment reduce bit rate so.In addition, if during video conference, bit rate continues to exceed bandwidth estimation bit rate (for example, bandwidth estimation bit rate continues to reduce), and some embodiment reduce bit rate so.

In addition, process 1500 unidirectional stand-by period of use and bandwidth estimation bit rate and determine whether to increase bit rate.But, those of ordinary skill will appreciate that in different embodiment, can determine whether to increase bit rate with any multiple network conditional parameter.For example, determining whether to increase bit rate can be only based on RTT delayed data or packet loss data.

C. video conference architecture

As mentioned above, Figure 16 video conference of two camera movement equipment of some embodiment of graphic extension and software architecture of processing module 1600 conceptually.As shown in the figure, video conference and processing module 1600 comprise client application 1665, video conference module 1602, exchange of media module 1620, buffer 1625, photographic images processing unit (CIPU) driver 1630, encoder-driven device 1635 and decoder driver 1640.In certain embodiments, buffer 1625 is to preserve the image of video so that the frame buffer showing on the display 1645 of two camera movement equipment.

In certain embodiments, client application 1665 is identical with the video conference client 945 of Fig. 9.As mentioned above, client application 1665 can be integrated in Another application, or is embodied as independently application.Client application 1665 can be to utilize the application of the video conference function of video conference module 1602, for example, such as video conference application, IP speech (VOIP) application (, Skype) or instant messaging application.

The client application 1665 of some embodiment sends instruction to video conference module 1602, such as starting the instruction of meeting and closing session, receive instruction from video conference module 1602, the instruction of user from two camera movement equipment is routed to video conference module 1602, and generation is presented on two camera movement equipment and allows the mutual user interface of user and application.

D. video conference manager

As shown in Figure 16, video conference module 1602 comprises video conference manager 1604, image processing manager 1608, networking manager 1614 and buffer 1606,1610,1612,1616 and 1618.In certain embodiments, video conference module 1602 is identical with the video conference module 925 of graphic extension in Fig. 9, thereby carries out above about some or all the identical function described in video conference module 925.

In certain embodiments, video conference manager 1604 is responsible in the time that video conference starts, some or all of other modules of initialization video conference module 1602 (for example, image processing manager 1608 and networking manager 1614), during video conference, control the operation of video conference module 1602, with in the time that video conference finishes, stop the operation of some or all of other modules of video conference module 1602.

The video conference manager 1604 of some embodiment is also processed the image that the one or more equipment from video conference receive, and by the image of one or two camera of two camera movement equipment, to be presented on two camera movement equipment.For example, the video conference manager 1604 of some embodiment fetches from buffer 1618 decoded picture that another equipment from participating in video conference receives, with fetch the image processed by CIPU 1650 (image of, being taken by two camera movement equipment) from buffer 1606.In certain embodiments, before image being presented on two camera movement equipment, video conference manager 1604 is gone back proportional zoom and composograph., in certain embodiments, video conference manager 1604 produces PIP or other synthetic view, to be presented on mobile device.The image that some embodiment proportional zooms are fetched from buffer 1606 and 1618, and the image that other embodiment proportional zoom is fetched from one of buffer 1606 and 1618.

Although Figure 16 is illustrated as video conference manager 1604 part for video conference module 1602, but some embodiment of video conference manager 1604 are realized as the assembly separating with video conference module 1602.Thereby single video conference manager 1604 can be used to management and control several video conference modules 1602.For example, some embodiment by local device, move independently video conference module with Multi-Party Conference in each party mutual, these video conference modules on local device are by a video conference manager administration and control.

Before image is encoded by encoder 1655, the image processing manager 1608 of some embodiment is processed the image by the camera of two camera movement equipment.For example, the image that some embodiment of image processing manager 1608 process CIPU 1650 is carried out the one or more operations in exposure adjustment, Focussing, perspective correction, dynamic range adjustment and image scaling.In certain embodiments, image processing manager 1608 is controlled the frame rate of the coded image of another equipment in video conference that sends to.

Some embodiment of networking manager 1614 manage the one or more connections between two camera movement equipment and another equipment of participation video conference.For example, the networking manager 1614 of some embodiment, in the time that video conference starts, is set up the connection between two camera movement equipment and another equipment of video conference, and in the time that video conference finishes, is interrupted these connections.

During video conference, networking manager 1614 sends the images of being encoded by encoder 1655 to another equipment of video conference, and the image path receiving from another equipment of video conference by decoder 1660 so that decoding.In certain embodiments, networking manager 1614, instead of image processing manager 1608, control sends the frame rate of the image of another equipment of video conference to.For example, some such embodiment of networking manager 1614 should be transmitted to some coded frame in the coded frame of another equipment of video conference, control frame speed by abandoning (that is, not transmitting).

As shown in the figure, the exchange of media module 1620 of some embodiment comprises camera source module 1622, video compressing module 1624 and video decompression module 1626.Exchange of media module 1620 is identical with the exchange of media module 310 shown in Fig. 3, and more details is provided simultaneously.Camera source module 1622 is by CIPU driver 1630, route messages and media content between video conference module 1602 and CIPU1650, video compressing module 1624 is by encoder-driven device 1635, route messages and media content between video conference module 1602 and encoder 1655, and video decompression module 1626 is by decoder driver 1640, route messages and media content between video conference module 1602 and decoder 1660.Some embodiment are a part that is included in TNR module 315 (not shown in Figure 16) in exchange of media module 310 and is embodied as camera source module 1622, and other embodiment is embodied as TNR module 315 part for video compressing module 1624.

In certain embodiments, CIPU driver 1630 and encoder-driven device 1635 are identical with encoder-driven device 320 with the CIPU driver 305 of graphic extension in Fig. 3.The decoder driver 1640 of some embodiment is served as the communication interface between video decompression module 1626 and decoder 1660.In such embodiments, decoder 1660 decoding receive from another equipment of video conference by networking manager 1614 and by the image of video decompression module 1626 routes.After image is decoded, image is sent back to video conference module 1602 by decoder driver 1640 and video compressing module 1626.

Except carrying out during video conference Video processing, the video conference of two camera movement equipment of some embodiment and processing module 1600 also, during video conference, are carried out audio frequency and are processed operation.The software architecture that Figure 17 graphic extension is such.As shown in the figure, video conference and processing module 1600 comprise video conference module 1602 (it comprises video conference manager 1604, image processing manager 1608 and networking manager 1614), exchange of media module 1620 and client application 1665.Other assembly and the module of the video conference shown in Figure 16 and processing module 1600 in Figure 17, are omitted, with simplified illustration.Video conference and processing module 1600 also comprise frame buffer 1705 and 170, audio frequency processing manager 1715 and audio driver 1720.In certain embodiments, audio frequency is processed manager 1715 and is realized as independent software module, and in other embodiments, audio frequency processing manager 1715 is realized as a part for exchange of media module 1620.

Audio frequency is processed manager 1715 and is processed the voice data of being caught by two camera movement equipment, to send another equipment in video conference to.For example, audio frequency is processed manager 1715 and is received the voice data of being caught by microphone 1725 by audio driver 1720, to audio data coding, afterwards the voice data of coding is kept in buffer 1705, to send described another equipment to.Audio frequency is processed manager 1715 and is also processed the voice data of being caught and being received from described another equipment by another equipment in video conference.For example, audio frequency is processed manager 1715 and is fetched voice data from buffer 1710, and to voice data decoding, decoded voice data is exported to loud speaker 1730 by audio driver 1720 subsequently.

In certain embodiments, video conference module 1602 processes together with audio frequency the part that manager 1715 and associated buffer thereof are larger meeting modules.When in the situation that not exchanging video content, carry out in many ways audio conferencing between several equipment time, 1600 of video conference and processing modules utilize networking manager 1614 and audio frequency to process manager 1715, so that easier by the audio exchange of Internet protocol (IP) layer.

With reference now to Figure 18,, the operation of the video conference manager 1604 of some embodiment is described.Figure 18 conceptually graphic extension by the video conference manager of some embodiment, the processing 1800 of carrying out such as the video conference manager 1604 of graphic extension in Figure 16.This can be equal to by the management level 935 of Fig. 9 carries out.In certain embodiments, for example, when the user of two camera movement equipment (accepts, by being presented at the user interface on two camera movement equipment) video conference request, or in the time that the user of another equipment accepts the request of user's transmission of two camera movement equipment, video conference manager 1604 is carried out and is processed 1800.

Start the instruction of video conference by receiving (1805), start to process 1800.In certain embodiments, instruction receives from client application 1665, or user interface on two camera movement equipment receives from user by being presented at, and is transmitted to video conference manager 1604 by client application 1665.For example, in certain embodiments, in the time that the user of two camera movement equipment accepts video conference request, instruction be receive by user interface and forwarded by client application.On the other hand, in the time of request that the user of another equipment accepts to send from local device, some embodiment receive instruction from client application, and there is no user interface interaction (but, can have the user interface interaction of previous transmission initial request).

Subsequently, process the first mutual module of 1800 initialization (1810) and video conference manager 1604.Some embodiment comprise that with the mutual module of video conference manager 1604 CIPU 1650, image processing manager 1608, audio frequency process manager 1715 and networking manager 1614.

In certain embodiments, initialization CIPU 1650 comprises the image of one or two camera of the two camera movement equipment of instruction CIPU 1650 beginning reasons.Some embodiment start to fetch image from buffer 1610 by instruction image processing manager 1608, and the image of fetching of processing and encode, and initialisation image is processed manager 1608.For initialization audio frequency is processed manager 1715, some embodiment instruction audio frequency are processed manager 1715 and are started the voice data that coding microphone 1725 is caught, be kept at the voice data (voice data receiving from another equipment) in buffer 1710 with decoding, to export to loud speaker 1730.The initialization of the networking manager 1614 of some embodiment comprises that 1614 foundation of instruction networking manager are connected with the network of another equipment in video conference.

Processing 1800 determines subsequently whether also surplus (1815) and has initialized any module of wanting., process 1800 and return to operation 1810, with another module of initialization while having the initialized module of wanting when surplus.In the time that the module of all needs has all been initialised, processes 1800 and produce (1820) for the composograph being presented on two camera movement equipment (, local display).These composographs can comprise that those composographs of being shown in the Figure 65 the following describes (, PIP or other compound display), and can comprise from the image of the camera of two camera movement equipment of this locality with from the various combinations of image of camera of another equipment that participate in video conference.

Next, process 1800 and determine whether (1825) have made change to video conference.Some embodiment by user be presented at the mutual of user interface on two camera movement equipment, receive the change to video conference, and other embodiment is by networking manager 1614, receive the change (, Long-distance Control) to video conference from another equipment.In certain embodiments, also can be from client application 1665, or other module in video conference module 1602 receives change that video conference is arranged.Video conference setting also can be because the variation of network condition changes.

In the time changing, process 1800 and determine whether (1830) are the changes to network settings to the change of video conference.In certain embodiments, described change or network settings change or image taking arrange change.In the time being change to network settings to the change of video conference, process amendment (1840) network settings, enter subsequently operation 1845.The network settings change of some embodiment comprises the bit rate changing Image Coding, or transmits the frame rate of image to another equipment.

In the time not being change to network settings to the change of video conference, processing 1800 and determine that described changes are changes that image taking is arranged, thereby enter operation 1835.Process 1800 and carry out subsequently the change that (1835) arrange image taking.In certain embodiments, can comprise and switch camera (the change of image taking setting, any camera video on switch dual camera movement equipment), Focussing, exposure is adjusted, show or do not show the image from one or two camera of two camera movement equipment, and zooming in or out the image being presented on two camera movement equipment, and other arranges change.

In operation 1845, process 1800 and determine whether to finish video conference.Determine while not finishing video conference when processing 1800, process 1800 and return to operation 1820.In the time that processing 1800 determines that video conferences will finish, process 1800 and finish.In the time that processing 1800 receives from client application 1665 instruction that finishes video conference (, receive owing to the user interface by local two camera movement equipment, or the instruction receiving from participating in another equipment of video conference), process 1800 the definite end of some embodiment video conference.

In certain embodiments, in the time that video conference finishes, video conference manager 1604 is carried out and is not shown in the various operations of processing in 1800.Some embodiment instruction CIPU 1650 stop producing image, and instruction networking manager 1614 interrupts being connected with the network of another equipment in video conference, and instruction image processing manager 1608 stops processing and coded image.

E. time reducing noise

Some embodiment comprise processing video image, to reduce the special time reducing noise module of the noise in video.The time reducing noise module of some embodiment compares the successive image in video sequence, to identify and to eliminate harmful noise from video.

Figure 19 is the software architecture of this time reducing noise (TNR) module 190 of some embodiment of graphic extension conceptually.Some embodiment are embodied as TNR module 1900 part (for example, being embodied as a part for exchange of media module as shown in Figure 3) for application, and other embodiment is embodied as TNR module 1900 independent utility being used by other application.Other embodiment is in addition embodied as TNR module 1900 part for the operating system of moving on two camera movement equipment.In certain embodiments, TNR module 1900 is realized by one group of API, and described one group of API provides some or all functions of TNR module 1900 to other application.

As shown in Figure 19, TNR module 1900 comprises TNR manager 1905, difference block 1910, the average module 1915 of pixel and motion history module 1920.Although Figure 19 is expressed as independently module three modules 1910,1915 and 1920, but some embodiment realize the function of these modules that the following describes in individual module.The TNR module 1900 of some embodiment receives input picture, benchmark image and motion history as input.In certain embodiments, input picture is the image of processing at present, and benchmark image is the formerly image that in video sequence, input picture compares with it.TNR module 1900 is exported output image (a kind of input picture of reducing noise of form) and output movement history.

TNR manager 1905 directs data of some embodiment flowing in TNR module 1900.As shown in the figure, TNR manager 1905 receives input picture, benchmark image and motion history.TNR manager 1905 is also exported output image and output movement history.TNR manager 1905 sends to difference block 1910 input picture and benchmark image, receives difference image from difference block 1910.

In certain embodiments, difference block 1910 is processed the data that receive from TNR manager 1905, and data after treatment are sent to TNR manager 1905.As shown in the figure, difference block 1910 receives input picture and benchmark image from TNR manager 1905.The difference block 1910 of some embodiment, by deduct the pixel value of an image from the pixel value of another image, produces difference image.Difference image is sent to TNR manager 1905.Difference between two images of difference image instruction of some embodiment, to identify compared with previous image the part having changed of input picture, and the part remaining unchanged of input picture.

TNR manager 1905 also sends to the average module 1915 of pixel input picture and benchmark image.As shown in the figure, some embodiment also send to the average module 1915 of pixel motion history.But, other embodiment can only send input picture and benchmark image, and does not send motion history.In any one embodiment, TNR manager 1905 is the image from average module 1915 reception & disposals of pixel all.

The average module 1915 of pixel of some embodiment utilizes motion history to determine whether, about the ad-hoc location in image, to obtain the mean value of pixel from input picture and benchmark image.In certain embodiments, motion history comprises the probable value of the each pixel in input picture.Specific probable value represents with respect to the respective pixel in benchmark image, the change probability of (, dynamic pixel) of the respective pixel in input picture.For example, if the probable value of the specific pixel in input picture is 20, indicate so with respect to the respective pixel in benchmark image, the probability that the specific pixel in input picture has changed is 20%.As another example, if the probable value of the specific pixel in input picture is 0, to indicate so with respect to the respective pixel in benchmark image, the specific pixel in input picture does not change (, static pixels).

Different embodiment differently preserves the probable value of input picture.Some embodiment can be kept at the probable value of each pixel of input picture in a data array.Other embodiment can be kept at (for example,, in a group pattern) in the matrix that dimension is identical with the resolution of the image of video probable value.For example, if the resolution of the image of video is 320 × 240, matrix is also 320 × 240 so.

When except input picture and benchmark image, when the average module 1915 of pixel also receives motion history from TNR manager 1905, the average module 1915 of pixel reads the probable value of the each pixel in input picture.For example, if the probable value of the specific pixel in input picture lower than the threshold value specifying (, 5%, 20%), average module 1915 bases of pixel are in this specific pixel so, probably can there is not motion, thereby the difference between the image of this pixel may be owing to the supposition of noise, ask the mean value of the respective pixel value in this particular pixel values and benchmark image.

If the probable value of the specific pixel in input picture is not less than the threshold value of regulation, the average module 1915 of pixel is not revised the specific pixel (, keeping the same with in input picture at the pixel value of this pixel) of input picture so.This is because motion is probably positioned at this specific pixel, thereby the difference between image is not probably the result of noise.In certain embodiments, in the time that motion history is not sent to the average module 1915 of pixel, the average module 1915 of pixel is asked the mean value of the respective pixel in each pixel and the benchmark image in input picture.The image after treatment of being exported and be sent to TNR manager 1905 by the average module 1915 of pixel comprises not by the input image pixels value of average any pixel, and by the average pixel value of the average any pixel of the average module 1915 of pixel.

In certain embodiments, motion history module 1920 is processed the data that receive from TNR manager 1905, and result data loopback to TNR manager 1905.The motion history module 1920 of some embodiment receives input picture and motion history from TNR manager 1905.Some embodiment are this data input Bayes estimator, to produce the new motion history (, one group of probable value) that can use in the pixel about next input picture is average.Other embodiment produces new motion history with other estimator.

With reference now to Figure 20,, the operation of TNR module 1900 is described.Figure 20 is the processing 2000 of the time noise of the image of the reduction video of some embodiment of graphic extension conceptually.Receive (2005) input picture, benchmark image and motion history from TNR manager 1905 and start to process 2000.Input picture is the image of processing for noise reduction at present.In certain embodiments, benchmark image is the previous image the image sequence of video receiving from CIPU.But, in other embodiments, benchmark image is the output image (, the output of TNR module 1900) being produced by the processing of previous input picture.Motion history is the output movement history being produced by the processing of previous input picture.

In the time that input picture is first image of video, the TNR module 1900 of some embodiment is not processed first image (, not to first image applications TNR).In other words, TNR manager 1905 receives first image, and just exports first image.In other embodiments, in the time that input picture is first image of video, first image is used as input picture and benchmark image, this image of TNR module 1900 processing as described below.In addition, in the time that input picture is the first image of video, motion history be empty (that is, and null value, Quan Weiling, etc.), TNR manager 1905 is just exported empty motion history, as output movement history.

TNR manager 1905 determines whether (2010) input picture is static subsequently.Determine in order to make this, some embodiment send to difference block 1910 input picture and benchmark image, receive difference image from difference block 1910.Difference between two images lower than regulation threshold value (for example, 5% difference, 10% difference etc.) time, some embodiment are categorized as still image input picture.

In the time that input picture is still image, TNR manager 1905 sends to the average module 1915 of pixel input picture and benchmark image, to calculate the mean value of the pixel of (2015) input picture and the pixel of benchmark image, to reduce any noise from still image.Processing subsequent enters the operation 2040 the following describes.

In the time that input picture is not still image, TNR manager sends to the average module 1915 of pixel input picture, benchmark image and motion history to process.The average module 1915 of pixel is selected a pixel in (2020) input picture.By utilizing motion history, whether the probability of motion that the average module 1915 of pixel is determined (2025) pixel is lower than specific threshold, as mentioned above.

If the probability of selected pixel is lower than specific threshold, the average module 1915 of pixel is calculated the mean value of the respective pixel in described pixel and the benchmark image of (2030) input picture so.Otherwise this pixel is not averaging, and output image is by identical with the input picture in described specific pixel.The average module 1915 of pixel determines whether (2035) remain and have any unselected pixel in input picture subsequently.If there is also not processed any pixel, process so and return to operation 2020 to select next pixel.The average module 1915 executable operations 2020-2030 of pixel, until assessed all pixels.

Processing subsequent is upgraded (2040) motion history.As shown in Figure 19 and as mentioned above, motion history module 1920 is upgraded motion history according to input picture.New motion history together with the processing from the average module of pixel after image exported by TNR manager.

F. image processing manager & encoder

Except the time reducing noise and image processing operations carried out by CIPU and/or CIPU driver, some embodiment carry out various image processing operations at the image processing layer 930 of video conference module 925.These image processing operations can comprise the adjustment of exposure adjustment, Focussing, perspective correction, dynamic range and image scaling etc.

Figure 21 conceptually graphic extension carries out the processing 2100 of such image processing operations.In certain embodiments, some or all operations of processing 2100 are carried out by the image processing manager 1608 of Figure 16 and the combination of encoder-driven device 1635.In some such embodiment, image processing manager 1608 is carried out the processing (for example, convergent-divergent, dynamic range adjustment, perspective correction etc.) based on pixel.Some embodiment are during video conference, to the image that is transmitted to another equipment that participates in video conference is carried out and processed 2100.

With reference now to Figure 16,, illustrate and process 2100.By fetch (2105) image from buffer 1606, start this processing.In certain embodiments, the image of fetching is the image (a, image in image sequence) of video.This video can be by the camera of carrying out the equipment of processing 2100 thereon.

Subsequently, process 2100 pairs of images of fetching and carry out (2110) exposure adjustment.Some embodiment are by being presented at the user interface on two camera movement equipment, and adjustment exposes.Operation is adjusted in the example exposure of this embodiment of Figure 22 graphic extension.

The three phases 2210,2215 and 2220 of the UI 2205 of Figure 22 reference device 2200, illustrates that exposure adjustment operates.First stage 2210 graphic extension comprises the UI 2205 of viewing area 2225 and viewing area 1155.As shown in the figure, viewing area 2225 shows all images 2230 of black dull man of the sun and face and health.Black dull face and body shows that this man does not obtain appropriate exposure.Image 2230 can be the video image by the camera of equipment 2200.As shown in the figure, viewing area 1155 comprises the optional UI project 2250 for finishing video conference.In certain embodiments, the layout of the layout of viewing area 1155 and the viewing area of Figure 12 described above 1155 is identical.

The a certain region of the user by selecting viewing area 2225 of second stage 2215 graphic extension equipment 2200, initiates exposure and adjusts operation.In this example, the realization being Anywhere placed in viewing area 2225 by finger 2235 is selected.In certain embodiments, the menu that user adjusts from possible image setting, select exposure adjustment.

Phase III 2220 be presented at exposure adjust operation after, described man's image 2240.As shown in the figure, image 2240 is similar to image 2230, but man in image 2240 is by appropriate exposure.In certain embodiments, the image of appropriate exposure is the image of taking after the image of incorrect exposure.The exposure of operation adjustment by the successive image of the camera of equipment 2200 adjusted in the exposure of initiating in second stage 2215.

Return to Figure 21, process 2100 and subsequently image is carried out to (2115) Focussing.Some embodiment, by being presented at the user interface on two camera movement equipment, carry out Focussing.Figure 23 is the example of this Focussing operation of graphic extension conceptually.

Three different phases 2310,2315 and 2320 of the UI 2305 of Figure 23 reference device 2300, the operation of graphic extension Focussing.First stage 2310 graphic extension comprises the UI 2305 of viewing area 2325 and viewing area 1155.Viewing area 2325 presents the blurred picture 2330 by the man of the camera of equipment 2300.This fuzzy this man's of explanation image 2330 misalignment focuses.,, in the time of this man's of camera image 2330, the lens of camera are not focused in this man.In addition, image 2330 can be the video image by the camera of equipment 2300.As shown in the figure, viewing area 1155 comprises the optional UI project 2350 for finishing video conference.In certain embodiments, the layout of the viewing area 1155 of the layout of viewing area 1155 and the viewing area of Figure 12 described above 1155 is identical.

A region of the user by selecting viewing area 2325 of second stage 2315 graphic extension equipment 2300, initiates Focussing operation.In this example, the realization being Anywhere placed in viewing area 2225 by finger 2335 is selected.In certain embodiments, user selects Focussing from the menu of possible image setting adjustment.

After phase III 2320 has been presented at Focussing operation, described man's image 2340.As shown in the figure, image 2340 is identical with image 2330, but man in image 2340 seems more clear.The lens of this explanation camera are gathered in this man rightly.In certain embodiments, the appropriate image focusing on is the image of taking after the image of incorrect focal length.The Focussing operation adjustment of initiating in second stage 2315 is by the focal length of the successive image of the camera of equipment 2300.

Return to Figure 21, process 2100 pairs of images and carry out (2120) image scaling.Some embodiment carry out image scaling to image, to reduce for the bit number to Image Coding (, reducing bit rate).In certain embodiments, process and 2100 carry out image scaling, as below with reference to as described in figure 26.

Process 2100 and subsequently image is carried out to (2125) perspective correction.In certain embodiments, process 2100 and carry out perspective correction, as follows described in Figure 24.Such perspective correction relates to the data of utilization by the orientation of the two camera movement equipment of identification and the one or more accelerometers that move and/or gyro sensor acquisition.Described data are used to revise image subsequently, to proofread and correct incorrect perspective.

After image is carried out to perspective correction, process the dynamic range of 2100 adjustment (2130) images.In certain embodiments, the dynamic range of image is the scope that each pixel in image can have probable value.For example, the image with the dynamic range of 0-255 can be adjusted to the scope of 0-128, or any other number range.The dynamic range of adjusting image can reduce the quantity of the bit for to Image Coding (that is, reducing bit rate), thereby makes image smoothing.

The dynamic range of adjusting image also can be used for various other objects.An object is to reduce picture noise (for example, image is taken by noisy camera sensor).In order to reduce noise, can adjust the dynamic range of image, black level is redefined, to comprise somber (, the black of pulverizing (crush black)).In this way, the noise of image is reduced.Another object of dynamic range adjustment is to adjust one or more colors or color gamut, to strengthen image.For example, some embodiment can suppose that the image of positive camera is the image of face.Therefore, can adjust the dynamic range of this image, to strengthen redness and pink, make people's cheek seem ruddy/more ruddy.Dynamic range adjustment operation also can be used for other object.

Finally, process 2100 and determine that (2135) are for the one or more rate controller parameters to Image Coding.In certain embodiments, such rate controller parameter can comprise quantization parameter and frame type (for example, predictive frame, bidirectional frame, intracoded frame).This processing subsequent finishes.

Although each operation of processing 2100 is illustrated according to specific order and carries out, but, those of ordinary skill will appreciate that the many operations (exposure adjustment, Focussing, perspective correction etc.) in these operations can carry out according to random order, and irrelevant each other., the processing of some embodiment can be carried out Focussing before exposure is adjusted, or can similarly revise the processing of graphic extension in Figure 21.

1. perspective correction

As mentioned above, some embodiment, before showing or transmitting image, carry out perspective correction to image.In some cases, on two camera movement equipment one or multi-section camera can complete face to its subject, and in uncorrected image, object is known from experience the distortion that seems.Can process image with perspective correction, make image verily to reflect subject in the image appearance In the view of people.

Figure 24 summarizes the image processing manager of ground graphic extension by some embodiment, such as the perspective correction that the image processing manager of graphic extension in Figure 16 is carried out processes 2400.The processing 2400 of some embodiment is carried out by the image processing layer 930 shown in Fig. 9 (it can comprise image processing manager 1608).Some embodiment carry out and process 2400 in the operation 2125 of processing 2100, so that before showing or transmitting the video image of taking recently, and the perspective of proofreading and correct described image.

Process 2400 from autoacceleration flowmeter sensor receives (2405) data, in certain embodiments, described accelerometer sensor is a part for two camera movement equipment.The accelerometer sensor of some embodiment is measured along the rate of change (, the acceleration of equipment) of the speed of the equipment of one or more axles.This processing also receives (2410) data from gyro sensor, and in certain embodiments, described gyro sensor can be also a part for two camera movement equipment.The gyroscope of some embodiment and accelerometer sensor can be by either alone or in combination for identifying the orientation of two camera movement equipment.

Subsequently, process 2400 according to the data that obtain from accelerometer and gyro sensor, determine the perspective correction amount that (2415) will carry out.Conventionally,, in the time that described orientation is farther from axle, in order to produce best image, need more perspective correction.Some embodiment, according to the orientation of equipment, calculate the distortion parameter (warp parameter) that represents perspective correction amount.

After determining the perspective correction amount that will carry out, process 2400 and receive (2420) image by the camera of two camera movement equipment.Can carry out described processing to the each image in the video sequence by this camera.Some embodiment can be to carrying out independent calculating from each image of two cameras on two camera movement equipment respectively.

Described processing subsequent, according to definite perspective correction amount, is revised (2425) image.Except other expression of distortion parameter or perspective correction amount, some embodiment also use baseline image or out of Memory (for example, the point that should proofread and correct about it of user's input).After amendment image, process 2400 and finish.

Figure 25 conceptually example image of some embodiment of graphic extension processes operation.Figure 25 graphic extension is by the first image processing operations 2505 that does not use the first image processing module 2520 of perspective correction to carry out, and the second image processing operations 2550 of being carried out by the second image processing module 2565 of use perspective correction.

As shown in the figure, the first image 2510 of the block 2515 by the angled aerial prospective generation of watching block is downwards carried out to the first image processing operations 2505.Compared with the bottom of block 2515, the top of block from described perspective more close to.Thereby block 2515 seems towards the camera tilt of taking the first image 2510.Figure 25 has also represented the first image 2525 after treatment after being processed by the first image processing module 2520.As shown in the figure, there is identical reprocessing in the block 2515 in the first image 2525 after treatment, because the first image processing module 2520 does not carry out any perspective correction.

The second image 2555 to block 2560 carries out the second image processing operations 2550.Block 2515 in block 2560 and the first image 2510 is identical.Figure 25 is also illustrated in by the perspective correction device 2570 of the second image processing module 2565 and carries out the second image 2575 after treatment after the processing of the second image 2555.Perspective correction device 2570 can use processes 2400, to proofread and correct the perspective of the second image 2555.According to the data that become downward angle to tilt from accelerometer with the gyrostatic camera of pointing out to take the second image 2555 (may also according to other data), perspective correction device 2575 can be proofreaied and correct the second image, make in the second image 2575 after treatment, it seems that block seem to watch as the crow flies.

2. convergent-divergent and bit stream processing

Image scaling and bit stream processing belong to the row of the function of carrying out with reference to the image processing layer 930 by some embodiment of Figure 21 explanation in the above.Image scaling (carrying out in operation 2130) relates to ratio in certain embodiments and amplifies or ratio downscaled images (, amendment is used for the number of the pixel that represents this image).In certain embodiments, bit stream processing relates in the big or small data insertion bit stream of image after instruction convergent-divergent.In certain embodiments, this convergent-divergent and bit stream are processed and for example, are carried out by encoder-driven device (, driver 1635).

Figure 26 is the software architecture of this encoder-driven device 2600 of some embodiment of graphic extension conceptually, and shows example convergent-divergent and the bit stream processing operation example image 2605 carried out by encoder-driven device 2600.In certain embodiments, image 2605 be by the camera of two camera movement equipment, for sending the image of video of another equipment of video conference to.Referring to Figure 16, in certain embodiments, video image propagates into buffer 1606 by CIPU driver 1630 and camera source module 1622 from CIPU 1650, and image processing manager 1608 is fetched described video image from buffer 1606.In image processing manager 1608, for example, through image processing (, Focussing, exposure adjustment, perspective correction), afterwards, image is sent to encoder-driven device 1635 by buffer 1610 and video compressing module 1624.

As shown in the figure, encoder-driven device 2600 comprises processing layer 2610 and rate controller 2645.In Figure 30 of explanation, illustrate below the example of the rate controller of some embodiment.Processing layer 2610 comprises image sealer 2615 and bit stream manager 2625.In certain embodiments, these modules are carried out various operations to image before image is encoded and afterwards.Although in this example, image sealer is expressed as a part for the processing layer 2610 of encoder-driven device 2600, but some embodiment are embodied as image sealer a part (, carrying out image scaling before image and size data are sent to encoder-driven device) for image processing manager 1608 instead of encoder-driven device 2600.

As shown in the figure, before image through-rate controller 2645 is sent to encoder 2650, image sealer 2615 zoomed images.Image 2605 is sent out by scaler 2615 and is dwindled into image 2630 by ratio.Except ratio downscaled images, some embodiment can also ratio enlarged image.

As shown in Figure 26, the image that some embodiment ratios are dwindled input (for example, image 2605), the image subsequently ratio being dwindled (for example, 2630) (for example overlap spatial redundancy image, image 2635) upper, the size (just as usually saying) identical with input picture (, the line number of the pixel of image 2605 is identical with line number and the columns of the pixel of spatial redundancy image 2635 with columns) of described spatial redundancy image.The upper left corner that the image 2630 that some embodiment dwindle ratio overlaps spatial redundancy image (as shown in the figure, to produce composograph 2640), and the doubling of the image that other embodiment dwindles ratio to the different piece of spatial redundancy image (for example, central authorities, the upper left corner, directly over, under, lower right etc.).

In certain embodiments, spatial redundancy image is to be all a kind of color (for example, black, blueness, redness, white etc.) substantially, or has the image of repeat patterns (for example, grid, striped etc.).For example, the spatial redundancy image 2635 shown in Figure 26 has the cruciform pattern of repetition.Due to repeated cause, the spatial redundancy part of composograph 2640 can be easily compressed into a small amount of data by encoder.In addition, if a series of image is all dwindled by ratio, and each image in sequence, the spatial redundancy image of use is all identical, and time compression can be used to further be reduced to the weight that presents the needed data of coded image so.

Some embodiment of image sealer 2615 also produce the size data 2620 of the size (for example, the size of the image 2630 of proportional zoom) of the image after instruction convergent-divergent, and the size data 2620 producing is sent to bit stream manager 2625.The size data 2620 of some embodiment is according to the line number of pixel of the image 2630 after convergent-divergent and the columns of pixel (, height and width), the size of the image 2630 after instruction convergent-divergent.In certain embodiments, size data 2620 is also indicated image 2630 after the convergent-divergent position in composograph 2640.

After image is scaled, composograph 2640 through-rate controllers 2645 are sent to encoder 2650.In certain embodiments, following further in detail described in, rate controller 2645 is controlled the bit rate (, size of data) of the image of being exported by encoder 2650.The encoder 2650 of some embodiment compresses and coded image.Encoder 2650 can use H.264 coding or another kind of coding method.

The bit stream manager 2625 of some embodiment receives the bit stream of one or more coded images from encoder 2650, and size data is inserted in bit stream.For example, in certain embodiments, bit stream manager 2625 receives size data 2620 from image sealer 2615, and size data 2620 is inserted the bit stream 2655 of the coding composograph 2640 receiving from encoder 2650.In this case, the output of bit stream manager 2625 is the amended bit streams 2660 that comprise size data 2620.Different embodiment inserts the diverse location in bit stream 2655 size data 2620.For example, bit stream 2660 shows that size data 2620 is inserted in the beginning of bit stream 2660.But, other embodiment size data 2620 be inserted in bit stream 2655 ending, be inserted in the centre of bit stream 2655 or be inserted in any other position in bit stream 2655.

In certain embodiments, bit stream 2655 is the bit streams that comprise a series of one or more coded images of composograph 2640.In some such embodiment, the image in this series is all scaled to identical size, and size data 2620 is indicated the size of the image after these convergent-divergents.Be transmitted to after the equipment of the video conference other end at image, receiving equipment can extract size information from bit stream, and utilize described size information correctly decode receive image.

Graphic extension is by the encoder-driven device of two camera movement equipment conceptually for Figure 27, and the image scaling of carrying out such as driver 2600 processes 2700.Processing 2700 for example, is started by the image (, image 2605) of the camera of two camera movement equipment from receiving (2705).When two camera apparatus are during with two camera images, some embodiment carry out and process 2700 the image from two cameras.

Subsequently, process the image that 2700 convergent-divergents (2710) receive.As mentioned above, differently zoomed image 2605 of different embodiment.For example, the image 2605 in Figure 26 is dwindled by ratio and is superimposed on spatial redundancy image 2635, thereby produces composograph 2640.

Process 2700 and subsequently the image after convergent-divergent (for example, comprising the composograph 2640 of the image 2630 after convergent-divergent) is sent to (2715) encoder 2650, so that coding.Some embodiment of processing 2700 send to encoder 2650 the image 2630 after convergent-divergent (being included in composograph 2640) by the rate controller of the bit rate of definite encoder encodes image.Encoder 2650 compressions of some embodiment and coded image (for example, utilizing discrete cosine transform, quantification, entropy coding etc.), and returning to encoder-driven device 2600 with the bit stream of coded image.

Subsequently, process 2700 the big or small data (for example, size data 2620) of the image after instruction convergent-divergent are sent to (2720) bit stream manager.As shown in Figure 26, in certain embodiments, (, a module in encoder-driven device 2600 sends to another module in encoder-driven device 2600 size data) carried out in this operation in encoder-driven device 2600.

After image after convergent-divergent is encoded by encoder 2650, process 2700 and receive (2725) bit stream from encoder.As shown in the figure, some embodiment receive described bit stream at the bit stream manager that has received in addition size data.The bit stream receiving comprises the composograph of coding, also can comprise the one or more other image in video sequence.

Process 2700 and subsequently the big or small data (for example, size data 2620) of the image after instruction convergent-divergent are inserted in bit stream, finish subsequently.As shown in Figure 26, in certain embodiments, this operation is also carried out by bit stream manager.As mentioned above, different embodiment inserts size data the different piece of bit stream.In the example of graphic extension, size data 2620 is inserted in the beginning of bit stream 2655, as shown in the bit stream 2660 being obtained.Now, this bit stream can be transmitted to and participate in another equipment of video conference, and at described another equipment, this bit stream can be decoded and be watched.

In certain embodiments, decoder driver (for example, driver 1640) is carried out the reverse functions of encoder-driven device.That is, the bitstream extraction size data of decoder driver from receiving, sends decoder to bit stream, and utilizes the image of size data convergent-divergent decoding.Figure 28 is the software architecture of this decoder driver 2800 of some embodiment of graphic extension conceptually, and represents the processing of example bit stream and the zoom operations of the bit stream 2825 of example being carried out by decoder driver 2800.

In certain embodiments, bit stream 2825 is by the bit stream of the coded image of the video of the camera of an equipment in video conference (for example to comprise, from the bit stream of the encoder-driven device such as driver 2600), and be transmitted to the equipment that decoder driver 2800 runs on.Referring to Figure 16, in certain embodiments, this bit stream is received by networking manager 1614, and is sent to buffer 1616, and described bit stream is fetched by video decompression module 1626 from buffer 1616, and is sent to decoder driver 1640.

As shown in the figure, decoder driver 2800 comprises processing layer 2805.Processing layer 2805 comprises image sealer 2810 and bit stream manager 2820.In certain embodiments, these modules 2810 and 2820 before image is decoded and afterwards, are carried out various operations to the image receiving.Although in this example, image sealer 2810 is expressed as a part for the processing layer 2805 of decoder driver 2800, but, some embodiment are embodied as image processing manager 1608 image sealer, instead of a part for decoder driver (, after decoder driver 2800 sends image, carry out image scaling).

As shown in the figure, (the bit stream manager 2820 of some embodiment receives one or more coded images, image in video sequence) bit stream, and extract size data from this bit stream, send to decoder 2835 to decode bit stream afterwards.For example, as shown in Figure 28, bit stream manager 2820 receives the bit stream 2825 of a coded image, extracts size data 2815 from bit stream 2825, and sends to decoder 2835 to decode the bit stream 2830 (there is no size data 2815) obtaining.As shown in the figure, in certain embodiments, bit stream manager 2820 sends to image sealer 2810 the size data 2815 extracting.

The size data 2815 of some embodiment and to insert size data 2620 in bit stream by encoder-driven device 2600 identical.As above described in the explanation of Figure 26, the size data 2815 of some embodiment, according to the columns of the line number of the pixel of subimage 2845 and pixel, is indicated the size of subimage 2845.Size data 2815 also can be indicated the position of subimage 2845 in larger spatial redundancy image 2840.In this example, bit stream 2825 shows the size data 2815 of the beginning that is inserted in bit stream 2825.But, as mentioned above, different embodiment inserts size data 2815 diverse location of bit stream 2825.

The image sealer 2810 of some embodiment is utilized the size data receiving from bit stream manager 2820, extracts subimage from image.For example, Figure 28 graphic extension image sealer 2810 receives from decoder 2835 image 2840 that comprises subimage 2845.As shown in the figure, the image sealer 2810 of some embodiment is extracted subimage 2845 from image 2840.The image extracting can be displayed on two camera movement equipment subsequently.

Figure 29 conceptually some embodiment of graphic extension by the decoder driver of equipment that participates in video conference, the image extraction process 2900 of carrying out such as driver 2800.For example, by receiving the bit stream (, bit stream 2825) of (2905) coded image, start this processing.Described bit stream can send oneself and carry out another equipment of video conference with the equipment that decoder driver runs on, or can be stored in the memory of described equipment.In the time that described equipment receives image from multiple sources, some embodiment process 2900 to the image from each source.

Subsequently, process 2900 from bitstream extraction (2910) size data.As mentioned above, can find described size data by the diverse location in bit stream.Some embodiment know and find wherein described size data, and other embodiment finds specific mark, and described mark points out that bit stream that described size data is arranged in reception where.In certain embodiments, size data is pointed out size (for example, the pixel count in the pixel count in every a line and each row) and the position of subimage in coded image of subimage.

Process 2900 and subsequently the size data extracting is sent to (2915) image sealer.As shown in Figure 28, in certain embodiments, (, a module in decoder driver sends to another module in decoder driver size data) carried out in this operation in decoder driver.

Process 2900 and also bit stream is sent to (2920) decoder, so that decoding.In certain embodiments, decoder decompresses and decoding bit stream (for example, utilizing inverse discrete cosine transform, inverse quantization etc.), and the image of reconstruct is returned to decoder driver.

After bit stream is decoded by decoder, process 2900 and receive (2925) decoded picture from decoder.As shown in the figure, some embodiment are also receiving the image sealer receipt decoding image of size data from bit stream manager.The size data that processing subsequent utilization receives, extracts (2930) subimage from decoded picture.As shown in the figure, as shown in size data 2815, extract subimage 2845 from the upper left corner of decoded picture 2840.Now, the subimage of extraction for example can be displayed on, on display device (, the screen of two camera movement equipment).

3. rate controller

In certain embodiments, two of equipment cameras have different many groups characteristics.For example, in certain embodiments, front camera is the low resolution camera of optimizing for the shooting of motion video image, and back side camera is the high-resolution camera of optimizing for the shooting of rest image.Due to a variety of causes such as cost, function and/or the geometry of equipment, other embodiment can use the various combination of the camera of different qualities.

The camera with different qualities can be introduced different pseudomorphisms.For example, compared with low resolution camera, high-resolution camera can show more noise.Compared with image by low resolution camera, can show higher levels of space or time complexity by the image of high-resolution camera.In addition, the different cameras that have different optical character can be introduced different γ values in the image of taking.Also can introduce different pseudomorphisms by different cameras for the different sensitization mechanism of photographic images.

Some in the distinctive pseudomorphism of these cameras are hidden the pseudomorphism being produced by other source.For example, in the image of being taken by the high-resolution camera with high-level noise, become not too visible as the pseudomorphism of the accessory substance of Video coding processing.In the time that coding noise (such as quantizing distortion) is hidden in after the distinctive pseudomorphism of camera, Video coding processing can obtain less bit rate with larger quantization step.On the other hand, in the time that camera is introduced less pseudomorphism (such as with regard to low resolution camera), Video coding processing can utilize more tiny quantization step, to avoid the unacceptable vision distortion level causing by quantizing.Thereby compared with ignoring the Video coding processing of these cameras distinctive characteristics, being optimized to the Video coding processing utilizing or compensate the distinctive characteristic of these cameras, can to obtain better rate-distortion compromise.

Compromise for distinctive these cameras characteristic being used for realizing rate-distortion, some embodiment realize two kinds of Video coding processing that the each camera with respect to two cameras is optimized respectively.Figure 30 graphic extension has the example about the system of two Video coding processing of two cameras 3060 and 3070.As shown in Figure 30, system 3000 comprises encoder-driven device 3010, rate controller 3020 and 3040 and video encoder 3030.Encoder 3030 is becoming bit stream 3080 and 3090 from video camera 3060 with 3070 encoding video pictures of taking.

In certain embodiments, video encoder driver 3010 is software modules of moving on one or more processing units.It provides other assembly of video encoder 303 and system, such as the interface between video camera, image processing module, network management module and storage buffer.Encoder-driven device 3010 is controlled the video image taken flowing from camera and image processing module to video encoder 3030, and it also provides the pipeline of coded bit stream 3080 and 3090 to storage buffer and network management module.

As shown in Figure 30, encoder-driven device 3010 comprises two different instances 3020 and 3040 of rate controller.These Multi-instances can be the rate controllers different for two of two different cameras, or about two different cameras, the rate controller configuring according to two kinds of different modes.Specifically, in certain embodiments, these two rate controllers 3020 and 3040 represent two independent rate controllers.On the other hand, in other embodiments, these two rate controllers 3020 and 3040 are the different configurations of two kinds of single rate controller.

Figure 30 also shows the encoder-driven device 3010 that comprises status buffer 3015, and status buffer 3015 is kept at the encoding state information of the speed control operation using during video conference.Specifically, in certain embodiments, these two different rate controllers, or during two kinds of differences of identical rate controller are configured in video conference, share the identical encoding state information being kept in status buffer 3015.Unified rate controller operation in the capture video meeting frequently of this shared permission double vision of state information.This sharing also allows in the meeting of single video capture video, best video coding (, permission is used the encoding state information being kept by the speed control operation of the coding about video that last camera is taken the photograph about the speed control operation of the coding of video that current camera is taken the photograph) during switching camera operation.Figure 30 is expressed as status buffer 3015 part for encoder-driven device 3010, but other embodiment can realize status buffer 3015 outside encoder-driven device 3010.

In status buffer 3015, different embodiment preserves the dissimilar data (for example, dissimilar coding parameter) that represent encoding state information.The current goal bit rate that an example of this encoding state information is video conference.A kind of mode of identification target bit rate has been described above in III.B joint.Other example of this encoding state information comprises bit rate and other encoding state information of the frame of buffer fullness, largest buffer degree of filling, one or more nearest codings.

Rate controller can use target bit rate (or being kept at another encoding state parameter in status buffer) to calculate the one or more parameters that use subsequently in its speed control operation.For example, as further described below, the rate controller of some embodiment carrys out the quantization parameter QP of computing macro block or frame with current target bit rate.For example, some embodiment utilize current target bit rate to calculate quantification and adjust parameter, and these embodiment obtain the quantization parameter QP of macro block and/or frame from quantizing to adjust parameter.Therefore, during camera handover operation in video conference, between (two rate controllers, or two kinds of a rate controller different configurations) two kinds of speed control operations, sharing target bit rate allows the speed control operation of the Video coding to current camera to benefit from the encoding state data of the formerly speed control operation of the Video coding to last camera.

Figure 30 graphic extension comprises the encoder-driven device 3010 of two different rate controller examples 3020 and 3040.But, in other embodiments, these rate controller examples 3020 and 3040 are built in video encoder 3030.Video encoder 3030 is becoming digital bit stream 3080 and 3090 by camera 3060 with 3070 encoding video pictures of taking.In certain embodiments, video encoder produces the bit stream that meets conventional video encoding standard (for example, H.264MPEG-4).In some such embodiment, video encoder is carried out the encoding operation that comprises estimation, discrete cosine transform (" DCT "), quantification and entropy coding.Video encoder also carries out the decode operation with the functions reversed of encoding operation.

In certain embodiments, encoder 3030 comprises the quantizer module 3032 for quantizing.Quantizer module is controlled by the quantization parameter 3022 or 3042 from rate controller 3020 or 3040.In certain embodiments, each quantization parameter is set by corresponding rate controller, and is the function of one or more attributes of the camera relevant to this rate controller, as further described below.Rate controller can reduce the bit number for encoding by setting larger quantization step, or increases by setting less quantization step the bit number using.By controlling quantization step, rate controller is also determined in the video image of coding can introduce how many distortions.Thereby rate controller can be realized trading off between bit rate and picture quality.Realizing rate-distortion when compromise, rate controller Monitor bit (n.) rate, so that can overflow storage buffer, underflow storage buffer or exceed transmission channel capacity.Rate controller is necessary control bit rate also, to best possible picture quality is provided and avoids the unacceptable image quality distortion causing by quantizing.In certain embodiments, each rate controller is being kept in status buffer 3015 according to the monitor data of one group of status data value representation.In certain embodiments, it is compromise that the distinctive attribute of rate controller 3020 and 3040 use camera is optimized rate-distortion.

In certain embodiments, each rate controller, by directly to its quantization parameter application correction factor, is optimized rate-distortion compromise.In some such embodiment, correction factor is predetermined and is built in together with camera in equipment; Equipment does not need these correction factors of dynamic calculation.In other embodiments, system utilization, by the input picture of camera, is dynamically determined the suitable correction factor specific to camera.In some such embodiment, system, in repeatedly encoding, is analyzed a series of inputted video image by camera, to collect some statistics about camera.System utilizes these statisticss to obtain the correction factor of the quantization parameter of optimizing about this camera subsequently.

In certain embodiments, the distinctive correction factor of these cameras, via the visual masking attribute of video image, is applied to quantization parameter.The visual masking attribute of image or a part of image is the instruction that can tolerate how many coding pseudomorphisms in described image or image section.Some embodiment calculate the visual masking attribute of the luminance energy that quantizes described image or image section, and other embodiment calculates the described image of quantification or the activation energy of image section or the visual masking attribute of complexity.With how, to shelter attribute irrelevant for computation vision, and some embodiment utilize visual masking attribute calculate the correction of a frame of video or shelter quantization parameter.Some such embodiment are calculated as frame level visual masking attribute sheltering quantization parameter with benchmark visual masking attribute function.In certain embodiments, use video masking attribute with the quantization parameter of revising is expressed as:

Wherein MQP _framebe frame shelter or revise quantization parameter, QP _nominitial or nominal quantized value, and β _frameit is the constant that is suitable for local statistics.In certain embodiments, benchmark visual masking attribute with nominal quantization parameter QP _nompredetermined according to the initial or periodic evaluation of network condition.

In certain embodiments, the visual masking attribute in equation (1) be calculated as

Wherein avgFrameLuma is the average brightness value of frame, and avgFrameSAD is mean value and the number of the absolute difference of frame.Constant alpha, β, C, D and E are suitable for local statistics.In certain embodiments, these constants are suitable for the distinctive characteristic of camera.

Some embodiment also calculate certain part of video image, such as the quantization parameter of sheltering of macro block.In these cases, shelter quantization parameter and be calculated as macro block visual masking attribute function:

Wherein, in certain embodiments, β _mBthe constant that is suitable for local statistics, MQP _frameutilize equation (1) and (2) to calculate.In certain embodiments, the visual masking attribute in equation (3) be calculated as

Wherein avgMBLuma is the average brightness value of macro block, and avgMBSAD is mean value and the number of the absolute difference of macro block.Constant alpha, β, A, B and C are suitable for local statistics.In certain embodiments, these constants are suitable for the distinctive characteristic of camera.

Be better than and utilize as mentioned above the distinctive constant of multiple cameras to calculate the quantization parameter of correction, some embodiment, by only utilizing the distinctive coefficient of single camera to calculate quantization parameter, realize the speed control specific to camera.For example, known visual masking attribute with and quantization parameter QP _frame, some embodiment utilize the distinctive coefficient μ of single camera to carry out the quantization parameter of computing macro block:

For calculation equation (5), some embodiment are used separately as visual masking attribute the complexity measure of frame and macro block with

Some embodiment are at QP _mBthe different distinctive coefficient of camera of application in calculation.For example, in certain embodiments, QP _mBbe calculated as

Wherein ρ is the coefficient of adjusting according to the distinctive characteristic of camera.

As mentioned above, during status buffer 3015 is kept at video conference, the encoding state information that two different rate controller examples 3020 and 3040 can be shared, to obtain better coding result from their speed control operation.In certain embodiments, target bit rate R _tit is an example of this shared state information.This bit rate is the desirable bit rate to series of frames coding.In general, this bit rate is with bps for unit representation, and is according to such as the processing of describing in III.B joint is above determined.

As mentioned above, the rate controller of some embodiment utilizes target bit rate to calculate it to export to video encoder 3030 frame and/or macroblock quantization parameter QP.For example, some embodiment utilize current target bit rate to calculate quantification and adjust parameter, and these embodiment obtain the quantization parameter QP of macro block and/or frame from the quantification adjustment parameter of calculating.In certain embodiments, quantizing to adjust parameter is to utilize by the bit rate of former frame, or the fraction representation that calculates divided by current target bit rate of the runtime average of the bit rate of former frame.In other embodiments, described adjustment parameter is not accurate Calculation in this manner, (1) is proportional to the runtime average of the bit rate of former frame or the bit rate of former frame on the contrary in general, and (2) are inversely proportional to current target bit rate.

Calculating after such quantification adjusts parameter, the rate controller of some embodiment utilizes its macro block and/or frame quantization parameter of calculating of this parameter adjustment.A kind of mode of carrying out this adjustment is that the macro block calculating and/or frame quantization parameter are multiplied by quantification adjustment parameter.The another kind of mode of carrying out this adjustment is to adjust calculation of parameter offset quantizing parameter value according to quantizing, and subsequently this offset parameter is applied to macro block and/or the frame quantization parameter (for example deducting) of calculating.The rate controller of these embodiment is exported to video encoder 3030 macro block and/or the frame quantization parameter adjusted subsequently.

In other embodiments, rate controller utilizes target bit rate to calculate other parameter using in its speed control operation.For example, in certain embodiments, rate controller utilizes the visual masking intensity of this target bit rate correction macro block or frame.

G. manager networks

Figure 31 is the networking manager 3100 of some embodiment of graphic extension conceptually, such as the software architecture of the networking manager 1614 of graphic extension in Figure 16.As mentioned above, the network between the remote equipment in its two camera movement equipment and video conference of running on of networking manager 3100 management connects (for example, connect and set up, connect monitoring, connection adjustment, disconnecting etc.).During video conference, the networking manager 3100 of some embodiment is also processed the data that send remote equipment to, and processes the data that receive from remote equipment.

As shown in Figure 31, networking manager 3100 comprises session negotiation manager 3105, transmitter module 3115, general transmission buffer 3120, general transmission buffer-manager 3122, virtual transportation protocol (VTP) manager 3125, receiver module 3130 and media transfer management device 3135.

Session negotiation manager 3105 comprises protocol manager 3110.Protocol manager 3110 guarantees during video conference, and transmitter module 3115 uses correct communication protocol to transmit data to remote equipment, and forces to obey the rule of the communication protocol using.Some embodiment of protocol manager 3110 support various communications protocols, such as real time transport protocol (RTP), transfer control agreement (TCP), User Datagram Protoco (UDP) (UDP) and HTTP (HTTP), etc.

Connection between one or more remote equipments that session negotiation manager 3105 is responsible for setting up two camera movement equipment and participating in video conference, and after meeting, interrupt these connections.In certain embodiments, session negotiation manager 3105 be also responsible for setting up between the remote equipment in two camera movement equipment and video conference (for example, transmission and receiver, video and/or audio stream) multimedia communication sessions (for example, utilizing Session initiation Protocol (SIP)).

Session negotiation manager 3105 also receives feedback data from media transfer management device 3135, according to feedback data, by general transmission buffer-manager 3122, determine the operation (being for example, transmit or abandon packet/frame) of general transmission buffer 3120.In certain embodiments, this feedback can comprise unidirectional stand-by period and bandwidth estimation bit rate.In other embodiments, feedback comprises packet loss information and round trip delay time (for example,, according to sending to the grouping of the remote equipment in video conference and determining from the reception of the confirmation of remote equipment).According to the information from media transfer management device 3135, session negotiation manager 3105 can determine whether sending too much grouping, with instruction general transmission buffer-manager 3122, allow general transmission buffer 3120 transmit less grouping (, adjustment bit rate as shown in Figure 15).

Transmitter module 3115 from video buffer (for example, the buffer 1612 of Figure 16) (for example fetch coded image, with the form of bit rate), and to image subpackage so that by general transmission buffer 3120 and virtual transportation protocol manager 3125, send the remote equipment in video conference to.Produce coded image and coded image is sent to instruction or the data that the mode of transmitter module 3115 can be based on receiving from media transfer management device 3115 and/or session negotiation manager 3105.In certain embodiments, to image, subpackage relates to the bit stream receiving is divided into a component group, (each grouping all has specific size, the size of being specified according to specific protocol by session negotiation manager 3105), for example, with the header (, address header, agreement are specified header etc.) that increases any needs.

General transmission buffer-manager 3122, according to the data and/or the instruction that receive from session negotiation manager 3105, is controlled the operation of general transmission buffer 3120.For example, can transmit data, stop transmitting data, abandon data etc. by instruction general transmission buffer-manager 3122 order general transmission buffers 3120.As mentioned above, in certain embodiments, in the time that the remote equipment of conference participation seems lost packets, according to the confirmation receiving from remote equipment, can identify this situation.In order to reduce packet loss, can instruction general transmission buffer-manager 3122 transmit grouping with lower speed to remote equipment.

General transmission buffer 3120 is preserved the data that receive from transmitter module 3115, and by VTP manager 3125, these data is sent to remote equipment.As mentioned above, according to the instruction receiving from general transmission buffer-manager 3122, the discardable grouping of general transmission buffer 3120 (for example, the image of video).

In certain embodiments, RTP is used to transmit packet (for example, audio packet and video packets) by UDP during video conference.Other embodiment utilizes RTP to transmit packet by TCP during video conference.In different embodiment, also can use other transport-layer protocols.

Some embodiment, with pair of end slogan (, source port number and destination port numbers), define the specific communications passage between two mobile devices.For example, a communication port between mobile device can (for example be used pair of end slogan, source port 50 and destination port one 00) definition, and another different communication port between mobile device can for example, define different port numbers (, source port 75 and destination port one 50) with another.Some embodiment also utilize a pair of Internet protocol (IP) address to define communication port.In certain embodiments, different communication ports is used to transmit dissimilar packet.For example, can in independent communication port, transmit video data packets, audio data packet and control signal packet.Thereby video communication passage transmits video data packets, and audio communication channel transmits audio data packet.

In certain embodiments, control communication port for the information receiving between local mobile device during video conference and remote equipment.The example of this information receiving comprises sending and receiving request, notice and the confirmation to described request and notice.Another example of information receiving comprises from an equipment to another equipment transmission remote control commands message.For example, by the control communication port through local device, send the instruction of the operation of Long-distance Control remote equipment from local device to remote equipment, (for example can carry out the remote control operation that the following describes, commander equipment only sends the image from particular camera, or only utilizes particular camera photographic images).Different embodiment utilizes different agreements, such as transfer control agreement (RTCP), RTP expansion, SIP etc. realize described control communication in real time.For example, some embodiment utilize RTP expansion in video conference, between two mobile devices, to transmit one group of control message, and between described two mobile devices, transmit another group control message during using SIP to be grouped in video conference.

The VTP manager 3125 of some embodiment by single communication passage (for example allows, utilize identical pair of end slogan), transmit and specify the dissimilar packet for example, transmitting by different communication ports (, utilizing different multipair port numbers).Relate to identification data packet type for a kind of technology of this object, be prescribed and will pass through the communication port of its transmission by extracting the port numbers identification data grouping of a pair of appointment of packet, with carry out specified data grouping by the pair of end slogan that the pair of end slogan of packet is modified as to described single communication passage will be by the transmission of described single communication passage (, all packets be transmitted by identical pair of end slogan).

In order to understand a pair of initial port number of every kind of packet, some embodiment preserve the mapping with respect to a pair of initial port number of data packet type.Some such embodiment utilize subsequently the packet type field of agreement to distinguish and are multiplexed into a different grouping in communication port.For example, some embodiment with VTP manager are multiplexed to audio frequency, video and control packet in a rtp streaming, utilize RTP packet type field to distinguish the audio frequency, video and the control packet that in a RTP passage, are transmitted to another equipment in video conference.In some such embodiment, VTP manager is also given described another equipment the control message route in SIP grouping.

Some embodiment identify and check packet signature (, packet header form), for example, to distinguish the different grouping (, distinguish and utilize the grouping of RTP transmission and the grouping that utilizes SIP to transmit) of utilizing different agreements to transmit.In such embodiments, after determining the packet of different agreement, check as mentioned above the field of the packet (for example, utilizing voice data and the video data of RTP) that uses same protocol, to identify different data types.In this manner, VTP manager 3125, by single communication passage, transmits the predetermined different pieces of information grouping transmitting by different communication ports.

Although a kind of mode of the data dissimilar by single communication combination of channels has been described, but other embodiment utilizes other technology that different packet types is multiplexed in a communication stream above.For example, a kind of technology of some embodiment relates to a pair of initial port number of being careful packet, and described a pair of initial port number is kept in packet itself, to extract after a while.Also exist the dissimilar data between two video conferencing participants are combined to a port to other technology in passage.

When VTP manager 3125 is by virtual communication passage, while receiving packet from remote equipment, VTP manager 3125 checks the different grouping that the signature of packet utilizes different agreements to send with identification.This signature can be used to distinguish SIP grouping and RTP grouping.The VTP manager of some embodiment also utilizes the packet type field of some or all groupings, and multichannel is decomposed the various dissimilar grouping (for example, audio frequency, video and control packet) being multiplexed in single virtual passage.After these dissimilar groupings of identification, the port check mark that VTP manager is preserved according to it and the mapping of packet type, associate each dissimilar grouping and its corresponding port check mark.The pair of end slogan that VTP 3125 divides into groups with the pair of end slogan Update Table of identification subsequently, and forwarding data divides into groups to unpack.Utilizing different technology different packet types to be multiplexed in other embodiment in single channel, VTP manager uses different technology to resolve grouping.

By utilizing, such technology is multiplexed decomposes different groupings with multichannel, VTP manager 3125 (for example creates single virtual communication port, single to port numbers), transmit video data, voice data and control signal data by described single virtual communication port, and pass through described single virtual communication port from remote equipment audio reception, video and control packet.Thereby, from the viewpoint of network, data are to transmit by described single virtual communication port, and from the viewpoint of session negotiation manager 3105 and protocol manager 3110, video data, voice data and control signal data are to transmit by different communication ports.

Be similar to the image that sends the remote equipment in video conference to, the image that the remote equipment from video conference transmits receives by packet format.Receiver module 3130 receives grouping, and grouping is unpacked so that reconstructed image, afterwards Image Saving for example, in video buffer (, the buffer 1616 of Figure 16) so that decoding.In certain embodiments, image is unpacked to relate to and remove any header, and only there is the bit stream of view data (may have size data) by grouping reconstruct.

Media transfer management device 3135 process from network receive feedback data (for example, unidirectional stand-by period, bandwidth estimation bit rate, packet loss data, round trip delay time data, etc.), dynamically and adaptively to adjust the speed (, bit rate) of transfer of data.In some other embodiment, media transfer management device 3135 is also fault-tolerant according to feedback data control after treatment, also feedback data can be sent to video conference manager 1604, to adjust other operation of video conference module 1602, such as proportional zoom, size are adjusted and coding.Except when the remote equipment in meeting can not process all groupings time, abandons outside grouping general transmission buffer, video conference module and encoder can use compared with low bit rate Image Coding, make will to send less grouping concerning each image.

In certain embodiments, other variable that media transfer management device 3135 also can watch-dog, such as how meeting impact configures power consumption and the thermal level of the operating power pattern of camera, as mentioned above.These data also can be used as adding the additional input (for example,, if equipment just overheats, media transfer management device 3135 can manage to make to process and slow down so) in feedback data.

With reference now to Figure 16,, several exemplary operations of networking manager 3100 are described.First explanation sends the remote equipment in video conference to the image of the camera by two camera movement equipment, illustrates afterwards from remote equipment and receives image.Transmitter module 3115 is fetched and will be transmitted to the coded image of the remote equipment video conference from buffer 1612.

Protocol manager 3110 is determined the appropriate agreement (for example, transmitting the RTP of Voice & Video) that will use, session negotiation manager 3105 this notice of settlement to transmitter module 3115.Afterwards, transmitter module 3115 carries out subpackage to image, and the image of subpackage is sent to general transmission buffer 3120.General transmission buffer-manager 3122 receives instruction from session negotiation manager 3105, transmits or abandon image with order general transmission buffer 3120.VTP manager 3125 receives grouping from general transmission buffer 3120, and processes grouping, to grouping is sent to remote equipment by single communication passage.

In the time receiving image from remote equipment, VTP manager 3125 is by virtualized single communication passage, receive the image of subpackage from remote equipment, and process grouping so as the communication port that is used for receiving image by distribution (for example, video communication passage), image is directed to receiver module 3130.

Receiver module 3130 unpacks with reconstructed image grouping, and image is sent to buffer 1616, to decoded by decoder 1660.Receiver module 3130 is also transmitted to media transfer management device 3130 (confirmation of for example, dividing into groups from the reception of the remote equipment in video conference) control signal message.

Several exemplary operations of networking manager 3100 have been described above.These are illustrative example, because various other embodiment will utilize different modules, or in the situation that various functions are differently distributed between module, carry out these operations or different operations.In addition, module or other module of networking manager 3100 can be carried out other operation, such as dynamic bit rate adjustment.

IV. in meeting, adjust and control operation

A. picture-in-picture amendment

1. lock onto corner (snap-to-corner)

Some embodiments of the present invention allow the user of two camera movement equipment to form one or more viewing areas of compound display by moving around, and amendment is presented at the compound display on equipment.Such example is the insertion viewing area that PIP shows that moves around.Figure 32 is illustrated in such example of carrying out during video conference.In video conference, due to a variety of causes, block the area-of-interest of background viewing area such as inserting viewing area when prospect, user may want mobile prospect to insert viewing area.

Figure 32 is with reference to five different phases 3210,3215,3220,3225 and 3230 of UI 3205, and the movement of viewing area 3240 in the UI 3205 of equipment inserted in graphic extension.First stage 3210 is illustrated in the UI 3205 during the video conference between the local user of this equipment and the long-distance user of remote equipment.UI 3205 in Figure 32 shows that, with after starting video conference, the PIP showing in the five-stage of Figure 11 shows that identical PIP shows.In this example, the video of being taken by local user's equipment is displayed on and inserts in viewing area 3240, and the video of being taken by long-distance user's equipment is displayed in background viewing area 3235.As shown in the figure, viewing area 1155 comprises the optional UI project 3245 for finishing video conference.In certain embodiments, the layout of the layout of viewing area 1155 and the viewing area of Figure 12 described above 1155 is identical.

Second stage 3215 graphic extension user by selecting are inserted viewing area 3240, start and lock onto corner operation.In this example, be placed on any place of inserting in viewing area 3240 by finger 3255, realize and selecting.As shown in the figure, with the thick frame 3260 that inserts viewing area 3240, show this selection.Different embodiment can point out such selection by different modes, such as by highlighting district 3240, by viewing area 3240 is swung etc.

Phase III 3220 is illustrated in user and starts PIP to show that 3250 insertion viewing area 3240 shows that from PIP a region 3250 moves to this PIP and shows the UI 3205 after another region in 3250.In this example, insertion viewing area 3240 has started to move to PIP from the lower left corner of PIP demonstration 3250 and has shown 3250 the lower right corner, as shown in arrow 3265.In this example, select to insert demonstration in second stage 3215 after, user drags its finger 3255, the mobile viewing area 3240 of inserting by show 3250 the lower right corner towards PIP.Some embodiment provide to move around in PIP shows 3250 and insert other technology of viewing area 3240.

Fourth stage 3225 is illustrated in the UI 3205 under the state of user after the screen of equipment 3200 is removed its finger 3255.In this state, inserting viewing area 3240 still moves towards the lower right corner of pointing the mobile PIP demonstration 3250 being identified according to the user in the phase III 3220.In other words, start after insertion viewing area 3240 is moved in the lower right corner of PIP demonstration 3250 at finger 3255, still keep this movement even if removed finger 3255, UI 3205.In order to keep described movement, before user removes its finger 3255, the UI 3205 of some embodiment requires user's drag operation to be greater than specific threshold quantity (for example, be greater than specific distance, or be longer than the specific time); Otherwise these embodiment can, after mobile a little insertion viewing area 3240, make to insert viewing area 3240 and remain on its initial position, the lower left corner, or the mobile viewing area 3240 of inserting.

But, even although user is before insertion viewing area arrives its reposition, stopped his drag operation, some embodiment still allow to insert viewing area and continue mobile, but other embodiment requires user to keep drag operation, arrives its reposition until insert viewing area.Some embodiment provide mobile other technology of inserting viewing area.For example, some embodiment require user insert viewing area 3240 actual start mobile before, where appointment will be directed to inserting viewing area 3240, etc.Some embodiment also, by only with different angles inclination mobile device, make viewing area can slide and lock onto corner.

Five-stage 3230 is illustrated in and inserts viewing area 3240 and arrive it and be positioned at the UI 3205 after the reposition that PIP shows 3250 the lower right corner.In five-stage 3230, the elimination instruction of thick frame 3260 locks onto corner operation and completes.

For make the above-mentioned the 3rd, the 4th and five-stage 3220,3225 and 3230 in the movement of graphic extension easier, once the UI 3205 of some embodiment adopts user to make to insert viewing area and moves towards certain corner of PIP demonstration 3250, just allows to insert the locking discipline (snapping rule) of viewing area quick lock in to this corner.For example, when user drags and inserts viewing area 3240 while exceeding threshold quantity towards particular corner, the direction of motion of viewing area 3240 is inserted in UI 3205 identifications of some embodiment, determine that described motion has exceeded threshold quantity, thereby automated movement inserts viewing area 3240 subsequently, and further input without user the next grid point inserting in the UI 3205 that can be locked into of viewing area 3240.In certain embodiments, the unique grid point providing for locking insertion viewing area 3240 is the grid point that is positioned at four corners of PIP demonstration 3250.Other embodiment provides and (for example inserts in the UI 3205 that can be locked into of viewing area 3240, PIP shows in 3250) other grid point (, insert the side of viewing area 3240 or summit can be set to thereon or other grid point of alignment with it).

Other embodiment can not adopt grid point, makes to insert viewing area and can be placed in the arbitrfary point in PIP demonstration 3250.Again other embodiment provide allow that user opens or closes UI lock onto grid point feature.In addition, except the video of taking from equipment, different embodiment can allow user to various projects, such as the execution such as icon lock onto corner operation.

Two other example 3330 and 3335 that locks onto corner operation in Figure 33 graphic extension UI 3205.What these were other locks onto corner operation demonstration according to user's vertical or diagonal angle drag operation, the insertion viewing area 3240 of being moved by vertical or diagonal angle in PIP shows 3250.

Insert the movement of viewing area within PIP shows even if Figure 32 and 33 illustrates, can utilize similar technology mobile display district in the compound display of the PIP of other type demonstration or other type but those of ordinary skill will appreciate that other embodiment.For example, as further described below, the PIP demonstration of some embodiment has the prospect of two or more and inserts demonstration, by utilization and technology like Figure 32 and 33 technology types that illustrate, can in PIP shows, move these insertions and show.In addition, some embodiment utilize the similar technology viewing area (for example, drag movement by user, a viewing area is moved to the right side of screen from the left side of screen) that moves around in compound display.In addition, the movement of one or more viewing areas of compound display can cause the variation of the image processing operations of two camera movement equipment, such as response user's input, makes video conference manager 1604 again synthesize the viewing area in compound display.As further described below, some embodiment adopt in the time that the second viewing area is moved to primary importance from the 3rd position, the locking that the first viewing area is pushed open from primary importance and push technology.

2. rotation

In the time that the user of the mobile device for video conference rotates described mobile device in the session, some embodiment are rotated in the PIP presenting during video conference and show.Figure 34 graphic extension in the time making equipment 3400 rotate to horizontal level from upright position, the rotation of the UI1105 of equipment 3400.When the long limit of screen is while being vertical, equipment 3400 is held by perpendicular, and when the long limit shielding be level time, equipment 3400 is held by horizontal.In Figure 34, in the example of graphic extension, UI 1105 rotates from the perpendicular longitudinal view optimized of holding for equipment the horizontal transverse views of optimizing of holding that becomes equipment 3400.This spinfunction makes to be held or when horizontal holding, user can watch the UI 1105 showing with stand up position by perpendicular when mobile device 3400.

Figure 34 is according to six different operational phases 3410,3415,3420,3425,3430 and 3435, the rotation of graphic extension UI 1105.First stage 3410 is illustrated in the UI1105 during the video conference between the local user of this equipment and the long-distance user of remote equipment.UI 1105 in Figure 34 shows and after setting up video conference, the PIP showing in the five-stage of Figure 11 shows that identical PIP shows 1180.In this example, the video that local user's equipment is taken is presented at and inserts in viewing area 1160, and the video that long-distance user's equipment is taken is presented in background viewing area 1170.Show that at PIP the viewing area 1155 under 1180 comprises that user can select (for example to finish video conference, dub by singly referring to) optional UI project 3485 (for example, " closing session (End Conference) " button 3485).

Second stage 3415 is illustrated in user and starts banking equipment 3400 UI 1105 afterwards.In this example, user starts equipment 3400 to be held from erecting to hold to tilt to by horizontal, as shown in arrow 3460.The outward appearance of UI 1105 does not change.In other cases, user may change into want equipment 3400 from horizontal hold to tilt to by perpendicular hold, in these cases, UI1105 switches to the view of vertical optimization from the view of level optimization.

Phase III 3420 is illustrated in equipment 3400 from being tilted to by the horizontal UI 1105 holding state afterwards by perpendicular holding.In this state, the outward appearance of UI 1105 does not still change.In certain embodiments, after making equipment 3400 tilt to exceed threshold quantity and keeping exceeding threshold quantity a period of time, trigger rotation operation.In Figure 34, in the example of graphic extension, suppose that threshold quantity and rotary speed can not cause UI 1105 to rotate, until equipment has been placed on horizontal level short period interval afterwards.Different embodiment has different threshold quantities and the stand-by period for triggering rotation operation.For example, some embodiment can have the threshold value of so low triggering rotation operation, make the not orientation of tube apparatus 3400, all make UI 1105 seem as it is always shown with stand up position.In other embodiments, the user of equipment 3400 can specify when can trigger rotation operation (for example, preferentially selecting to arrange by menu).In addition, some embodiment can, after equipment is tilted and exceedes threshold quantity, not postpone described rotation.In addition, different embodiment can allow to trigger rotation operation by different modes, such as by switching switch on mobile device, by sending voice commands, according to the selection of being undertaken by menu, etc.

Fourth stage 3425 is illustrated in and starts rotation operation UI 1105 afterwards.Some embodiment flash demo rotational display districts, to provide the feedback about rotation operation to user.The example of the such a kind of animation of Figure 34 graphic extension.Specifically, Figure 34 its fourth stage 3425 show viewing area 1180 start together with 1155 rotation.Viewing area 1180 and 1155 is around axle 3465 (, the z axle) rotation at the center through UI 1105.Viewing area 1180 and 1155 is rotated identical quantity, but along the direction rotation contrary with the rotation (for example,, by the inclination of equipment 3400) of equipment 3400.In this example, because equipment 3400 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), therefore rotate operation and can make viewing area 1180 and 1155 half-twist in the counterclockwise direction.In the time of 1180 and 1155 rotation of viewing area, viewing area 1180 and 1155 dwindles pari passu, to be applicable to UI 1105, viewing area 1180 and 1155 still can be appeared on UI 1105 completely.Some embodiment can provide the message (for example,, by showing words " Rotating (rotation) ") of the state of indicating equipment 3400.

Five-stage 3430 is illustrated in viewing area 1180 and 1155 and is rotated counterclockwise the UI 1105 after 90 ° from longitudinal view to transverse views.In this stage, viewing area 1180 and 1155 has been rotated, but is not also deployed into the whole width of UI 1105.The ending that arrow 3475 is pointed out at five-stage, viewing area 1180 and 1155 will start side direction launches, to be applicable to the whole width of UI 1105.Different embodiment can not comprise this stage because can with fourth stage 3425 in rotation side by side carry out described expansion.

The 6th stage 3435 was illustrated in viewing area 1180 and 1155 and has been unfolded, thereby occupied the UI 1105 after the whole demonstration of UI 1105.As mentioned above, other embodiment can differently realize this rotation.Concerning some embodiment, only the screen rotation of equipment is exceeded to threshold quantity and just can trigger described rotation operation, and the orientation of tube apparatus 3400 not.

In addition, other embodiment can be provided for the different animations of instruction rotation operation.The rotation operation of carrying out in Figure 34 relates to viewing area 1180 and 1155 central rotations around UI 1105.On the other hand, can make viewing area around they separately the central shaft of viewing area rotate separately.A kind of such method has been shown in Figure 35.Figure 35 represents the alternative approach of the rotation of the viewing area 1170 and 1160 of the PIP demonstration 1180 of flash demo UI 1105.In Figure 35, in the PIP of graphic extension demonstration 1180 and Figure 11, the PIP of graphic extension shows that 1180 is identical.

Figure 35 is according to six different operational phases 3410,3415,3420,3525,3530 and 3535, and graphic extension PIP shows 1180 rotation.The operation of the front three phases of UI 1105 is with as identical in the operation of the front three phases of explanation in the UI 1105 in Figure 34.In the phase III of Figure 34 and 35, equipment 3500 all becomes horizontal holding from perpendicular holding, and the rotation of UI 1105 does not also start.

The alternative approach of rotating described in fourth stage 3525 graphic extension flash demos.In fourth stage, rotation operation starts.Specifically, fourth stage 3525 shows the beginning of the rotation of viewing area 1170 and 1160.Viewing area 1170 and 1160 rotates around the axle 3567 and 3565 (, z axle) at the center through each viewing area respectively.Viewing area 1170 and 1160 is rotated identical amount, but along the direction rotation contrary with the rotation (for example,, by the inclination of equipment 3500) of equipment 3500.With graphic extension similar in the fourth stage 3425 of Figure 34 above, because equipment 3500 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), therefore rotate operation and can make viewing area 1170 and 1160 half-twist in the counterclockwise direction.In the time of the rotation of viewing area 1170 and 1160, viewing area 1170 and 1160 scaled to be applicable to UI 1105, still can appear on UI 1105 viewing area 1170 and 1160 completely.

Five-stage 3530 is illustrated in viewing area 1170 and 1160 and is all rotated counterclockwise the UI 1105 after 90 ° from longitudinal view to transverse views.In this stage, viewing area 1170 and 1160 has been rotated, but is not also deployed into the whole width of UI 1105.In addition, viewing area 1160 is not also moved to its final position.The final position of insertion viewing area 1160 in PIP shows 1180 be the location positioning in PIP shows 1180 (for example, insert viewing area 1160 and show 1180 the lower left corner at PIP) by the insertion viewing area 1160 as shown in the first stage 3410.In this stage, insert viewing area 1160 still in the upper left corner of UI 1105.

The ending that arrow 3580 is pointed out at five-stage 3530, viewing area 1170 and 1160 will start side direction launches, until main viewing area 1170 is applicable to the whole width of the UI 1105 of the horizontal equipment of holding.In addition, arrow 3575 is pointed out to insert viewing area 1160 and will be slided into the lower left corner of PIP demonstration 1180.

Different embodiment can differently realize this point.In certain embodiments, the movement of inserting viewing area 1160 can occur with the expansion of main viewing area 1170 simultaneously, or can occur in sequence.In addition, some embodiment can insert viewing area 1160 by convergent-divergent before launch main viewing area 1170, in expansion or after launching, and show 1180 to produce new PIP.In this example, in the time of 1160 and 1170 rotation of viewing area, viewing area 1155 disappears.But, in certain embodiments, viewing area 1155 can during rotation be retained on UI 1105, and rotates together with 1170 with viewing area 1160.

The 6th stage 3535 was illustrated in inserts viewing area 1160 and arrives its repositions, and viewing area 1160 and 1170 appropriately launched, to be applicable to the UI 1105 after the whole width of UI 1105.In this example, insertion viewing area 1160 is positioned at now PIP and shows 1180 the lower left corner, overlaps on main viewing area 1170.PIP show 1180 existing have with the PIP of first stage 3410 show that 1180 identical demonstrations arrange.In the 6th stage, show that at PIP the viewing area 1155 instruction rotation operations that occur under 1180 complete.As mentioned above, only the screen rotation of equipment is exceeded to threshold quantity and just can trigger described rotation operation, and the orientation of tube apparatus 3500 not.

In example with reference to Figure 34 and 35 explanations, the orientation of viewing area 1170 also changes (, from longitudinally becoming laterally) in the above.That is, in the phase III 3420, after rotational display district 1170, show 1180 by horizontal development PIP, make it be applicable to whole UI 1105, the orientation of viewing area 1170 is from longitudinally becoming laterally.In certain embodiments, in the time that equipment 3500 is rotated, the video of being taken by remote equipment rotates, but shows that the orientation of the viewing area of the video of being taken by remote equipment remains unchanged.In Figure 36, illustrate such example.Figure 36 is similar to Figure 35, and except being presented at the video rotation in viewing area 1170, but viewing area 1170 keeps showing along the longitudinal direction.

In Figure 36, also illustrate wherein the example that viewing area 1155 remains on the rotation operation of same position (instead of rotate like that as shown in Figure 35 and horizontal development, show 1180 to fill up PIP).In addition, Figure 36 comprises and the layout of the viewing area 1155 that the layout in viewing area 1155 illustrated in fig. 12 is identical above.As shown in the figure, in the stage 3640,3645,3650,3655,3685 and 3690, in the time that equipment 3500 rotates, viewing area 1155 is retained in same position.

Some embodiment provide a kind of rotation operation, wherein show the directed change (instead of keeping like that as shown in Figure 35 identical orientation) of the viewing area of the video of being taken by local device, to be reflected in, local device is rotated to the operation orientation of local device afterwards.Figure 36 six different stages 3640,3645,3650,3655,3685 and 3690 of reference, the example of this rotation operation of graphic extension UI1105.In Figure 36, the first stage 3640 shows along the longitudinal direction inserts viewing area 1160, inserts viewing area 1160 and shows the video by the camera of equipment 3500.Second is identical with second and the phase III 3415 and 3420 of Figure 35 with the phase III 3645 and 3650, because their indication equipments 3500 are in the inclination in each stage of rotation operation.Now, the camera of equipment 3500 is along horizontal direction photographic images.In order to point out this transformation, some embodiment provide as the 4th and five-stage 3655 and 3685 as shown in animation, and other embodiment does not provide any animation.

In fourth stage 3655, be presented at the image inserting in viewing area 1160 and be rotated, itself be not rotated but insert viewing area 1160 because second and the phase III 3445 and 3650 in the inclination of equipment 3500 rotate to horizontal direction inserting viewing area 1160.In five-stage 3685, the image rotating inserting in viewing area 1160 is inserted viewing area 1160 by horizontal development to fill, insert viewing area 1160 and start to show that towards PIP 1180 left underside area moves, to be placed in and to insert viewing area 1160 at the identical relative position of the PIP of first stage 3640 demonstration inserting viewing area 1160.

In certain embodiments, show that the orientation of the viewing area of the video of being taken by remote equipment also changes, to be reflected in, remote equipment is rotated to the operation orientation of remote equipment afterwards.Four different phases of the UI 1105 of Figure 37 graphic extension equipment 3500, wherein (1) shows the change in orientation of the viewing area (viewing area 1160 in this example) of the video of being taken by local device, to be reflected in, local device is rotated to the operation orientation of local device afterwards, (2) change in orientation of the viewing area (viewing area 1170 in this example) of the video that demonstration is taken by remote equipment, is rotated the operation orientation of local device afterwards to be reflected in to remote equipment.

In the first stage 3705, the UI 1105 in UI 1105 and Figure 36 is identical.Specifically, the first stage 3705 shows the viewing area 1160 and 1170 of portrait orientation, because equipment 3500 shows by portrait orientation, and remote equipment is the (not shown) of portrait orientation.From the first stage 3705 to second stage 3710, by equipment 3500 from stand up position to lateral attitude half-twist, local device is rotated to operation.The rotation that second stage 3710 is presented at finishing equipment 3500 operates UI 1105 afterwards.In second stage, the video being presented in viewing area 1170 and 1160 has rotated to stand up position.But, only have the viewing area 1160 of local capture video to rotate to transversal orientation from portrait orientation, because only local device (, equipment 3500) has been carried out to rotation operation.Viewing area 1170 keeps portrait orientation.

From second stage 3710 to the phase III 3715, by remote equipment is rotated to lateral attitude (not shown) from stand up position, remote equipment is rotated to operation.The rotation that phase III 3715 has been presented at remote equipment operates UI 1105 afterwards.In the phase III, be presented at video in viewing area 1170 and the viewing area 1170 of long-range capture video and rotate to transversal orientation from portrait orientation, because only remote equipment has been carried out to rotation operation.Thereby the phase III of UI 1105 all shows the viewing area 1170 and 1160 of local and remote capture video by transversal orientation.

From the phase III 3715 to fourth stage 3720, by equipment 3500 from lateral attitude to stand up position half-twist, local device is rotated to operation.Fourth stage 3720 has been presented at the UI 1105 after this rotation operation.In fourth stage 3720, the video being presented in viewing area 1160 and 1170 has rotated to stand up position.But, only have the viewing area 1160 of local capture video to rotate to portrait orientation from transversal orientation, because only local device (, equipment 3500) has been carried out to rotation operation.Viewing area 1170 keeps transversal orientation.

From fourth stage 3720 to the first stage 3705, by remote equipment from lateral attitude to stand up position half-twist (not shown), remote equipment is rotated to operation.In this case, the first stage 3705 has been presented at this rotation operation viewing area 1170 afterwards.So the UI 1105 in this stage shows the viewing area 1160 and 1170 of portrait orientation.Although Figure 37 illustrates a series of different rotary operation, but other embodiment can be according to any many different sequences, carry out any many rotation operations.

Figure 34,35,36 and 37 illustrates the rotation operation of during video conference, local and remote device being carried out.In the time local mobile device being rotated to operation, some embodiment are rotating operational notification to remote equipment, for example, to allow remote equipment carry out any amendment (viewing area of the video of rotational display local device) to the video of local device.Similarly, in the time remote equipment being rotated to operation, remote equipment to local device, makes local device carry out any amendment to the video of remote equipment this operational notification.Some embodiment provide during video conference, transmit the control communication port of the notice of rotation operation between local device and remote equipment.

Although Figure 34,35,36 and 37 graphic extensions can realize the different modes of the animation of rotation, but will appreciate that other embodiment, those of ordinary skill can show by different modes the animation of rotation.In addition, the animation of rotation operation can cause the variation of the image processing operations of local mobile device, such as make video conference manager 1604 in UI 1105 with different angle again compound display district, and proportional zoom is presented at the image in viewing area.

3. window size adjustment

The size of the insertion viewing area that the PIP that some embodiment allow the user of mobile device to be adjusted to present during video conference shows.Different embodiment provides convergent-divergent to insert the different technologies of viewing area.Figure 38 graphic extension convergent-divergent inserts a kind of method of viewing area.In the method, the user by selecting of mobile device is inserted a corner of viewing area, expands subsequently or dwindles and insert viewing area, adjusts the size of inserting viewing area.

In Figure 38, the UI 3800 of mobile device 3825 with the long-distance user's of another mobile device video conference during, present PIP and show 3865.PIP shows that 3865 comprise that two videos show: the main viewing area 3830 of background and prospect are inserted viewing area 3835.The main viewing area 3830 of background occupies most PIP and shows 3865, and that prospect is inserted viewing area 3835 is less and overlap on the main viewing area 3830 of background.In this example, the main viewing area 3830 of background presents the personage's who holds guitar video, and this personage is assumed that the personage by positive its video of camera of remote equipment, or by the personage of the back side its video of camera of remote equipment.Prospect is inserted viewing area 3835 and is presented the personage's who wears a hat video, and in this example, this personage is assumed that the personage by positive its video of camera of local device, or by the personage of the back side its video of camera of local device.PIP show below 3865 be comprise be marked as " End Conference (closing session) " optional UI project 3860 (for example, button 3860) viewing area 1155, optional UI project 3860 allows this project of user by selecting to finish video conference.

PIP shows that 3865 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace and having for the showing compared with overall background of the video from remote equipment, larger background shows it can is the video from local device, and less prospect inserts to show it can is the video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear in UI3800 two viewing area (for example, left and right display window, or upper and lower display window) side by side, or in the viewing area of arranging, two diagonal angles.In certain embodiments, the mode that PIP shows or acquiescence display mode can be specified by user.In other embodiments, PIP shows that also can comprise a larger background shows, and two less prospects are inserted demonstration.

Figure 38 is according to four operational phases of UI 3800, graphic extension zoom operations.In the first stage 3805, prospect is inserted viewing area 3835 and is significantly less than the main viewing area 3830 of background.In addition in this example, prospect insertion viewing area 3835 is positioned at the lower right corner of PIP demonstration 3865.In other example, prospect is inserted viewing area 3835 can have different sizes, or is arranged in the zones of different of PIP demonstration 3865.

In second stage 3810, initiate zoom operations.In this example, for example, by selecting user to want a corner (, being placed on the upper left corner of inserting viewing area 3835 by finger 3840) of the insertion viewing area 3835 of convergent-divergent, initiate this operation.Second stage 3810 use of UI 3800 are inserted the thick frame 3845 of viewing area 3835 and are pointed out this selection.In this stage, user can expand or dwindle insertion viewing area 3835 (for example,, by showing at PIP on 3865 away from inserting viewing area 3835 or dragging its finger 3840 towards inserting viewing area 3835).

Phase III 3815 is illustrated in as shown in arrow 3850, by moving its finger 3840 (in this example away from inserting viewing area 3835, move his finger by the diagonal angle, the upper left corner towards UI 3800), user starts to expand insertion viewing area 3835 UI 3800 afterwards.As shown in arrow 3855, the movement of finger 3840 has expanded insertion viewing area 3835 pari passu at height and Width in addition.In other example, user can utilize identical technology (, by dragging finger towards inserting viewing area 3835), dwindles and inserts viewing area 3835.

Fourth stage 3820 is presented at the UI 3800 after the convergent-divergent that completes insertion viewing area 3835.In this example, reached desirable size once insert viewing area 3835, user is just by stopping its finger 3840 drags so, and shows that from PIP 3865 remove his finger, completes the convergent-divergent of insertion viewing area 3835.As the result of this process, the insertion viewing area 3835 after convergent-divergent is greater than its initial size in the first stage 3805.The elimination instruction of thick frame 3845 has completed insertion viewing area zoom operations now.

Some embodiment provide and allow user during video conference, and in PIP shows 3865, convergent-divergent inserts other technology of viewing area 3835.A kind of other such technology of Figure 39 graphic extension.An a kind of edge (that is, inserting one of side of viewing area 3835) that inserts viewing area 3835 by selection of Figure 39 graphic extension, expands subsequently or dwindles and insert viewing area 3835, carrys out convergent-divergent and inserts the technology of viewing area 3835.

Figure 39 is according to four operational phases of the UI of Figure 38 3800, this zoom operations of graphic extension.First stage 3805 in first stage 3805 and Figure 38 of Figure 39 is identical.Specifically, in this stage, UI 3800 graphic extensions of equipment 3925 have the main viewing area 3830 of larger background and are positioned at the PIP demonstration 3865 of the less prospect insertion viewing area 3835 in the lower right corner of PIP demonstration 3865.Although Figure 38 and 39 is illustrated in convergent-divergent in identical UI 3800 and inserts two kinds of different technologies of viewing area 3835, but will appreciate that some embodiment, those of ordinary skill will in identical UI, not provide this two kinds of technology.

The beginning of second stage 3910 graphic extension zoom operations.In this example, this user of user by selecting wants a side (for example, finger 3840 is placed on the top margin or side that inserts viewing area 3835) of the insertion viewing area 3835 of convergent-divergent, initiates zoom operations.In this example, user is placed on his finger 3840 on the top margin that inserts viewing area 3835, to realize described selection.Second stage 3910 use are inserted the thick frame 3845 of viewing area 3835 and are indicated this selection.

Phase III 3915 is illustrated in as shown in arrow 3950, and user moves apart insertion viewing area 3835 (that is, vertically moving towards the top of PIP demonstration 3865) by making it point 3840, starts to expand insertion viewing area 3835 UI 3800 afterwards.In addition, as shown in arrow 3955, the movement of finger 3840 has all expanded insertion viewing area 3835 in proportion at height and Width.In other example, user, by utilizing identical technology (, by dragging finger 3840 towards inserting viewing area 3835), dwindles viewing area 3835.

Fourth stage 3920 has been presented at the UI3800 after the convergent-divergent that inserts viewing area 3835.In this example, reach after desirable size once insert viewing area 3835, user is just by stopping dragging of its finger 3840 so, and removes his finger 3840 from the display screen of equipment, completes the convergent-divergent of insertion viewing area 3835.As the result of this process, the insertion viewing area 3835 after convergent-divergent is greater than its initial size in the first stage 3805.The elimination instruction of thick frame 3845 has completed insertion viewing area zoom operations now.

Response drag operation, some embodiment, at height and Width, adjust the size of inserting viewing area 3835, as shown in Figure 38 and 39 pari passu.Other embodiment can allow user to adjust the height and/or the width that insert viewing area 3835, and does not affect another attribute.The example of a kind of such convergent-divergent process of Figure 40 graphic extension.

Specifically, the similar UI 3800 of Figure 40 graphic extension mobile device UI 3,800 4025 and Figure 38, except when select the level in the lump at the edge that inserts viewing area 3835 or vertically when mobile this edge, the UI 3800 of Figure 40 allows to insert outside viewing area 3835 along continuous straight runs and/or vertical direction expansion.PIP in Figure 40 graphic extension UI 3800 and Figure 38 shows that 3865 similar PIP show 3865, except present insertion viewing area 3835 is the upper right corner of PIP demonstration 3865.PIP shows that 3865 comprise that two videos show: the main viewing area 3830 of background and prospect are inserted viewing area 3835.In this example, the main viewing area 3830 of background presents by the front camera of remote equipment or the video of back side camera.Prospect is inserted viewing area 3835 and is presented the video of being taken by front camera or the background camera of local device.

Be similar to Figure 38, Figure 40 is according to four operational phases of UI 3800, graphic extension zoom operations.First stage 4005 is similar with the first stage 3805 of Figure 38, except present insertion viewing area 3835 is the upper right corner.Other three phases 4010,4015 and 4020 is similar to three phases 3910,3915 and 3920, except insert viewing area 3835 base selection and mobile only cause inserting viewing area 3835 and vertically expand, and do not affect outside the width of insertion viewing area 3835.

Figure 38,39 and 40 provides the corner or the side that insert viewing area 3835 by selection, allows user's convergent-divergent PIP to show the example embodiment of 3865 insertion viewing area 3835.Some embodiment provide convergent-divergent to insert other technology of window 3835.For example, some embodiment of Figure 41 graphic extension allow to insert by selection the inside of viewing area 3835, and convergent-divergent inserts viewing area 3835.In this method, user is placed on screen by pointing 4155 and 4156 two, and drags away from each other or closely described two fingers, adjusts the size of inserting viewing area 3835.

In Figure 41, the UI 3800 of mobile device 4140 with the long-distance user's of another mobile device video conference during, provide PIP to show 3865.In order to simplify the explanation of UI 3800, in Figure 41 graphic extension UI 3800, show that with the PIP of Figure 38 3865 similar PIP show 3865.

Figure 41 is according to seven operational phases of UI 3800, this zoom operations of graphic extension.Front four-stage 3805,4110,4115 and 4120 represents to insert the expansion of viewing area 3835, and then three phases represents to insert dwindling of viewing area 3835.First stage 3805 in first stage 3805 and Figure 38 in Figure 41 is identical.Specifically, in this stage, UI 3800 graphic extensions have the main viewing area 3830 of larger background and less prospect and insert the PIP of viewing area 3835 and show 3865.In this example, the main viewing area 3830 of background presents by the front camera of remote equipment or the video of back side camera.Prospect is inserted viewing area 3835 and is presented by the front camera of local device or the video of back side camera.

Second stage 4110 is illustrated in initiates zoom operations UI 3800 afterwards.In this example, this user of user by selecting wants the insertion viewing area 3835 (for example, by two fingers 4155 and 4156 are placed on and are inserted in viewing area 3835) of convergent-divergent, initiates zoom operations.Second stage 4110 use of UI 3800 are inserted the thick frame 4190 of viewing area 3835 and are pointed out this selection.

Phase III 4115 is illustrated in as shown in arrow 4160, (user by mutually remotely moving its finger 4155 and 4156, show 3865 upper left corner moveable finger 4155 towards PIP, show 3865 lower right corner moveable finger 4156 towards PIP), start to expand insertion viewing area 3835 UI 3800 afterwards.As shown in arrow 4165, the movement of finger 4155 and 4156 has expanded insertion viewing area 3835 pari passu at height and Width.

Fourth stage 4120 has been presented at the UI3800 after the convergent-divergent that inserts viewing area 3835.In this example, user is by stopping its finger 4155 and 4156 drags, and removes its finger 4155 and 4156 from the display screen of equipment, completes the convergent-divergent that inserts viewing area 3835.As the result of this process, the insertion viewing area 3835 after convergent-divergent is greater than its initial size in the first stage 3805.The elimination instruction of thick frame 4190 has completed now inserts viewing area zoom operations.

At five-stage 4125, by two fingers 4155 and 4156 are placed on and are inserted on viewing area 3835, user reselects and inserts viewing area 3835.The 6th stage 4130 was illustrated in as shown in arrow 4170, and user, by move its finger 4155 and 4156 close to each otherly, starts to dwindle insertion viewing area 3835 UI 3800 afterwards.As shown in arrow 4175, the movement of finger 4155 and 4156 has been dwindled insertion viewing area 3835 pari passu at height and Width.

The 7th stage 4135 was similar to the fourth stage 4120 of Figure 41, except by described operation, outside the size of inserting viewing area 3835 has been dwindled.The elimination instruction of thick frame 4190 has completed now inserts viewing area zoom operations.

The above-mentioned explanation of Figure 38-41 for example understands several example user interface of the insertion viewing area that allows user's convergent-divergent PIP demonstration.In certain embodiments, the convergent-divergent that inserts viewing area can cause the variation of the image processing operations of two camera movement equipment, such as making video conference manager 1604 respond user's input, changes the proportional zoom of the insertion viewing area in PIP demonstration with synthetic.In addition, in certain embodiments, the layout of viewing area 1155 in Figure 38-41 is identical with the layout of the viewing area of Figure 12 described above 1155.

4. identification area-of-interest

Some embodiment allow user during video conference, area-of-interest (ROI) in the video that identification shows, for example, so that the processing of amendment image (, image processing manager 1608 in Figure 16), coding (for example, encoder 1655 in Figure 16), the behavior during video conference of mobile device and camera thereof, or their combination.Different embodiment provides different technology to identify this area-of-interest in video.Figure 42 graphic extension is for identifying the area-of-interest of video, to improve the user interface of some embodiment of the picture quality of video.

In Figure 42, the UI 4200 of mobile device 4225 with the long-distance user's of another mobile device video conference during, present PIP and show 4265.PIP in Figure 42 show substantially with Figure 41 in PIP show similar.Specifically, the demonstration of the PIP in Figure 42 comprises that two videos show: the main demonstration 4230 of background and prospect are inserted and shown 4235.In this example, the main demonstration 4230 of background presents the personage's who sets and wear a hat video, described tree and personage are assumed that tree and the personage by positive its video of camera of remote equipment, or by tree and the personage of the back side its video of camera of remote equipment.Prospect is inserted and is shown 4235 videos that present man, and in this example, described man is assumed that the man by positive its video of camera of local device, or by the personage of the back side its video of camera of local device.Below PIP shows, be comprise be marked as " End Conference (closing session) " optional UI project 4260 (for example, button 4260) viewing area 1155, optional UI project 4260 allows this project of user by selecting to finish video conference.

This PIP shows a kind of mode of the synthetic view that just presents the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace and having for the showing compared with overall background of the video from remote equipment, larger background shows it can is the video from local device, and less prospect inserts to show it can is the video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear in UI two viewing area (for example, left and right display window, or upper and lower display window) side by side, or in the viewing area of arranging, two diagonal angles.In other embodiments, PIP shows that also can comprise a larger background shows, and two less prospects are inserted demonstration.In certain embodiments, the mode that PIP shows or acquiescence display mode can be specified by user.

Figure 42 is according to four operational phases of UI 4200, and graphic extension ROI identifies operation.As shown in the first stage 4205, be presented on background and show that the video in 4230 has very low quality (, video image is fuzzy).In this example, the user of mobile device 4225 is intended to background to show in 4230 and occur that the region of personage's face 4270 is identified as area-of-interest.

In second stage 4210, initiate the operation of identification area-of-interest.In this example, (be for example presented on region that background shows that user among the video in 4230 wants to be identified as area-of-interest by selection, by on the screen of equipment, near the position character facial 4270 of the demonstration in background shows 4230 dubs finger 4250), initiate this operation.

As shown in the phase III 4215, the region that user's region is selected that UI 4200 is drawn and selected around user for example, around frame 4275 (, dashed rectangle 4275).Fourth stage 4220 is presented at the UI 4200 after the mark that finishes area-of-interest.As the result of this process, compared with the first stage 4205, the quality of the video in area-of-interest significantly improves.Complete now ROI around the elimination instruction of frame 4275 and selected operation.In certain embodiments, ROI identification process also causes to the same video being presented on remote equipment the identical variation of variation local device 4225 being caused with it.For example, in this example, the image quality being presented in the area-of-interest of the same video on remote equipment also significantly improves.

In certain embodiments, user can around frame 4275 (for example zoom in or out in the phase III 4215, be placed in display frame by finger 4250, and towards the upper right corner moveable finger 4250 of screen, with amplifying ring around frame 4275, or towards the lower left corner moveable finger 4250 of screen, to dwindle around frame 4275).Some embodiment also allow user at phase IIIs 4215 rotating ring for example, around frame 4275 (, be placed in display frame by finger 4250, and in display frame level or vertical moveable finger 4250).In some other embodiment, the selection in described region can not make UI 4200 draw around frame 4275 in the phase III 4215.

Other embodiment provides the different technologies that allows user to identify the area-of-interest in video.A kind of other such technology of Figure 43 graphic extension.In Figure 43, user, by drawing the shape of Sensurround region-of-interest, identifies area-of-interest.In this example, described shape is rectangle, but also can be other shape (for example, any other polygon, circle, ellipse etc.).Some embodiment, in the equipment UI of that technology that graphic extension in Figure 42 is also provided, provide the alternative technique of Figure 43.But, other embodiment does not provide these two kinds of technology in identical UI.

Figure 43 is according to five operational phases of UI 4200, and this ROI mark of graphic extension operates.First stage 4205 in first stage 4205 and Figure 42 in Figure 43 is identical.Specifically, in the first stage 4205, UI 4200 graphic extensions have the main demonstration 4230 of larger background, and are positioned at the PIP demonstration 4265 of the less prospect insertion demonstration 4235 in the lower left corner of PIP demonstration 4265.

In second stage 4310, initiate the operation of identification area-of-interest.In this example, select primary importance that definition is presented on the area-of-interest in the video presenting in the main demonstration 4230 of background (for example by continuing for some time, by on the screen of equipment, continuing for some time finger 4350 is placed on background and shows near the position shown character facial 4270 in 4230), initiate described operation.In the phase III 4315, UI 4200 utilizes the round dot 4355 that closes on the selected primary importance on background viewing area 4230, and described primary importance 4370 has been selected in instruction.

Fourth stage 4320 is illustrated in the UI 4200 after the second place 4375 that user selected definition area-of-interest.In this example, by as shown in arrow 4360, after round dot 4355 occurs, start to drag finger 4350 from primary importance in the screen of equipment, and stopping at the position between the cap of the demonstration in background viewing area 430 and the tree of demonstration, user selects the second place 4375.As shown in fourth stage, this rectangular shaped rim 4365 that causes UI4200 to draw area-of-interest that drags, described rectangular shaped rim 4365 has the first and second positions 4370 and 4375 on its relative summit.

The mark that five-stage 4325 is illustrated in area-of-interest has completed UI4200 afterwards.In this example, once user by having identified the area-of-interest of expecting, just stops pointing 4350 drag and remove finger 4350 from the display screen of equipment, complete the mark of area-of-interest.Five-stage 4325 graphic extensions are by this dilatory (drawing) process, and compared with the first stage 4205, the quality of the video in area-of-interest significantly improves.In certain embodiments, the same with the variation that it causes local device 4225, this dilatory process also causes identical variation to the display frame on remote equipment.For example, in this example, the picture quality being presented in the area-of-interest of the same video on remote equipment can significantly improve.

Figure 42 above and 43 description for example understand the area-of-interest in identification video, to improve the different modes of the picture quality of institute's identified region.In certain embodiments, the image quality of improving the area-of-interest of identification can cause the variation of the encoding operation of two camera movement equipment, such as when to Video coding, to the more bit of identified region allocation.

Some embodiment allow user to identify the area-of-interest in video, so that mobile device or its camera are made to different changes.For example, the area-of-interest in Figure 44 graphic extension identification video, to expand or to dwindle the example of area-of-interest in display frame.In this method, user, by selecting a certain region in display frame as the center of area-of-interest as, expands subsequently or dwindles described area-of-interest, identifies the area-of-interest in video.

In Figure 44, the UI 4400 of mobile device 4425 with the long-distance user's of another mobile device video conference during, present PIP and show 4265.PIP in Figure 44 shows that 4265 show that with the PIP of Figure 42 4265 is similar substantially, but the insertion of the prospect of Figure 44 shows that 4235 are positioned at the lower left corner of PIP demonstration 4265.

Figure 44 is according to four operational phases of UI 4400, and graphic extension ROI selects operation.As shown in the first stage 4405, background shows that 4430 present and have the man who shows 4430 left side in background, and shows the video of the tree 4440 on 4430 right side in background.In addition, tree 4440 is relatively little, only occupies the right side of background viewing area 4430.In this example, the user of mobile device 4425 is intended to the region that occurs tree 4440 on background viewing area 4430 to be identified as area-of-interest.

In second stage 4410, initiate the operation of identification area-of-interest.In this example, be presented on background by selection and (for example show region 4440 among the video in 4430, that user wishes to be identified as area-of-interest, by two fingers 4445 and 4446 being placed on background viewing area 4430, being shown the place of tree 4440), initiate described operation.In second stage 4410, by dragging away from each other its finger 4445 and 4446, user can make area-of-interest 4440 expand, and occupies the background viewing area 4430 of greater part.By close to each other its finger 4445 and 4446 that drags, user can also make area-of-interest 4440 dwindle, thereby occupies the background viewing area 4430 of smaller portions.

Phase III 4415 is illustrated in as shown in arrow 4450, (user by moving away from each other its finger 4445 and 4446, finger 4445 moves towards the upper left corner of background viewing area 4430, finger 4446 moves towards the lower right corner of background viewing area 4430), start to make area-of-interest 4440 to expand to occupy the UI4400 after the background viewing area 4430 of greater part.In certain embodiments, finger is mobile also causes to the demonstration of remote equipment the same variation of variation local device being caused with it.For example, in this example, the area-of-interest of same video will expand, thereby occupy the background viewing area 4430 of the greater part of remote equipment.In certain embodiments, the expansion of the area-of-interest in local demonstration and/or long-range demonstration causes one or two mobile device or their one or more other operations of its camera amendment, as further described below.

The mark that fourth stage 4420 is presented at area-of-interest has completed UI 4400 afterwards.In this example, once area-of-interest reaches after the ratio of expectation in background viewing area 4430, user is by stopping user's the dragging of finger 4445 and 4446 so, and removes finger 4445 and 4446 from the display screen of equipment, completes the mark of area-of-interest.As the result of this process, area-of-interest has occupied most background viewing area 4430.Complete now the mark operation of area-of-interest.

Some examples above illustrate user and how to identify the area-of-interest in video, to improve the picture quality (for example,, by increasing the bit rate of the encoding region of interest to video) in the selected area-of-interest in video.In certain embodiments, the area-of-interest in identification video causes the image processing operations of mobile device, such as the variation of exposure, proportional zoom, focusing etc.For example, the area-of-interest of identification in video can cause the differently image (for example, the area-of-interest of zoom is wanted in identification) of proportional zoom and synthetic video of video conference manager 1604.

In other embodiments, the area-of-interest in identification video causes the variation (for example, frame rate, zoom, exposure, proportional zoom, focusing etc.) of the operation of the camera of mobile device.In other other embodiment, the area-of-interest in identification video causes the variation of the encoding operation of mobile device, such as the more bit of region allocation to identification, proportional zoom etc.In addition, although example ROI mark described above operation only can cause the one amendment in above-mentioned amendment to mobile device or its camera, but, in some other embodiment, ROI mark operation meeting causes more than a kind of amendment the operation of mobile device or its camera.In addition, in certain embodiments, the layout of viewing area 1155 of Figure 42-44 is identical with the layout of the viewing area of Figure 12 described above 1155.

B. switch camera

Some embodiment provide during video conference, switch the method for camera (, changing the camera of photographic images).Different embodiment provides and realizes the distinct methods that switches camera operation.The camera that some embodiment provide this equipment of switching of being carried out by two camera movement equipment (, local switching) method, and other embodiment switches the method for the camera (, long-range switching) of described another equipment for two camera movement equipment provides another pair of camera movement equipment in instruction video conference.In addition other embodiment provides the method for carrying out local switching and long-range switching simultaneously.Local processing of switching camera operation on two camera movement equipment, is carried out in explanation by IV.B.1 joint.IV.B.2 joint carries out explanation the processing of long-range switching camera operation on two camera movement equipment.

1. the local camera that switches

Some embodiment of Figure 45 graphic extension carry out at local two camera movement equipment, with comprise the video conference of remote-moving apparatus of at least one camera during, the processing 4500 of switching between two cameras of local device.In certain embodiments, processing 4500 is carried out by the video conference manager 1604 shown in Figure 16.For the purpose of illustrating, discussion below will call camera 1 a camera of local two camera movement equipment, and another camera of local two camera movement equipment is called to camera 2.

By start (4505) video conference between local two camera movement equipment and remote-moving apparatus, start to process 4500.Subsequently, process 4500 the video image of the current selected camera (for example, camera 1) from local two camera movement equipment is sent to (4510) remote-moving apparatus, to show on remote-moving apparatus.4510, process 4500 video images that also receive from remote-moving apparatus according to described video image and it, produce and demonstration composograph.

Process 4500 and determine subsequently whether (4515) receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer local two camera movement equipment user request (for example, by the user interface of local two camera movement equipment), or answer remote-moving apparatus user request (for example, by the user interface of remote-moving apparatus), finish video conference.In the time that processing 4500 receives the request that finishes video conference, process 4500 and finish.

Do not receive while finishing the request of video conference when processing 4500, process 4500 users that determine subsequently (4520) local two camera movement equipment whether this local device of instruction switch the camera for video conference.In the time processing 4500 definite (4520) described local devices by instruction switching camera, process 4500 and return to operation 4510.But, in the time processing 4500 definite (4520) described local devices by instruction switching camera, process 4500 and enter operation 4525.

4525, process 4500 and send and point out that local two camera movement equipment will switch the notice of camera to remote-moving apparatus.In certain embodiments, process 4500 by as mentioned above, send described notice by the multiplexed video conference control channel of VTP manager 3125 with voice-grade channel together with video channel.

After sending its notice, process 4500 and carry out (4530) switching camera operation.In certain embodiments, carry out (4530) switching camera operation and comprise that instruction CIPU stops, with camera 1 capture video image, starting with camera 2 capture video images.The image of catching from the pel array associated with camera 2 is switched in these instructions only instruction CIPU, and starts to process these images.On the other hand, in certain embodiments, can follow one group of initiation parameter to the instruction of CIPU, described one group of initiation parameter instruction CIPU:(1) according to specific one group of setting operation camera 2, (2) catch with specific frame rate the video being produced by camera 2, and/or (3) for example, process the video image from camera 2 according to specific one group of setting (, resolution etc.).

In certain embodiments, switch camera instruction (4530) and also comprise the instruction that untapped camera is switched to the 4th kind of operating power pattern as above.In this example, switching camera instruction comprises to instruction camera 2, that be switched to its 4th kind of operating power pattern.In addition, switch camera instruction also comprise to camera 1, be switched to another kind of operating power pattern from its 4th kind of operating power pattern, such as the first operating power pattern, to save electric power, or be switched to the third operating power pattern, make in the time being required photographic images, it can be switched to the 4th kind of operating power pattern fast, and starts the instruction of photographic images.Switch the image that camera operation 4530 also relates to the synthetic image (instead of image of being taken by camera 1) of being taken by the camera 2 of local two camera movement equipment and receives from remote-moving apparatus, to show on two camera movement equipment.

After camera is switched in 4530 instructions, process 4500 and on local two camera movement equipment, carry out (4535) and switch camera animation, to be presented at from the demonstration of the image of camera 1 with from the transformation between the demonstration of the image of camera 2.After switching camera animation on local two camera movement equipment, process 4500 circulations through operation 4510-4520, finish video conference request or new switching camera request until receive.

How some embodiment of Figure 46 graphic extension allow the UI1105 request by two camera movement equipment to switch camera operation, and these embodiment how flash demo switch an example of camera operation.Figure 46 is according to eight of the UI of this equipment 1105 different operational phases 4610,4615,4620,4625,4630,4635,4640 and 4645, and camera operation is switched in graphic extension.The front four-stage graphic extension of UI 1105 receives the example of user's switching camera request.In some embodiments of the invention, the user of this equipment has other mechanism that produces this request.

The first stage 1130 of the UI 1105 of first stage 4610 and Figure 11 is identical, and it is illustrated in sets up video conference UI 1105 afterwards.In this stage, UI 1105 shows the PIP demonstration that comprises that two videos show: the larger background from remote camera shows, and inserts and show from the less prospect of local camera.In this example, the main viewing area 1170 of background presents a Ms's video, in this example, described Ms is assumed that the Ms who is taken its video by remote equipment, and prospect is inserted the video that viewing area 1160 presents a man, in this example, described man is assumed that the man by positive its video of camera of local device.

Second stage 4615 shows subsequently by the selection of the PIP viewing area 1180 of UI 1105, starts to switch camera operation.As shown in the figure, show on 1180 by user's finger 4670 being placed on to PIP, realize described selection.Phase III 4620 shows and comprises for during video conference, the UI 1105 of the optional UI project 4675 (for example, switching camera button 4675) that request is switched between the camera of local device 4600.For example, after the user that fourth stage 4625 is illustrated in local device 4600 selects (, by singly refer to dub) optional UI project 4675, and UI 1105 after pointing out this selection by highlighting of optional UI project 4675.By selecting this optional UI project 4675, user instruction equipment 4600, during video conference, is switched to the back side camera of equipment 4600 from the front camera of equipment 4600.In other example of the back side camera video of equipment 4600, user is switched to the front camera of equipment 4600 from the back side camera of equipment 4600 to the selection meeting commander equipment 4600 of optional UI project 4675 therein.After fourth stage, video conference manager sends instruction to CIPU, and remote equipment starts to switch camera operation.

The example of the switching camera animation on rear four-stage 4630,4635, the 4640 and 4645 graphic extension local devices of UI 1105.This animation intention causes from the front camera of local device and the video of back side camera and is presented at concomitantly the impression the tow sides of observing pane, and at any time, described observation pane is merely able to allow user see one of its tow sides.When in the time that video conference intermediate request is switched camera, make this observation pane seemingly around vertical axis revolving, make to show that the video of the camera presenting on the side of video of a camera is rotated away from user observing the first forward direction user of pane, until another side of its observed pane replaces, described another side shows the video of another camera.The rotation animation of this observation pane of experiencing and phenomenon are in the viewing area for a camera by (1), dwindle gradually the video image from this camera, and to described video image application perspective correction operation, (2) are in described viewing area subsequently, expand gradually the video image from another camera, and reduce the perspective correction operation of described video image to realize.

Therefore, five-stage 4630 graphic extensions are around the beginning of vertical axis 4682 " observing the rotation of pane ".Observe the rotation phenomenon of pane in order to provide, UI 1105 has reduced the size of the video image of the front camera in video viewing area 1160, and has applied pivot operation, and to seem, the right side of video image is further from user compared with the left side of video image.

Pane half-twist is observed in the 6th stages 4635 graphic extensions, makes user can only see the edge of pane, as is presented at as shown in the central fine rule 4686 of viewing area 1160.The graphic extension of the 7th stages 4640 is observed pane and is continued rotation, the back side of observing pane 4688 is engendered now, to show from the video of user's back side camera in face of user.Equally, in certain embodiments, this performance of rotation animation is video image by reducing the back side camera size in video viewing area 4688, and applies pivot operation, to make compared with the right side with video image, realize further from user in the left side of video image.

The 8th stages 4645 graphic extension represents to switch the completing of animation of camera operation.Specifically, this stage shows the video image by the automobile of the back side camera of equipment 4600 in viewing area 1160.

Example above with reference to Figure 46 explanation calls switching camera operation by switching camera user interface.Other embodiment differently calls switching camera operation.For example, some embodiment, by making to switch the UI during the optional UI project of camera is for good and all presented at video conference, for example, on the UI 1105 of Figure 47, call switching camera operation.In Figure 47, switch camera button 1289 and be displayed in viewing area 1155 together with mute button 1285 and closing session button 1287.The layout of viewing area 1155 is identical with the layout of the viewing area 1155 above with reference to Figure 12 explanation.

Figure 47 is according to six stages: 4610,4790,4630,4635,4640 and 4645, and the switching camera operation of graphic extension UI 1105.The first stage 4610 of Figure 47 is similar to the first stage 4610 of Figure 46, except the layout of viewing area 1155 shows mute button 1285, and closing session button 1287 and switching camera button 1289, instead of outside single closing session button.The user that second stage 4790 is illustrated in local device 4600 select (for example, by utilizations point 4670 singly refer to dub) the optional UI project of switching camera 1289 UI 1105 afterwards.In this example, by selecting optional UI project 1289, user instruction equipment 4600, during video conference, is switched to the back side camera of equipment 4600 from the front camera of equipment 4600.The rear four-stage of Figure 47 and the rear four-stage of Figure 46 are similar, except the layout of viewing area 1155 is identical with the layout illustrating in the first stage 4610 above, then be not described further, to avoid by the fuzzy description of the invention of unnecessary details.

In certain embodiments, when from the image of the different cameras of local two camera movement equipment (remote-moving apparatus receives, local two camera movement equipment switches camera) time, remote-moving apparatus also switches camera animation, to be presented at from the demonstration of the image of a camera of two camera movement equipment with from the transformation between the demonstration of the image of another camera of local two camera movement equipment.Figure 48 is according to five operational phases 4810,4815,4820,4825 and 4830 of UI 4805, the example of one of this switching camera of graphic extension animation.Figure 48 represents the example switching camera animation on remote-moving apparatus 4800.Each operational phase is identical with the example animation of Figure 46, and except being that image to being presented in viewing area 4835 carries out animation, viewing area 4835 is on remote-moving apparatus 4800, to show the place from the image of local two camera movement equipment.Thereby, the image that is presented at the man in viewing area 4835 is become Rotate 180 ° on the vertical axis 4855 that is positioned at viewing area 4850 central authorities seemingly by flash demo, to be illustrated in the transformation between the demonstration of image of the man in viewing area 4835 and the demonstration of the image of automobile 4870.Realizing of the realization of the switching camera animation of some embodiment and animation described above is identical.

Example above illustrates the switching camera animation on the remote equipment with particular user interface layout.Other embodiment can realize this switching camera animation on the remote equipment with different user interface layout.For example, Figure 49 graphic extension has a this example of the remote equipment 4800 of different user interface layout 4805.Especially, the UI 4805 of Figure 49 has mute button 1285, the closing session button 1287 being included in viewing area 1155 and switches camera button 1289, and during video conference, described viewing area 1155 water are presented at a side of compound display 4850 for a long time.The layout of these three buttons has been described above with reference to Figure 48.Except different user interface layout, the double teacher 4810,4815,4820,4825 and 4830 of Figure 49 is identical with the double teacher of Figure 48 4810,4815,4820,4825 and 4830.

2. long-range switching camera

During Figure 50 is illustrated in video conference, switch the processing 5000 of two cameras of remote double camera apparatus.Processing 5000 is carried out by the video conference manager of the equipment that comprises at least one camera.In the following discussion, user is called as local device by the equipment of the long-range switching camera of its instruction, and the equipment that switches its two cameras is called as remote equipment.In addition, in the following discussion, remote equipment is considered to switch between its front camera (camera 1 in other words) and its back side camera (camera 2 in other words).

Below with reference to Figure 51,52,53 and 54, the processing 5000 of Figure 50 is described.The UI 5105 of Figure 51 graphic extension local device 5100, during video conference, user asks remote equipment to switch between its two cameras by local device 5100.Eight different operational phases 5110,5115,5120,5125,5130,5135,5140 and 5145 of Figure 51 graphic extension UI 5105.Figure 54 graphic extension receives the UI 5405 of the remote equipment 5400 that switches camera request from local device 5100.Six different operational phases 5410,5415,5420,5425,5430 and 5435 of Figure 54 graphic extension UI 5405.

As shown in Figure 50, by start (5005) video conference between local device and remote equipment, start to process 5000.Process 5000 (5010) receptions subsequently for example, from the image of a camera (, from the front camera of each equipment) of each equipment, according to the composograph of these image generating video meetings.5010, process 5000 also sending to remote equipment from the video image of local device.

Subsequently, process 5000 and determine whether (5015) receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer the user's of Local or Remote equipment request, finish video conference.In the time that processing 5000 receives the request that finishes video conference, process 5000 and finish.

In the time that processing 5000 does not receive the request that finishes video conference, processing 5000 determines subsequently and (5020) carries out the user (, the user of local device) of processing 5000 equipment thereon whether this device request remote equipment of instruction switches between the camera for video conference at it.When it is not initiated long-range switching camera by instruction when processing 5000 definite (5020), process 5000 and return to operation 5010.When it has been initiated long-range switching camera by instruction when processing 5000 definite (5020), process 5000 and enter the operation 5025 further illustrating below.

Front four-stage 5110,5115,5120 and 5125 graphic extensions of the UI 5105 of Figure 51 receive the example of the request of the camera of user's switching remote equipment.The first and second stages 5100 and 5115 are identical with the first and second stages 4610 and 4615 of Figure 46.Phase III 5120 is identical with the phase III 4620, except the phase III 5120 not only comprises that request local device 5100 switches the optional UI project 5175 of camera, and comprises that request remote equipment 5100 switches outside the optional UI project 5180 of camera.The user of fourth stage 5125 graphic extension local devices 5100 selects to ask remote equipment to switch the UI project 5180 (for example, by optional UI project 5180 singly refer to dub 5170) of camera.By highlighting optional UI project 5180, point out described selection.Figure 51 represents to carry out an example of this operation, but other embodiment can differently ask remote equipment to switch the operation of camera.

The example illustrating above with reference to Figure 51, by long-range switching camera user interface, calls long-range switching camera operation.Other embodiment differently calls long-range switching camera operation.For example, some embodiment, by making to switch the UI during the optional UI project of camera is for good and all presented at video conference, for example, on the UI 5105 of Figure 52, call switching camera operation.In Figure 52, long-range switching camera button 5288 is displayed in viewing area 1155 together with mute button 5282, closing session button 5284 and the local camera button 5286 of switching.

Figure 52 is according to six different stages 5110,5290,5130,5135,5140 and 5145, the long-range switching camera operation of the UI 5105 of graphic extension equipment 5100.The first stage 5110 of Figure 52 is similar to the first stage 5110 of Figure 51, except the layout of viewing area 1155 shows mute button 5282, local switching camera button 5286, long-range switching camera button 5288 and closing session button 5284.The user that second stage 5290 is illustrated in local device 5100 selects the optional UI project 5288 of (for example,, by singly referring to dub 5170) long-range switching camera UI 1105 afterwards.The rear four-stage of Figure 52 is similar to the rear four-stage of Figure 51, except the layout of viewing area 1155 is identical with the layout of describing in the first stage 5110 above, then be not described further, to avoid by the fuzzy description of the invention of unnecessary details.

Some embodiment provide with Figure 52 in the similar layout of layout of graphic extension, show in 5165 except the optional UI project of long-range switching camera is presented at PIP, instead of outside being presented in viewing area 1155.The layout 5105 that Figure 53 graphic extension is such.Specifically, Figure 53 represents that the PIP with the optional UI project 5180 of long-range switching camera shows, and only has mute button 5282, local viewing area 1155 of switching camera button 5286 and closing session button 5284.

As mentioned above, in the time that user asks long-range switching camera, process 5000 and enter operation 5025.In operation 5025, process 5000 the request of switching camera is sent to remote equipment.In certain embodiments, this request is by as mentioned above, is sent together with video channel with voice-grade channel by the multiplexed video conference control channel of VTP manager.

After the request of switching camera is received, processes 5000 and determine whether (5030) remote equipment has responded the request of switching camera.In certain embodiments, remote equipment, automatically by video conference control channel, sends acceptance response (, send and confirm) to local device.But, in other embodiments, the user of remote equipment must accept this request by the user interface of remote equipment.

The first two stages 5410 and 5415 graphic extension long-distance user of the UI 5405 of Figure 54 accepts the example of the request of the camera that switches remote equipment 5400.First stage 5410 shows that (1) is for showing the viewing area 5440 of described request being notified to long-distance user's text, (2) for accept the camera that switches remote equipment request optional UI project 5465 (for example, " permission (Allow) " button 5465), (3) for example, for refusing the optional UI project 5470 (, " refusal (Reject) " button 5470) of request of the camera that switches remote equipment.The user that second stage 5415 is illustrated in remote equipment subsequently (for example selects, by singly referring to dub 5480) for accepting the UI 5405 after the UI project 5465 of the request of switching camera, by highlighting optional UI project 5465, the described selection of indicating user.

Determine (5030) it while not receiving the response from remote equipment, is processed 5000 and determine whether (5035) receive the request that finishes video conference when processing 5000.If so, process so 5000 end.Otherwise, process and receive (5040) image from the currently used camera of remote equipment and local device, the composograph that produces video conference according to these images, sends remote equipment to the video image of local device, and returns to subsequently operation 5030.

Determine (5030) it while having received the response from remote equipment, is processed 5000 and determine the request whether (5045) remote equipments accept to switch camera when processing 5000.If not, process so 5000 and return to operation 5010, continue to receive the image from the described camera of another equipment.Otherwise, process 5100 and receive (5050) image from another camera of remote equipment, on local device, carry out subsequently (5055) and switch camera animation, to show the transformation between the video of remote camera of previous utilization and the video of the remote camera of current utilization (, at the reception image that operates 5050).After operation 5055, process and return to operation 5010 described above.

About an example of this long-range switching camera animation on rear four operational phases 5130,5135,5140 and 5145 graphic extension local device 5100 of UI 5105 graphic extensions in Figure 51.The example animated type of this example animation and graphic extension in the stage 4815,4820,4825 and 4830 of Figure 48 seemingly, except Figure 51 is in viewing area 5150, the video of demonstration by the tree of the back side camera of remote equipment, outside the animation of replacement by the Ms's of the positive camera of remote equipment video.The animation that the rear four-stage graphic extension of Figure 52 and Figure 53 is identical with the animation in Figure 51, except the viewing area 1155 of Figure 52 and 53 comprises the optional UI project different from viewing area 1155 in Figure 51.

In certain embodiments, in the time that remote equipment switches camera, the UI of remote equipment also switches camera animation, to show two transformations between camera.In the time that remote equipment 5400 switches camera, be presented at the example of the switching camera animation on remote equipment 5400 about rear four operational phases 5420,5425,5430 and 5435 graphic extension of UI 5405 graphic extensions in Figure 54.The animated type of this animation and graphic extension in the stage 4630,4635,4640 and 4645 of Figure 46 seemingly, except the video of the animation in viewing area 5445 by the tree of the back side camera of remote equipment 5400, replace outside the video by the Ms of the positive camera of remote equipment 5400.

As mentioned above, Figure 46,47,48,49,51,52,53 and 54 is illustrated in the various examples of the switching camera animation of carrying out in user interface.In certain embodiments, switch the variation that camera animation causes the image processing operations of corresponding pair of camera movement equipment, such as the proportional zoom that can be carried out by video conference manager 1604 and image processing manager 1608, synthetic and perspective distortion.

C. exposure is adjusted

During the video conference between two camera movement equipment and another mobile device, different embodiment provides the different technologies of adjustment by the exposure of the image of the camera of any one mobile device.Some embodiment provide the technology of adjustment by the exposure of the image of the camera of another equipment to the user of two camera movement equipment, and other embodiment provides the technology of adjustment by the exposure of the image of the camera of two camera movement equipment to user.Some exemplary examples will be described in detail below.

During Figure 55 is illustrated in video conference, on two camera movement equipment of some embodiment, carry out the processing 5500 of long-range exposure adjustment operation.In the following discussion, the equipment that user adjusts its exposure by its instruction remote equipment is called as local device.In certain embodiments, processing 5500 is carried out by the video conference manager of local device.In addition, ask remote equipment to expose with reference to Figure 56,57 and 58 explanation processing 5500, Figure 56,57 and 58 users that illustrate local device and adjust the variety of way of operation.

As shown in Figure 55, by start (5505) video conference between local device and remote equipment, start to process 5500.Process 5500 and receive subsequently (5510) video from remote equipment, with the demonstration screen display at local device.Subsequently, process 5500 and determine whether (5515) receive the request that finishes video conference.As mentioned above, some embodiment can receive the request that finishes video conference from the user of Local or Remote equipment.In the time that processing 5500 receives the request that finishes video conference, process 5500 and finish.

But, in the time that processing 5500 does not receive the request that finishes video conference, process 5500 and determine subsequently whether (5520) receive the request of the exposure of the camera of adjusting remote equipment.In the time processing the request of 5500 definite exposures that do not receive the camera of adjusting remote equipment, process 5500 and return to operation 5510, receive from the other video of remote equipment shooting.Figure 56,57 and 58 graphic extensions provide three different examples that produce this request to user.In Figure 56,57 and 58, first stage 5610,5710 and 5810 all shows that the PIP of local device 5600,5700 and 5800 shows 5625,5750 and 5835, PIP shows that 5625,5750 and 5835 show two videos: by the video of the camera of local device, and by another video of the camera of remote equipment.In the first stage 5610,5710 and 5810, background shows that the man in 5635,5760 and 5845 is more black dull, indicates this man not by appropriate exposure.

Second stage 561 graphic extensions of Figure 56 by select remote equipment video (for example, by click background show 5635), the user of local device 5600 ask remote equipment expose adjustment a kind of mode.In this manner, to area-of-interest, the expectation of adjusting that exposes associates the instruction remote equipment that UI 5605 makes the selection of user to the area-of-interest being limited by square frame 5645 and user automatically, thereby the video conference manager contact remote equipment of instruction local device, adjusts operation to expose.The area-of-interest of definition is the calculating for exposing and adjusting by remote equipment.

The second stage 5615 that is similar to Figure 56, the second stage 5715 of Figure 57 represents the selection of the video of local user to remote equipment, except the optional UI project 5770 that this selection instruction UI 5705 showed as shown in the phase III 5720.The user of fourth stage 5725 graphic extension local devices selects optional UI project 5770, exposes as mentioned above and adjusts operation with instruction remote equipment.

The second stage 5815 of Figure 58 is similar to the second stage of Figure 57, but replace the selection instruction UI of the video of user to remote equipment to show single optional UI project, user's selection instruction UI 5805 shows the menu of optional UI project 5855,5860,5865 and 580, as shown in the phase III 5820.Optional UI project comprises " automatic focus (Auto Focus) " project 5855, " automatic exposure (Auto Exposure) " project 5860, " switching camera (Switch Camera) " project 5865 and " cancelling (Cancel) " project 5870.In certain embodiments, " switching camera " optional UI project 5865 is used to ask local switching camera operation, and in other embodiments, " switching camera " optional UI project 5865 is used to ask long-range switching camera operation.Fourth stage 5825 graphic extension users selections " automatic exposure,, project 5860, exposes as mentioned above and adjusts operation with instruction remote equipment.

In the time processing 5500 definite (5520) local user's instruction local device request exposure adjustment operation, process 5500 and send (5525) order by video conference control channel to remote equipment, to adjust by current shooting and to the exposure of the video of the camera of local device transmission video.After operation 5525, process 5500 and return to operation 5510 described above.

In certain embodiments, before remote equipment exposes adjustment operation, require the user of remote equipment that license is provided, and in other embodiments, when from local device receives request, remote equipment automatically expose and adjust operation.In addition, in certain embodiments, some video conference functions are realized by video conference manager 1604.In some such embodiment, video conference manager 1604 is adjusted the exposure setting of the transducer of the remote equipment camera using by instruction CIPU 1650, carries out exposure and adjusts operation.

Figure 56,57 and 58 final stage 5620,5730 and 5830 show the video of remote equipment more brightly, and this indicates this man by appropriate exposure.Although Figure 56,57 and 58 provides the example of the exposure adjustment request that receives the exposure of proofreading and correct remote equipment, but some embodiment provide request local device to adjust the mode of the exposure of the camera of local device to the user of local device.The mode that can adjust the exposure of its camera with the request remote equipment of graphic extension in Figure 56,57 and 58 produces such request similarly.

Figure 56-58 described above represent to adjust for exposing several user interfaces of operation.In certain embodiments, the change that operation can cause the image processing operations of two camera movement equipment is adjusted in exposure, such as calling the exposure adjustment processing 5900 further describing below.Exposure adjustment operates the change of the operation of the camera of two camera movement equipment that can also cause capture video, such as changing the exposure setting of camera.

1. exposure adjustment method

Figure 59 conceptually graphic extension by the image processing manager of some embodiment, the exposure adjustment processing 5900 of carrying out such as the image processing manager of graphic extension in Figure 16.In certain embodiments, processing 5900 is the parts that operate above with reference to the exposure adjustment of Figure 55,56,57 and 58 explanations.In some such embodiment, image processing manager 1608 is carried out and is processed 5900, and by send the exposure setting of instruction adjustment camera to video conference manager 1604, described video conference manager 1604 instruction CIPU 1650 adjust camera sensor 405a or 405b, as mentioned above.

In certain embodiments, process 5900 and carried out by the image processing layer 930 shown in Fig. 9, and in other embodiments, process 5900 and carried out by the statistics engine 465 shown in Fig. 4.The image of the camera of some embodiment to (Local or Remote) equipment by video conference processes 5900, and other embodiment for example, carries out and processes 5900 with the form of a part (, operation 2110) for the processing 2100 of graphic extension in Figure 21.Some embodiment expose and adjust operation, with expose by the camera of two camera movement equipment be not too bright, image that neither be too dark.In other words, carry out and process 5900, with the maximized as far as possible mode photographic images of the quantity according to making details.

By receiving (5905) image by the camera of two camera movement equipment, start to process 5900.In certain embodiments, in the time that the image receiving is the first two field picture by the camera of the equipment in video conference, the first two field picture is not processed to 5900 (,, before the first two field picture, not having any image of determining exposure value according to it).Process 5900 and read subsequently the pixel value in the regulation region in (5910) reception image.Different embodiment differently defines described region.Some such embodiment define difform region, and such as square, rectangle, triangle, circle etc., and other this embodiment is at the diverse location of image, such as center, directly over, under etc. the described region of definition.

Subsequently, the mean value of the pixel value in the regulation region of processing 5900 calculating (5915) images.Process 5900 and determine whether the calculating mean value of (5920) pixel value equals specific setting.Different embodiment defines different particular values.For example, some embodiment are defined as described particular value the median pixel value of the dynamic range of image.In certain embodiments, the scope of definition numerical value, instead of single value.In such embodiments, process 5900 calculating mean values of determining (5920) pixel values whether within the prescribed limit of numerical value.

In the time that the calculating mean value of pixel value is not equal to specific setting, process 5900 according to calculating mean value adjustment (5925) exposure value.In the time that the calculating mean value of pixel value equals specific setting, process 5900 and finish.In certain embodiments, exposure value represents the time quantum that camera sensor is exposed.In certain embodiments, the exposure value after adjustment is used to exposure by the next frame image by taking the camera that receives image.After adjusting exposure value according to the mean value calculating, process 5900 and finish.

In certain embodiments, repeat and process 5900, until the calculating mean value of pixel value equals specific setting (or in prescribed limit of numerical value).Some embodiment, during video conference, constantly process 5900, and other embodiment is during video conference, for example, process 5900 with official hour interval (, 5 seconds, 10 seconds, 30 seconds etc.).In addition,, during video conference, the processing 5900 of some embodiment was being processed before 5900, dynamically redefined specific pixel value.

Figure 60 is the example of the exposure adjustment operation of some embodiment of graphic extension conceptually.Example 6000,6010 and 6015 is all presented at left side the image 6020 of the camera by two camera movement equipment.Specifically, image 6020 shows the black dull personage back to the sun.The exposure of black dull personage's indicating image be not high enough to expose personage's face or health.The right side of each example 6000,6010 and 6015 is illustrated respectively in the image 6025,6030 and 6035 of taking after image 6020.In certain embodiments, the image on image 6020 and right side is the image by the video of the camera of two camera movement equipment.In other embodiments, the image on image 620 and right side is at the rest image of not taking in the same time by the camera of two camera movement equipment.

The operation that first example 6000 graphic extensions are adjusted without any exposure.Thereby it is identical with image 6020 that image 6025 seems.Owing to not carrying out any exposure adjustment, therefore with image 6020 in figure kind seemingly, the personage in image 6025 is still black dull.

In second example 6010, image 6020 is exposed and adjusts operation.In certain embodiments, exposure adjustment operates by processing 5900 and utilizes regulation region 6040 to carry out.Adjust operation according to exposure, the exposure of camera is adjusted, the exposure photographic images 6030 after camera utilization is adjusted.As shown in Figure 60, the personage in image 6030 is not as black dull in image 6025.But, in image 6030, personage's face and health are still unintelligible.

The 3rd example 6015 represents the exposure adjustment operation that image 6020 is carried out.Be similar to second example 6010, the exposure adjustment operation of the example 6015 of some embodiment utilizes regulation region 6045 to carry out by processing 5900.Adjust operation according to exposure, the exposure of camera is adjusted, the exposure photographic images 6035 after camera utilization is adjusted.As shown in Figure 60, the personage in image 6035 is by correct exposure, because personage's face and health are all visible.

In certain embodiments, the selection in regulation region can be undertaken by the user of two camera movement equipment.The feedback loop that equipment itself can be adjusted by the exposure of mentioning in CIPU 400 above, automatically adjusts it and adjusts the regulation region operating for exposing.Statistics engine 465 in Fig. 4 can be collected data, to determine whether exposure is suitable for the image of taking, and adjusts accordingly camera sensor (for example,, by being connected with the direct of sensor assembly 415).

D. Focussing

During Figure 61 is illustrated in video conference, adjust the processing 6100 of the focal length of two camera movement equipment.In the following discussion, the equipment that user adjusts the camera focal length of remote equipment by its instruction remote equipment is called as local device.In certain embodiments, the processing 6100 of Figure 61 is carried out by the video conference manager 1604 of local device.In addition, illustrate that below with reference to Figure 62 and 63 processing the user that 6100, Figure 62 and 63 are local device provides request remote equipment to carry out two kinds of way of example of Focussing operation.

As shown in Figure 61, by start (6105) video conference between local device and remote equipment, start to process 6100.Process 6100 and receive subsequently (6110) from video of remote equipment, so that in the demonstration screen display of local device.Subsequently, 6115, process 6100 and determine whether to receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer the user's of Local or Remote equipment request, finish video conference.In the time that processing 6100 receives the request that finishes video conference, process 6100 and finish.

Otherwise, process 6100 and determine (6120) whether it receives the request of focal length of the remote camera of adjusting remote equipment.In the time that processing 6100 determines that it does not receive the request of focal length of the remote camera of adjusting remote equipment, to process 6100 and return to operation 6110, reception is from the other video of remote equipment.Three kinds of different modes of Figure 62, the 63 this requests of generation that provide to user with the different embodiment of 64 graphic extension.In Figure 62,63 and 64, first stage 6210,6310 and 6472 all shows that the PIP of local device 6200,6300 and 6471 shows 6225,6335 and 6482, PIP shows that 6225,6335 and 6482 all show two videos, the video of being taken by local device, and another video of being taken by remote equipment.Viewing area 1155 and 1155 in Figure 62 and 63 shows closing session button.But, in Figure 64, the layout of viewing area 1155 is identical with the layout of the viewing area of Figure 12 described above 1155.In addition, can be chosen in the switching camera button 6488 shown in viewing area 1155, to call in certain embodiments local switching camera operation, or call in other embodiments long-range switching camera operation.As shown in the first stage 6210,6310 and 6472, the video that is presented at the remote equipment in background demonstration 6235,6345 and 6480 is fuzzy.

Second stage 6215 graphic extensions of Figure 62 for example, by selecting simply the video (, by the single-point 6240 on the video of remote equipment) of remote equipment, and the user of local device is to the method for remote equipment request Focussing.According to the method, user, the selection to the area-of-interest being limited by square frame 6245 and user's instruction remote equipment is to described area-of-interest executable operations (for example automatically for UI 6205, Focussing operation) expectation associate, thereby the video conference manager of instruction local device 6,200 1604 contact remote equipment, operate to carry out adjustment (such as, Focussing operation).The area-of-interest of definition is the calculating for Focussing by remote equipment.

The second stage 6315 of Figure 63 represents the selection of local user to long-distance video (for example, by user the video to remote equipment dub) similarly.But, be different from the example of graphic extension in Figure 62, this selection instruction UI in Figure 63 6305 shows as the menu of the optional UI project 6355,6360,6365 and 6370 (they can be realized as optional button) as shown in the phase III 6320.These optional UI projects comprise " automatic focus (Auto Focus) " project 6360, " automatic exposure (Auto Exposure) " project 6365, " switching camera (Switch Camera) " project 6370 and " cancelling (Cancel) " project 6355.In certain embodiments, " switching camera " optional UI project 6370 is used to ask local switching camera operation, and in other embodiments, " switching camera " optional UI project 6370 is used to ask long-range switching camera operation.Fourth stage 6325 subsequently graphic extension local user is selected automatic focus project 6360.

The second stage 6474 of Figure 64 represents the selection of local user to long-distance video (for example, by user the video to remote equipment dub) again similarly.But, be different from the example of graphic extension in Figure 63, this selection instruction UI 6478 in Figure 64 asks Focussing operation (, in second stage 6474).After finishing Focussing operation, UI 6478 shows the menu (, in the phase III 6476) of optional UI project 6484 and 6486, and optional UI project 6484 and 6486 can be realized as optional button.These optional UI projects comprise " automatic exposure " project 6486 and " cancellation " project 6484.

In the time processing 6100 definite (6120) local user's instruction local device request Focussing operation, process 6100 by video conference control channel, send (6140) order to remote equipment, catch at present and transmit the focal length of the camera of its video to adjust remote equipment.After 6140, process and return to operation 6110 described above.

In certain embodiments, before remote equipment carries out this operation, the user of remote equipment must provide permission, and in other embodiments, when receiving after the request of local device, remote equipment automatically performs this operation.In addition, in certain embodiments, the focal length setting of the camera using during video conference of Focussing operation adjustment remote equipment.In some such embodiment, as mentioned above, some video conference functions are realized by video conference module 1602.In these embodiments, video conference manager 1604 instruction CIPU 1650 adjust the transducer of the remote equipment camera using.

Figure 62,63 and 64 final stage 6220,6330 and 6476 represent the video of the appropriate remote equipment focusing on.Although Figure 62,63 and 64 provides the example of the Focussing request that receives the focal length of proofreading and correct remote equipment, but, some embodiment allow the user of local device to ask local device to adjust the focal length of the camera of local device.Can be similar to the method for adjusting the focal length of its camera at the request remote equipment shown in Figure 62,63 and 64, produce such request.

Three kinds of example user interface that Figure 62,63 and 64 graphic extensions allow user to carry out Focussing operation.In certain embodiments, Focussing operation causes the change of the operation of the camera of taking the two camera movement equipment that are presented at the video in UI, such as changing the focal length of camera.

As above described in Figure 56 and 62, the area-of-interest of regulation is respectively used to the exposure adjustment of video and the calculating of Focussing by remote-moving apparatus.But, in some other embodiment, user can be used to instruction remote equipment to the selection of area-of-interest and carry out one or more operations.For example, in certain embodiments, can, according to the area-of-interest of definition, expose and adjust and Focussing, thereby instruction remote equipment carry out two operations.

E. frame rate control

During video conference, some embodiment may wish to adjust or keep transmitting the speed (, frame rate) by the video image of the camera of two camera movement equipment to another equipment in video conference.For example, suppose that bandwidth fixes, some such embodiment reduce the frame rate of video, and to improve the image quality of video image, and other such embodiment increases the frame rate of video, with smoothed video (, reducing shake).

Different embodiment provides during video conference, controls the different technologies of the frame rate of video image.The example illustrating is above adjusted the VBI of the sensor assembly 415 of camera, to control the speed of processing by the image of camera.As another example, some embodiment of the management level 935 of the video conference module 925 shown in Fig. 9 carry out control frame speed by abandoning image.Similarly, some embodiment of image processing layer 930 carry out control frame speed by abandoning image.Some embodiment provide other technology in addition to carry out control frame speed, such as the frame abandoning in general transmission buffer 3120.

V. two cameras

A. assembled view

1. picture-in-picture: the demonstration of two remote camera

Some embodiment allow any one in arranging according to several demonstrations of two camera movement equipment, show from during video conference the video of taking from this mobile device and the video of taking from another pair of camera movement equipment.The difference of the video that Figure 65 graphic extension is taken from one or more pairs of camera movement equipment shows the example of arranging.In Figure 65, the another one user of the user of two camera movement equipment 6500 (device A) and another pair of camera movement equipment 6505 (equipment B) is carrying out video conference mutually.

Four kinds of examples that Figure 65 arranges the demonstration of device A are illustrated in left side.Four kinds of device A show that arranging is the first view 6510, the second view 6515, three-view diagram 6520 and the 4th view 6525.In addition, four kinds of examples that Figure 65 also arranges the demonstration of equipment B are illustrated in right side.Four kinds of equipment B show that arranging is the first view 6565, the second view 6570, three-view diagram 6575 and the 4th view 6580.In this example, device A only shows two videos from the camera of device A, and equipment B shows from two videos of the camera of device A, and shows from one or two video of the camera of equipment B.

In the first view 6510, the UI 6585 of device A provides compound display 6512.Compound display 6512 comprises two viewing areas: demonstration is from the viewing area 6530 of the video of the back side camera of device A, and demonstration is from the viewing area 6535 of the video of the positive camera of device A.In this example, viewing area 6530 is positioned at the first half of compound display 6512, and viewing area 6535 is positioned at the Lower Half of compound display 6512.In the first view 6510, these two viewing area sizes are identical.The viewing area 6530 on top is just showing the video on mountain range, and the supposition of described mountain range is just by the mountain range of the back side camera of device A.Viewing area 6535 is just showing tree and the man that attaches the names of pre-determined candidates, and described tree and the man that attaches the names of pre-determined candidates supposition is just by tree and the man that attaches the names of pre-determined candidates of the positive camera of device A

UI 6585 in the second view 6515 provides compound display 6517, compound display 6517 comprises two viewing areas identical with the first view 6510, except viewing area 6535 (showing the video from the positive camera of device A) is being positioned at the first half of compound display 6517, viewing area 6530 (showing from the video of the back side camera of device A) is positioned at outside the Lower Half of compound display 6517.

In three-view diagram 6520, UI 6585 provides PIP to show 6595.PIP shows and 6595 comprises two viewing areas: as a setting viewing area, show from the viewing area 6535 of the video of the positive camera of device A, and as prospect insert viewing area, show from the viewing area 6530 of the video of the back side camera of device A.In this view, background viewing area 6535 occupies most PIP and shows 6595, and it is less and overlap on a part of background viewing area 6535 to insert viewing area 6530.

UI 6585 in the 4th view 6525 also presents and comprises that the PIP of the viewing area as shown in three-view diagram 6,520 6530 and 6535 shows 6598.Be different from PIP and show 6595, PIP shows that 6598 comprise the viewing area 6530 (taking from the back side of device A camera) of main demonstration as a setting, and inserts as prospect the viewing area 6535 (taking from the front of device A camera) showing.In addition, PIP shows that 6598 present (, PIP shows that 598 width is greater than height) by transverse views.

Four kinds of possible synthetic views of the UI of example graphic extension device A above-wherein show longitudinal two the stacked views in two viewing areas 6530 and 6535 of the video of two cameras of the first equipment, and two PIP views.Concerning the UI of device A, other view is also possible.For example, two viewing areas can level or diagonal angle stacked, or can use different PIP views.

The UI that represents equipment B about each view of equipment B graphic extension can adopt different views.These views comprise two cameras from device A, and one or the video of multi-section camera of equipment B.In the first view 6565 of equipment B, the UI6590 of equipment B provides PIP to show 6568.PIP shows that 6568 comprise the compound display district 6569 identical with being presented at compound display 6512 in device A, and shows the insertion viewing area 6550 of the video that one of the camera by equipment B (for example, front camera) take.Compound display district 6569 comprises the viewing area 6531 of demonstration from the video of the back side camera of device A, and demonstration is from the viewing area 6536 of the video of the positive camera of equipment B.The compound display 6569 of the video of display device A occupies most PIP and shows 6568, and it is less and overlap on compound display 6569 to insert viewing area 6550.Viewing area 6550 shows smiling face's video, and described smiling face supposition is just by the smiling face of positive its video of camera of equipment B.

The UI 6590 of the equipment B in the second view 6570 provides PIP to show 6572.PIP shows that 6572 comprise viewing area 6550 (showing from the video of the positive camera of equipment B), and has the compound display 6573 of demonstration from the viewing area 6531 and 6536 of the video of the camera of device A.Compound display 6517 in the second view 6515 of compound display 6573 and device A is identical, and occupies most PIP and show 6572.Show in 6568 equally with PIP in the first view 6565, viewing area 6550 is less and overlap on compound display 6573.Specifically, in these two views, viewing area 6550 overlaps demonstration a part for the viewing area 6531 of the video of the back side camera of device A.

In three-view diagram 6575, UI 6590 provide with the three-view diagram 6520 of device A in PIP show 6595 similar PIP demonstrations 6577.PIP shows that 6577 also comprise other viewing area 6550, and as the second insertion viewing area, second inserts viewing area 6550 overlaps on background viewing area 6536.Two are inserted the bottom of viewing area 6531 and 6550 Horizontal Tiles in the main viewing area 6536 of background.

UI 6590 in the 4th view 6580 provides compound display 6582.Compound display 6582 comprises three demonstration: PIP demonstrations 6583, viewing area 6550 and viewing areas 6540 (for example,, for showing the video by the back side camera of equipment B).PIP shows that 6583 show that with the PIP in the 4th view 6525 of device A 6598 is identical, and occupies most compound display district 6582.Viewing area 6540 and 6550 is less, and Horizontal Tile is under PIP viewing area 6583.

Although Figure 65 illustrates four kinds of possibility views of equipment B, but other many views are also possible.The background compound display of the video of device A can Horizontal Tile, instead of vertical tile, inserting viewing area can overlap on the camera viewing area, front of device A, instead of overlap on camera viewing area, the back side, larger viewing area can display device B the video of camera, instead of the video of the camera of device A, insert viewing area and can be positioned at different positions, etc.

The every group of arrow 6560 that comes from each view of device A shows not exist to being illustrated in the demonstration in device A and being illustrated in the requirement of the relevance between the demonstration in equipment B.For example, for example, even if the list view district video that device A is being pressed view 6510 (, the arrangement of selecting according to the user of device A), equipment B also can be by four of graphic extension kind of arrangement, or press unshowned multiple other in Figure 65 and arrange one of any display video (arrangement of for example, selecting according to the user of equipment B).In other words, the demonstration arrangement of device A and the demonstration of equipment B are arranged irrelevant.Some embodiment do not send another equipment viewing area to from an equipment, and just transmit video (for example, with the form of encoding), and described video is presented in its corresponding viewing area by equipment.

2. special PIP

Some embodiment allow the user of two camera movement equipment during video conference, in PIP shows, the prospect of video are superimposed upon on another video.In certain embodiments, the prospect of video is mixed in another video in such a manner, makes them show as the demonstration by the single video of single camera.Figure 66 is illustrated in PIP demonstration, the example of this stack of the prospect of insertion video on background video.

Figure 66 is according to seven operational phases 6620,6625,6630,6635,6640,6660 and 6665 of UI 6670, and this video superimpose of graphic extension operates.During first stage 6620 is illustrated in the video conference with remote equipment, there is the UI 6670 that PIP shows two camera movement equipment 6600 of 6682.As shown in the first stage 6620, PIP shows that 6682 comprise that two videos show: the main demonstration 6610 of background and prospect are inserted and shown 6605.The main demonstration 6610 of background occupies most UI 6670, and prospect is inserted and shown that 6605 is less and overlap in the main demonstration 6610 of background.

In this example, background viewing area 6610 shows the video on mountain range, and the supposition of described mountain range is the mountain range that one of camera by remote equipment is taken.Prospect is inserted the video that viewing area 6605 shows the personage that attaches the names of pre-determined candidates, and described personage's supposition is the personage that one of camera by local device is taken its video.PIP show 6682 be below labeled as " End Conference (closing session) " optional UI project 6685 (for example, button 6685), described optional UI project 6685 (for example allows this project of user by selecting, by clicking or double-click this button), finish video conference.

Calling of the second stage 6625 optional menus 6675 of graphic extension.In certain embodiments, for example, by selecting (touching) PIP viewing area 6682, can call the menu of optional UI project 6675.Replace this call operation, or be combined with this call operation, some embodiment also allow user to pass through other operation, such as the touch screen by different operates or utilizes one or more other physics of equipment to input, call the menu of optional UI project 6675.

Phase III 6630 shows UI 6670, and UI 6670 has a group of calling for selecting the optional UI project of video superimpose operation.In this example, in PIP demonstration 6682, show the pop-up menu 6675 with several optional UI projects.The menu 6675 of optional UI project comprises " Flip PIP (upset PIP) ", and optional UI project 6640 (for example, button 6640), " Specialized PIP (special PIP) " optional UI project 6645 (for example, button 6645), " Cancel (cancellation) " optional UI project 6690 (for example, button 6690).In this example, select " Flip PIP " button 6640 can cause UI 6670 to exchange background and show that 6610 show 6605 (will discuss in detail in the next section) with inserting, select " Specialized PIP " button 6645 can cause UI 6670 to start the operation of video superimpose, and select " Cancel " button 6690 to show 6682 and to remove pop-up menu 6675 from PIP.Other embodiment comprises different or more project at PIP pop-up menu 6675.

Fourth stage 6635 is illustrated in user and has selected " Specialized PIP " button 6645 (for example dubbing button 6645 by the finger 6695 with him) UI 6670 afterwards.By showing on 6670 and highlight button 6645 at UI, indicate this selection.Some embodiment use different indicated number (for example, highlighting the text in frame or the selected item of selected item).

Five-stage 6640 is illustrated in and starts video superimpose operation UI 6670 afterwards.In this stage, UI 6670 allows user to select him to want from which video to extract using as prospect, and which video he wants to use as the background in overlay video.UI 6670 shows the pop-up menu with several optional UI projects 6680 on 6682 by being presented at PIP, and variety of option is provided.The pop-up menu 6680 of optional UI project comprises " Select Inset (select to insert and show) ", and optional UI project 6655 (for example, button 6655), " Select Main (selecting main demonstration) " optional UI project 6650 (for example, button 6650), " Cancel " optional UI project 6692 (for example, button 6692).

Select " Select Inset " button 6655 can cause UI 6670 that the prospect (man who, attaches the names of pre-determined candidates) of the insertion video 6605 of the camera from local device is superimposed upon on the main video 6610 of background from the camera of remote equipment.On the other hand, select " Select Main " button 6650 can cause UI 6670 that the prospect (, mountain range) of the main video 6610 of background of the camera from remote equipment is superimposed upon from the insertion video 6605 on the camera of local device.In certain embodiments, this causes the switching of two video push (feed), make will occupy most UI 6670 at the video inserting in viewing area 6605 at present, the video in main viewing area 6610 will be applied on ahorse main video at present.Select " Cancel " button 6692 can abandon video superimpose operation, and remove pop-up menu 6680 from PIP viewing area 6682.

The 6th stage 6600 was illustrated in user and selects " Select Inset " button 6655 (for example, dubbing button 6655 by the finger 6695 with him) UI 6670 afterwards.By showing on 6670 and highlight button 6655 at UI, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the text in frame or the selected item of selected item).

The 7th stage 6665 was illustrated in the UI 6670 after video superimpose operation.As shown in UI 6670, from viewing area 6605, extract the prospect (man who, attaches the names of pre-determined candidates) of inserting viewing area 6605.From screen, eliminate in addition to insert and show 6605 window frame and background (, other except described prospect all).Finally, in such a manner prospect (, the man who attaches the names of pre-determined candidates) is mixed in background video 6610, makes it show as single video.Can use various technology to remove the background of inserting video.Some embodiment identify the pixel that do not move with respect to other pixel, find invariable pattern or color, use the baseline image of comparing from the image that comprises prospect and deduct difference or use different technology.

Although the example graphic extension of Figure 66 is in the time being superimposed on background viewing area 6610, the prospect of inserting viewing area 6605 is retained in the same position in UI, but this example that just how stack can realize.Some embodiment move to ad-hoc location in UI 6670 (for example, one of center, corner, etc.) prospect video.Be similar in the feature shown in IV.A.1 and IV.A.3 joint, some embodiment allow the user of local device in UI, to drag back and forth the prospect video of stack, or change the size of the prospect video of stack.

Can determine which or which part of video image is " prospect " of video superimpose operation as above by different technology.A this method of some embodiment determines which or which part (if any) of video image is dynamic.Dynamic part is considered to " prospect ", because normally static (, without the motion) of the background of video image.In such embodiments, at specific a period of time inner analysis video image.If in described a period of time, the difference between the value of specific pixel is not more than the threshold value (for example, 5%, 10%, 15%) of regulation, thinks that so described specific pixel is static pixels.After having analyzed the each pixel in video image, the dynamic pixel of video image (, not being static pixel) is considered to " prospect " of video image.

Figure 67 graphic extension can be by an example of this technology of video conference manager 1604 or image processing manager 1608 prospect that carry out, that determine video image.Specifically, Figure 67 graphic extension shows a series of six image 6705-6730 of the video of the personage that attaches the names of pre-determined candidates and tree.In this example, suppose that described personage is not completely still on one's feet, may speak.As mentioned above, analyze the each pixel in video image, to determine that pixel is dynamic or static.For example, determine whether the difference between the value of the pixel 6735 in image 6705-6730 is greater than the threshold value of regulation.Here due to the pixel 6735 part ground of representative except personage, therefore think that pixel 6735 is static.After all pixels of having analyzed in image 6705-6730, determine that the personage in image is dynamic, and the remainder of image is static.Thereby described personage is " prospect " that the operation by illustrating above with reference to Figure 66 is extracted.

3. the video in the demonstration of exchange picture-in-picture

Some embodiment allow the user of two camera movement equipment during video conference, two viewing areas (that is, in PIP shows, insert viewing area and become background viewing area, background viewing area becomes inserts viewing area) during exchange PIP shows.During Figure 68 is illustrated in video conference, the insertion viewing area 6605 in exchange PIP demonstration 6682 and the example of background viewing area 6610.

Figure 68 is according to eight operational phases of the UI 6670 of the equipment 6800 in Figure 66, and graphic extension exchange PIP operates.Front three phases in Figure 68 is identical with the front three phases in Figure 66.In these stages, select by the touch-screen that utilizes local device, user appears in UI 6670 menu 6675.

The fourth stage 6840 of Figure 68 is illustrated in user and selects " Flip PIP " button 6640 (for example, dubbing button 6640 by the finger 6695 with him) UI 6670 afterwards.By showing on 6670 and highlight button 6640 at UI, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the text in frame or the selected item of selected item).

Five-stage 6845 is illustrated in and starts to exchange PIP operation UI 6670 afterwards.Some embodiment are by rotary movement, and flash demo inserts demonstration 6605 and background shows 6610 exchange.An example of this flash demo of Figure 68 graphic extension.In this example, show that by PIP 6682 (before carrying out swap operation) are in its one side, new PIP shows the upset of 6684 (after carrying out swap operation) the observation pane on its another side, and this animation can be described.Observe pane around vertical axis 6686 Rotate 180s ° that are positioned at PIP and show 6682 center.At five-stage 6845, observe pane and start to rotate around vertical axis 6686.

In the 6th stage 6850, observe pane and be expressed as rotating about 90 °.This is to indicate with the fine rule 6688 that is presented at screen center (, observing the edge of pane).The graphic extension of the 7th stages 6855 is observed the rotation of pane close to completing.New PIP shows that 6684 another sides that start from observing pane occur, and horizontal extension is to the screen that is full of equipment.PIP demonstration 6684 is included in carries out swap operation two viewing areas 6605 and 6610 afterwards.Present the man's that attaches the names of pre-determined candidates the viewing area 6605 of video (from the camera of local device) now in PIP shows 6684 background, present the demonstration 6610 of video (from the camera of remote equipment) on mountain range now in PIP shows 6684 prospect, described prospect overlaps and shows on 6605.The 8th stage 6860 represented that exchange showed completing of operation.

Those of ordinary skill will appreciate that the animation shown in Figure 68 is PIP insertion/background swap operation multiple one in may animation.For example, different embodiment can observe along horizontal rotational shaft that pane, moment exchange these two viewing areas, a viewing area of expansion dwindles another viewing area simultaneously, etc.It is a kind of always for the animation of swap operation that some embodiment provide, and other embodiment allows user to select from several animations, or use different animation (for example,, by random selection).In addition, swap operation can cause the variation of the image processing operations of two camera movement equipment, such as causing video conference manager 1604 to respond user's input, changes the proportional zoom of video and synthesizes.

4. lock onto corner

Some embodiments of the present invention allow the user of two camera movement equipment to form one or more viewing areas of compound display by moving around, and revise compound display.An example of this movement has been described above in IV.A.1 joint.In the time that PIP demonstration comprises more than one insertion viewing area, it is also possible inserting this movement showing.

Figure 69 is illustrated in this example of carrying out during video conference.In Figure 69, the example of graphic extension is similar with the example of graphic extension in Fig. 3, except moving around, Figure 69 graphic extension comprises that two are inserted viewing area 6905 and 6910, instead of outside the insertion viewing area 6910 of the only PIP demonstration 6965 of a this insertion viewing area.

In Figure 69, with the long-distance user's of another equipment video conference during, the UI 6960 of mobile device 6900 presents PIP and shows 6965.PIP in Figure 69 shows that 6965 comprise that three videos show: the main demonstration 6915 of background and two prospects are inserted and shown 6905 and 6910.In this example, the main demonstration 6915 of background presents the personage's who plays the guitar while singing video, and described video is assumed to be it is the video by the back side camera of remote equipment.Prospect is inserted and is shown that 6905 present the personage's who holds racket video, and in this example, this video is assumed to be it is the video by the back side camera of local device.Another prospect is inserted and is shown that 6910 present the personage's who wears a hat video, and in this example, this personage's supposition is the personage by positive its video of camera of local device.PIP shows that 6965 is below the optional UI project 6970 (for example, button 6970) that is labeled as " End Conference ", optional UI project 6970 allow user by selecting it and finish video conference.

PIP shows that 6965 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace have from the video of remote equipment show 6915 compared with overall background, larger background shows that 6915 can be the video from local device, less prospect is inserted and is shown that 6905 and 6910 can be the video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear to insert to show 6905 and 6910 in a side, and background shows that 6915 at opposite side, or in the UI 6910 that is arranged side by side of all these three demonstrations.In other embodiments, PIP shows that 6965 can comprise larger background demonstration 6915 and/or less prospect insertion demonstration.In certain embodiments, the mode of PIP demonstration 6965 or acquiescence display mode can be specified by user.

Figure 69 is by reference to five different operational phases 6920,6925,6930,6935 and 6940, and in the UI 6960 of graphic extension equipment 6900 two insert the movement of one of viewing areas.First stage 6920 is illustrated in the UI 6960 during the video conference between the local user of equipment 6900 and the long-distance user of remote equipment.

Second stage 6925 graphic extensions are inserted viewing area 6910 by selection, and user starts to lock onto corner operation.In this example, be placed on and insert in viewing area 6910 Anywhere by finger 6950, make a choice.As shown in the figure, show this selection with the thick frame 6962 that inserts viewing area 6910.Different embodiment can point out such selection by different modes, such as inserting viewing area 6910 by highlighting, swings by making to insert viewing area 6910, etc.

Phase III 6930 is illustrated in user and starts PIP to show that 6965 insertion viewing area 6910 shows that from PIP a region 6965 moves to PIP and shows the UI 6960 after another region in 6965.In this example, insert viewing area 6910 and started to move to the upper right corner of PIP demonstration 6965 from the lower right corner of PIP demonstration 6965, as shown in arrow 6955.After selecting to insert viewing area 6910, user drags its finger 6950, the mobile viewing area 6910 of inserting by show 6965 the upper right corner towards PIP.Some embodiment provide to move around in PIP shows 6965 and insert other technology of viewing area 6910.

Fourth stage 6935 is illustrated in user and removes the UI 6960 its finger 6950 state afterwards from the screen of equipment 6900.In this state, inserting viewing area 6910 still moves towards the upper right corner of pointing the mobile PIP demonstration 6965 of identifying according to the user in the phase III.In other words, start and insert viewing area 6910 after the movement in the upper right corner of PIP demonstration 6965 at finger 6950, even after removing finger 6950, UI 6960 still keeps this movement.In order to keep described movement, the UI 6960 of some embodiment required before user removes its finger, and user's drag operation is greater than specific threshold quantity (for example, be greater than specific distance, or be longer than the specific time); Otherwise these embodiment can, after mobile a little insertion viewing area, make this insertion viewing area remain on its initial position, the lower right corner, or mobile this insertion viewing area.

But, even although user is before insertion viewing area arrives its reposition, stopped his drag operation, some embodiment still allow to insert viewing area and continue mobile, but other embodiment requires user to keep drag operation, arrives its reposition until insert viewing area.Some embodiment provide mobile other technology of inserting viewing area.For example, some embodiment require user viewing area 6910 actual start mobile before, appointment will be directed to viewing area 6910 where, etc.Some embodiment also, by only with different angles inclination mobile device, make viewing area can slide and lock onto corner.

Five-stage 6940 is illustrated in the UI 6960 inserting after the reposition that viewing area 6910 arrives its upper right corner that is positioned at PIP viewing area 6965.In five-stage, the elimination of thick frame 6962 instruction locks onto corner operation and completes.

For make the above-mentioned the 3rd, the 4th and five-stage 6930,6935 and 6940 in the movement of graphic extension easier, once the UI 6960 of some embodiment adopts user to make to insert viewing area 6910 and moves towards certain corner of PIP demonstration 6965, just allows to insert the locking discipline of viewing area 6910 quick lock ins to this corner.For example, when user drags and inserts viewing area 6910 while exceeding threshold quantity towards particular corner, the direction of motion of viewing area 6910 is inserted in UI 6960 identifications of some embodiment, determine that described motion has exceeded threshold quantity, thereby automated movement inserts viewing area 6910 subsequently, and further input without user the next grid point inserting in the UI 6960 that can be locked into of viewing area 6910.In certain embodiments, the unique grid point providing for locking insertion viewing area 6910 is the grid point that is positioned at four corners of PIP demonstration 6965.Other embodiment provides other grid point of (for example, PIP shows in 6965) in the insertion UI 6960 that viewing area 6910 can be locked into.

Other embodiment can not adopt grid point, makes to insert viewing area 6910 and can be placed in the arbitrfary point in PIP demonstration.Again other embodiment provide allow that user opens or closes UI lock onto grid point feature.In addition, except the video of taking from equipment, different embodiment can allow user to various projects, such as the execution such as icon lock onto corner operation.As mentioned above, the movement of one or more viewing areas of compound display can cause the variation of the image processing operations of two camera movement equipment, such as causing video conference manager 1604 to respond user's input, again synthesizes the viewing area in compound display.

5. push and locking

The user of the two camera movement equipment of example graphic extension permission in Figure 69 inserts one of viewing areas two, and the corner showing from PIP moves to not to be inserted into and shows that the corner that locks onto in another corner occupying operates.Some embodiment can realize and insert and show the position that moves to the second insertion demonstration first, also insert and show the propelling movement feature that is pushed to reposition second.Figure 70 is illustrated in a this example of carrying out during video conference.

Figure 70 is by reference to six different phases 7020,7025,7030,7035,7040 and 7045 of UI 6960, and the movement that shows another corner showing to the PIP not occupied by another insertion demonstration from a corner of PIP demonstration is inserted in graphic extension.First stage 7020 is illustrated in the UI 6960 during the video conference between the local user of this equipment and the long-distance user of remote equipment.UI 6960 in Figure 70 represents and after starting video conference, the PIP identical at the PIP shown in the first stage of Figure 69 shows 6965.In this example, the video of being taken by local user's equipment is displayed on and inserts in viewing area 6905 and 6910, and the video of being taken by long-distance user's equipment is displayed in background viewing area 6915.

Second stage 7025 graphic extension user by selecting are inserted viewing area 6905, start to lock onto corner operation.In this example, be placed on any place of inserting in viewing area 6905 by finger 7055, select.As shown in the figure, show this selection with the thick frame 7065 that inserts viewing area 6905.Different embodiment indicates such selection by different modes, such as by highlighting district 6905, and by viewing area 6905 is swung, etc.

Phase IIIs 7030 graphic extension is as shown in arrow 7050, start inserting the lower right corner of shifting to PIP demonstration 6965 from the lower left corner of PIP demonstration 6965 in viewing area 6905 (by during phase III user, selecting to insert after viewing area 6905, show that towards PIP 6965 the lower right corner drags his finger 7055) UI 6960 afterwards.Some embodiment provide to move around in PIP shows 6965 and insert other technology of viewing area 6905.

Fourth stage 7035 is illustrated in inserts viewing area 6905 contact insertion viewing areas 6910 UI 6960 afterwards.In the time of contact, insert viewing area 6910 and move towards the nearest corner of the next one.In this example, inserting viewing area 6910 starts along direction (as shown in the arrow 7075) movement in the upper right corner of PIP demonstration 6965.With the thick frame 7070 that inserts viewing area 6910, show the activation of this push operation.Different embodiment can indicate this activation by different modes, such as by highlighting district 6910, etc.

Five-stage 7040 is illustrated in and inserts viewing area 6905 and be locked into the UI under the state being previously inserted into after the lower right corner that viewing area 6910 occupies.In this example, still move towards the upper right corner of PIP demonstration 6965 this insertion viewing area.In addition, no longer show thick frame 7065.As long as user's drag operation of phase III 7030 is greater than and makes to insert viewing area 6905 and lock onto the threshold value that right corner falls, just remove from its corner inserting viewing area 6910, and lock onto next minimum corner always.

Some embodiment comprise definite one group of rule that adopts which kind of mode to push the second insertion viewing area 6910.In Figure 70, graphic extension in the situation that, some embodiment manage to continue to keep inserting the rotation of viewing area.That is, move in the counterclockwise direction owing to inserting viewing area 6905, therefore viewing area 6910 is also moved counterclockwise.Some embodiment provide the classification of the possible position that pushed insertion viewing area 6910 can move to, and select first position not being occupied in this list.For example, in the time that the insertion viewing area that is positioned at the lower right corner is pushed by the insertion viewing area from the lower left corner, the upper right corner may be first position in this list.But, if the 3rd insertion viewing area has been positioned at the upper right corner, some embodiment can move to the next option (for example, the upper left corner, center or the lower left corner) in list so.Other embodiment can insert viewing area along with the second insertion viewing area pushes the 3rd together, makes equipment not need to determine that the 3rd inserts the reposition of viewing area.

The 6th stage 7045 was illustrated in the UI 6960 inserting after the reposition that viewing area 6910 arrives its upper right corner that is positioned at PIP viewing area 6965.In this stage, the elimination instruction of thick frame 7070 locks onto corner-push operation and completes.Be similar to reference to the corner that is pushed to of Figure 68 explanation and operate, the movement of one or more viewing areas of compound display can cause the variation of the image processing operations of two camera movement equipment, such as causing video conference manager 1604 to respond user's input, again synthesize the viewing area in compound display.

6. rotation

In the time that the user of the mobile device for video conference rotates this equipment in the session, some embodiment are rotated in the PIP presenting during video conference and show.Figure 71 graphic extension is in the time that equipment 7100 is rotated to horizontal level from upright position, and the UI of this equipment shows 7175 rotation.When the long limit of screen is while being vertical, equipment 7100 is held by perpendicular, and in the time of the level of the long limit of screen, equipment 7100 is held by horizontal.In Figure 71, in the example of graphic extension, UI shows that 7175 rotate to the horizontal transverse views of optimizing of holding for equipment 7100 from the perpendicular longitudinal view optimized of holding for equipment.This spinfunction makes to be held or when horizontal holding, user can both watch the UI 7175 showing with stand up position by perpendicular when mobile device 7100.In Figure 71, the example of graphic extension is similar to the example of graphic extension in Figure 34, except Figure 71 graphic extension rotation comprises that two are inserted viewing area and the not just PIP demonstration of an insertion viewing area.

In Figure 71, with the long-distance user's of another mobile device video conference during, the UI 7175 of mobile device presents PIP and shows 7180.PIP in Figure 71 shows that 7180 comprise that three videos show: the main demonstration 7115 of background and two prospects are inserted and shown 7110 and 7160.In this example, the main demonstration 7115 of background presents the video on mountain range, and described video supposition is by the front of remote equipment or the video of back side camera.Prospect is inserted and is shown that 7110 present the video of the smiling face in room, and described video supposition is by front or the back side camera of local device.Another prospect is inserted and is shown that 7160 present the guitar player's of singing video, and described guitar player supposition is just by the guitar player of another its video of camera of local device.PIP shows that 7180 is below " closing session " button 7155, and user can select this button for example, to finish video conference (, by singly refer to dub).This PIP shows a kind of mode of the synthetic view that just presents the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view, such as tiled view or different PIP demonstrations.

Figure 71 is according to six different operational phases 7120,7125,7130,7135,7140 and 7145, the rotation of graphic extension UI 7175.First stage 7120 is illustrated in the UI7175 during the video conference between the local user of this equipment and the long-distance user of remote equipment.

Second stage 7125 is illustrated in user and starts banking equipment 7100 UI 7175 afterwards.In this example, equipment 7100 has started equipment 7100 to hold from erecting to hold to tilt to by horizontal, as shown in arrow 7185.The outward appearance of UI 7175 does not change.In other cases, user may change into want equipment 7100 from by horizontal hold to tilt to by perpendicular hold, in these cases, UI shows that 7175 are switched to the view of vertical optimization from the view of level optimization.

Phase III 7130 is illustrated in equipment 7100 from being tilted to by the horizontal UI 7175 holding state afterwards by perpendicular holding.In this state, UI shows that 7175 outward appearance does not still change.In certain embodiments, be tilted and exceed threshold quantity at equipment 7100, and after keeping exceeding threshold quantity a period of time, trigger rotation operation.In Figure 71, in the example of graphic extension, suppose that threshold quantity and rotary speed can not make UI show 7175 rotations, until equipment has been placed on horizontal level short period interval afterwards.Different embodiment has different threshold quantities and the stand-by period for triggering rotation operation.For example, some embodiment can have the threshold value of so low triggering rotation operation, make the not orientation of tube apparatus 7100, all make UI 7175 seem as it always shows with stand up position.In other embodiments, the user of equipment 7100 can specify when can trigger rotation operation (for example, preferentially selecting to arrange by menu).In addition, some embodiment can, after equipment is tilted and exceedes threshold quantity, not postpone described rotation.In addition, different embodiment can allow to trigger rotation operation by different modes, such as by switching switch on mobile device, by sending voice commands, according to the selection of being undertaken by menu, etc.

Fourth stage 7135 is illustrated in and starts rotation operation UI 7175 afterwards.Some embodiment flash demo rotational display districts, to provide the feedback about rotation operation to user.The example of the such a kind of animation of Figure 71 graphic extension.Specifically, Figure 71 its fourth stage 7135 show viewing area 7110,7115 start together with 7160 rotation.Viewing area 7110,7115 and 7160 is around axle 7165 (, the z axle) rotation at the center through UI demonstration 7175.Viewing area 7110,7115 and 7160 is rotated identical quantity, but along the direction rotation contrary with the rotation (for example,, by the inclination of equipment 7100) of equipment 7100.In this example, because equipment 7100 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), therefore rotate operation and can make viewing area 7110,7115 and 7160 half-twist in the counterclockwise direction.In the time of 7110,7115 and 7160 rotation of viewing area, viewing area 7110,7115 and 7160 dwindles pari passu, shows 7175 to be applicable to UI, and viewing area 7110,7115 and 7160 still can be appeared in UI 7175 completely.Some embodiment can provide the message (for example,, by showing words " Rotating (rotation) ") of the state of indicating equipment 7100.

Five-stage 7140 is illustrated in viewing area 7110,7115 and 7160 and is rotated counterclockwise the UI 7175 after 90 ° from longitudinal view to transverse views.In this stage, viewing area 7110,7115 and 7160 has been rotated, but is not also deployed into the whole width of UI 7175.The ending that arrow 7170 is pointed out at five-stage, viewing area 7110,7115 and 7160 will start side direction launches, to be applicable to the whole width of UI 7175.Different embodiment can not comprise this stage because can with fourth stage 7135 in rotation side by side carry out described expansion.

The 6th stage 7145 was illustrated in viewing area 7110,7115 and 7160 and has been unfolded, to occupy the UI 7175 after the whole demonstration of UI 7175.As mentioned above, other embodiment can differently realize this rotation.Concerning some embodiment, only the screen rotation of equipment is exceeded to threshold quantity and just can trigger described rotation operation, and the orientation of tube apparatus 7100 not.

In addition, other embodiment can be provided for the different animations of instruction rotation operation.The rotation operation of carrying out in Figure 71 relates to UI and shows that 7175 show 7175 central rotation around UI.On the other hand, can make viewing area around they separately the central shaft of viewing area rotate separately.In Figure 72, represented a kind of such method, Figure 72 represents the alternative approach of the rotation of the PIP viewing area 7180 of flash demo UI 7175.In Figure 72, in the PIP of graphic extension demonstration and Figure 71, the PIP of graphic extension shows that 7180 is identical.

Figure 72 is according to six different operational phases 7120,7125,7130,7220,7225 and 7230, and graphic extension PIP shows 7180 rotation.The operation of the front three phases of UI 7175 is with as identical in the operation of the front three phases of explanation in the UI 7175 in Figure 71.In the phase III of Figure 71 and 72, equipment all becomes horizontal holding from perpendicular holding, and the rotation of UI 7175 does not also start.

The alternative approach of rotating described in fourth stage 7220 graphic extension flash demos.In fourth stage, rotation operation starts.Specifically, fourth stage 7220 shows the beginning of the rotation of viewing area 7110,7115 and 7160.Viewing area 7110,7115 and 7160 rotates around the axle 7250 (, z axle) at the center through each viewing area respectively.Viewing area 7110,7115 and 7160 is rotated identical amount, but along the direction rotation contrary with the rotation (for example, by the inclination of equipment 7100) of equipment 7100.In this example, because equipment 7100 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), therefore rotate operation and can make viewing area 7110,7115 and 7160 half-twist in the counterclockwise direction.In the time of 7110,7115 and 7160 rotation of viewing area, they are also scaled, show 7175 to be applicable to UI, and viewing area 7110,7115 and 7160 still can be appeared on UI 7175 completely.

Five-stage 7225 is illustrated in viewing area 7110,7115 and 7160 and is all rotated counterclockwise the UI 7175 after 90 ° from longitudinal view to transverse views.In this stage, viewing area 7110,7115 and 7160 has been rotated, but is not also deployed into the whole width of UI 7175, or arrives their final position.Viewing area PIP show final position in 7115 by as in the viewing area as shown in the first stage 7120, the location positioning in PIP demonstration 7115 is (for example, insert and show that 7110 show 7180 the lower left corner at PIP, and insertion shows that 7160 show 7180 the lower right corner at PIP).

The ending that arrow 7170 is pointed out at five-stage, viewing area 7115,7110 and 7160 will start side direction launches, until main viewing area 7115 is applicable to the whole width of UI 7175.In addition, arrow 7255 is pointed out to insert viewing area 7110 and 7160 and will be moved, thereby arrives their final positions in PIP shows 7180.In other words, insert viewing area 7110 and will move towards the lower left corner of PIP demonstration 7180 downwards, and the lower right corner of PIP demonstration 7180 is shifted in another insertion viewing area 7160.Different embodiment can differently realize this animation, for example, by utilizing locking and the push operation of graphic extension in Figure 71.Thereby the 6th stage 7230 was illustrated in viewing area 7110,7115 and 7160 and has been unfolded and occupies the whole demonstration of UI 7175 and moved to the UI 7175 after their final position.

As mentioned above, other embodiment can differently realize this rotation.For example, similar with graphic extension in Figure 36 and 37, some embodiment provide a kind of rotation operation, wherein show the directed change of the viewing area of the video of being taken by local device, to be reflected in, local device is rotated to the operation orientation of local device afterwards, some embodiment provide a kind of rotation operation, wherein show the directed change of the viewing area of the video of being taken by remote equipment, to be reflected in, remote equipment is rotated to the operation orientation of remote equipment afterwards, the rotation operation that some embodiment provide viewing area 1155 to be wherein retained in same position, some embodiment provide different layouts in viewing area (for example, the layout of the viewing area 1155 of Figure 12), or their combination.

Concerning some embodiment, only make the screen rotation of equipment exceed threshold quantity and just can trigger rotation operation, and the orientation of tube apparatus 7100 not.Described above in addition, local device and remote equipment can be by controlling communication port, and the rotation operation that notice is carried out an equipment in two equipment mutually, to allow another equipment the video of a described equipment to be carried out to the amendment of any correspondence.In addition, the animation of rotation operation also can cause the variation of the operation of camera or the image processing operations of two camera movement equipment, such as making video conference manager 1604 one or more viewing areas of the one-tenth different angles in synthetic UI 1105 again, and proportional zoom is presented at the image in one or more viewing areas.

7. select to watch remote view

As mentioned above, some embodiment allow the user of two camera movement equipment before starting video conference, or in the time starting video conference, select which camera for video conference.Replace this ability, or be combined with this ability, some embodiment allow the user of this equipment to select between two videos, described two videos be in video conference, show and or from two video cameras of remote equipment, or from two video cameras of user's local device.Figure 73 is illustrated in and in meeting, selects a video in two long-distance videos, and Figure 74 is illustrated in and in meeting, selects a video in two local videos.

Figure 73 is according to six operational phases 7335,7340,7345,7350,7355 and 7360 that are presented at the UI 7375 on local device 7300, the selection of long-distance video described in graphic extension.First stage 7335 graphic extension have with there is long-distance user's the video conference of mobile device of two cameras during, the initial p IP presenting shows 7390 UI 7375.

As shown in the first stage 7335, initial p IP shows that 7390 comprise three demonstrations: the main demonstration 7315 of background and two prospects are inserted and shown 7305 and 7310.Background shows that 7315 occupy most PIP viewing area 7390, and prospect insertion shows that the 7305 and 7310 part backgrounds that overlap respectively on UI 7375 show on 7315.In this example, background shows that 7315 are presented on microphone personage's before video, and described video supposition is the video by the back side camera of remote equipment.The first prospect is inserted and is shown that 7305 present the video of man's face, and in this example, this video supposition is the video that one of camera by local device 7300 is taken.The second prospect is inserted and is shown that 7310 present the personage's who wears a hat video, and in this example, this video supposition is the video by the positive camera of remote equipment.

Initial p IP demonstration 7390 just presents a kind of mode by the synthetic view of the video of the camera of local device and remote equipment.Some embodiment can provide other synthetic view.For example, background shows the video that can present one of camera from local device, and less prospect is inserted demonstration and can be presented from the front camera of remote equipment and the video of back side camera.In addition, in some cases, PIP shows that can only include all background videos from remote equipment shows and a prospect video demonstration.In certain embodiments, the mode that PIP shows or acquiescence display mode can be specified by user.

Second stage 7340 graphic extension videos are selected the beginning of operation.In this example, by calling one group of optional UI project that will be presented in PIP demonstration 7390, initiate this operation.Described one group of optional UI project presents for selecting to show the option with long-distance video.In certain embodiments, for example, by selecting any viewing area of the broadcasting long-distance video on (, by touching) UI 7375, call described one group of optional UI project.In other embodiments, for example, by selecting any place on (, by touching) UI 7375, can call described each project.Replace this call operation, or be combined with this call operation, some embodiment also allow user to pass through other operation, such as the touch screen operation by different, or utilize one or more other physics of equipment to input, and call described one group of optional UI project.

Phase III 7345 shows UI 7375, UI 7375 have one group call for selecting the optional UI project 7380 of long-distance video.In this example, the one group of optional UI project 7380 that is pop-up menu form is displayed in PIP viewing area 7390, overlaps in this PIP demonstration.Described one group of optional UI project 7380 (they can be realized as optional button) (for example comprises " Select R1 " optional UI project 7320, button 7320), " Select R2 " optional UI project 7325 (for example, button 7325), " Select Both " optional UI project 7330 (for example, button 7330), and " Cancel " optional UI project 7385 (for example, button 7385).In this example, select " Select R1 " button 7320 can make the video (be presented on background show 7315 in) of 7375 demonstrations of UI by the back side camera of remote equipment.Select " SelectR2 " button 7325 can make the video (be presented on prospect insert show 7310 in) of 7375 demonstrations of UI by the positive camera of remote equipment.Select " Select Both " button 7330 can make UI 7375 continue to show by the front camera of remote equipment and two videos of back side camera.Select " Cancel " button 7385 can cancel this operation.In certain embodiments, the video of being taken by local device is not subject to the impact of the selection of carrying out on this menu.

Fourth stage 7350 is illustrated in user and has selected " Select R1 " button 7320 (for example, dubbing button 7320 by the finger 7365 with him) UI 7375 afterwards.By highlighting the button 7320 on UI 7375, indicate this selection.Some embodiment utilize different indicated number (for example, highlight the frame of selected item, or text in selected item).

Five-stage 7355 is illustrated in user and has selected from the video of R1 so that the animation of the UI 7375 after showing.In this example, by as shown in arrow 7370, make unwanted prospect insertion viewing area 7310 skid off PIP and show 7390 right hand edge, UI 7375 removes unwanted prospect and inserts viewing area 7310.Other embodiment utilizes different animations to eliminate unwanted insertion viewing area, such as making this insertion viewing area fade out or fade out, move this insertion viewing area or eliminate immediately simply this insertion viewing area along different directions.

After the 6th stage 7360 was presented at and finishes video selection operation, the UI 7375 during video conference.Video viewing area 7310 is no longer presented on UI 7375.In this stage, UI7375 presents new PIP and shows that 7395, PIP shows that 7395 comprise the video viewing area 7315 of main demonstration as a setting, and as inserting the video viewing area 7305 showing.

In certain embodiments, this video selects to operate the video that also can make remote equipment only show selection, but in other embodiments, video is selected to operate remote equipment without any impact.In certain embodiments, this video selects operation can make remote equipment stop transmitting unwanted video to local device.In fact, in certain embodiments, lid video selects operation can make the camera of remote equipment stop taking unwanted video.In certain embodiments, the user of remote equipment can veto these impacts on remote equipment.

The remote view that example above illustrates selection is the situation that has been presented at the remote view in the main demonstration of background.In certain embodiments, in the time that user selects to be presented at the remote view inserting in one of demonstration, the remote view of selection is displayed in the main demonstration of background.In this case, animation like some such embodiment utilizations and the animated type that shows in Figure 68.In addition, the selection of long-distance video can cause the variation of the image processing operations of local two camera movement equipment, such as making video conference manager 1604 respond user's input, only synthesizes one or more long-distance videos of selecting in compound display.

8. select to watch local view

Figure 74 is according to six operational phases 7435,7440,7445,745,7455 and 7460 that are presented at the UI 7475 on local device 7400, the selection of graphic extension local video.First stage 7435 graphic extension UI 7475, UI 7475 have with there is the initial p IP presenting during long-distance user's the video conference of mobile device of at least one camera and show 7490.PIP demonstration 7490 and the PIP in the first stage 7335 in Figure 73 show similar, except being different from Figure 73, background demonstration 7415 presents the video by the man of the camera of remote equipment, the prospect insertion demonstration 7410 in left side presents the video by the personage who holds guitar of the back side camera of local mobile device, and the prospect insertion demonstration 7405 on right side presents outside the video by the man who wears a hat of the positive camera of local mobile device 7400.Thereby, only show a long-distance video, and show two local videos.

Second stage 7440 graphic extension videos are selected the beginning of operation.In this example, to be presented in PIP demonstration 7490 by calling, show one group of optional UI project with long-distance video for selecting, initiate this operation.In certain embodiments, for example, by selecting (, by touching) UI to show any viewing area of the broadcasting local video on 7475, can call described one group of optional UI project.In other embodiments, for example, by selecting (, by touching) UI to show any place on 7475, can call described each project.Replace this call operation, or be combined with this call operation, some embodiment also allow user to pass through other operation, such as the touch screen operation by different, or utilize one or more other physics of equipment to input, and call described one group of optional UI project.

Phase III 7445 shows UI 7475, UI 7475 have one group call for selecting the optional UI project 7480 of local video.In this example, the one group of optional UI project 7480 that is pop-up menu form is displayed in PIP viewing area 7490, overlaps in this PIP demonstration.Described one group of optional UI project 7480 (for example comprises " Select L1 " optional UI project 7420, button 7420), " Select L2 " optional UI project 7425 (for example, button 7425), " Select Both " optional UI project 7430 (for example, button 7430), and for example, for cancelling " Cancel " optional UI project 7485 (, button 7485) of this operation.In this example, select " Select L1 " button 7420 can make the video (be presented on prospect insert show 7410 in) of 7475 demonstrations of UI by the back side camera of local device.Select " SelectL2 " button 7425 can make the video (be presented on prospect insert show 7405 in) of 7475 demonstrations of UI by the positive camera of local device.Select " Select Both " button 7430 can make UI 7475 continue to show two videos by two camera of local device, and select " Cancel " button 7485 can cancel this operation.In certain embodiments, the video of being taken by remote equipment is not subject to the impact of the selection of being undertaken by this menu.

Fourth stage 7450 is illustrated in user and has selected " Select L2 " button 7425 (for example, dubbing button 7425 by the finger 7465 with him) UI 7475 afterwards.By highlighting the button 7425 on UI 7475, indicate this selection.Some embodiment utilize different indicated number (for example, highlight the frame of selected item, or text in selected item).

Five-stage 7455 is presented at user and has selected from the video of L2 so that the animation of the UI 7475 after showing.In this example, by as shown in arrow 7470, unwanted prospect is inserted and shown that 7410 skid off the left hand edge of PIP demonstration 7490, UI 7475 removes unwanted prospect and inserts demonstration 7410.Other embodiment utilizes different animations to eliminate unwanted insertion viewing area, such as making this insertion viewing area fade out or fade out, move this insertion viewing area or eliminate immediately simply this insertion viewing area along different directions.

The 6th stage be presented at video select operation after, the UI7475 during video conference.Video viewing area 7410 is no longer on UI 7425.In this stage, UI 7475 presents new PIP and shows that 7495, PIP shows that 7495 comprise that the long-distance video of main demonstration as a setting shows 7415, and shows 7405 as inserting the local video showing.In certain embodiments, this video selects operation only to show impact this locality, because two videos are still transmitted to remote equipment.Other embodiment stops the camera video from removing.

The local view that example above illustrates selection is the situation that has been presented at the view in the main demonstration of background.In certain embodiments, in the time that user selects to be presented at the local view inserting in one of demonstration, the local view of selection is displayed in the main demonstration of background.In this case, animation like some such embodiment utilizations and the animated type that shows in Figure 68.In the time that the local view in the main demonstration of background is removed, other embodiment will utilize and insert remote view.

Be similar to above with reference to the remote view of Figure 73 explanation and select operation, the selection of local video can cause the variation of the image processing operations of local two camera movement equipment, such as making video conference manager 1604 respond user's input, in compound display, only synthesize one or more long-distance videos of selecting.The selection of local video also can cause the variation of the operation of one or more cameras of local device.For example, some embodiment make the camera of unwanted video stop transmitting unwanted video to remote equipment, and other embodiment makes this camera stop taking unwanted video.

9. select to transmit local view

Each height joint above for example understands the amendment in meeting, video being shown.Some embodiment also allow the user of two camera movement equipment before starting video conference, select which camera for video conference.Figure 75 is illustrated in before meeting, selects a video, for the operation of video conference from two videos being taken by two camera movement equipment of user.

Figure 75 is according to eight operational phases of UI 7500, and graphic extension is for the selection of the local video of video conference.The UI 7500 of the two camera movement equipment 7518 of first stage 7502 graphic extension, UI 7500 have user ask to start with the long-distance user's of mobile device video conference after the initial p IP demonstration 7542 that presents.

As shown in the first stage 7502, initial p IP shows that 7542 comprise that two videos show: the main demonstration 7520 of background and prospect are inserted and shown 7522.The main demonstration 7520 of background and prospect are inserted and are shown 7522.The main demonstration 7520 of background occupies most of display screen of equipment, and prospect is inserted and shown that 7522 is less and overlap in the main demonstration of background.In this example, background shows that 7520 present the personage's who holds guitar video, and this video supposition is the video by the back side camera of equipment.Prospect is inserted and is shown that 7522 present the personage's who wears a hat video, and in this example, this video supposition is the video by the positive camera of equipment.

Initial p IP demonstration 7542 just presents a kind of mode by the synthetic view of the video of the camera of local device.Some embodiment can provide other synthetic view.For example, background shows the video that can present from the front camera of equipment, and less prospect is inserted and shown the video that can present from the back side camera of equipment.In addition, some embodiment allow two videos to appear in UI 7500 two viewing area (for example, left and right display window, or upper and lower display window) side by side, or in the viewing area of arranging, two diagonal angles.In certain embodiments, the mode that PIP shows or acquiescence display mode can be specified by user.PIP is the optional UI project 7540 (for example, button 7540) that is labeled as " End Conference " below showing, optional UI project 7540 allows this project of user by selecting and finishes video conference.

In the first stage 7502, the user of mobile device 7518 has asked to carry out video conference with long-distance user, and is waiting for that long-distance user responds." Preview, the Waiting for response... (preview, positive wait-for-response ...) " of the bottom surface of display frame explain graphic extension this waiting period.

Second stage 7504 graphic extension videos are selected the beginning of operation.In this example, by calling one group of optional UI project that will be presented in PIP demonstration 7542, initiate this operation.Described one group of optional UI project presents for selecting local video, to send the variety of option of remote equipment of video conference to.In certain embodiments, in waiting for long-distance user's response, for example, by selecting (, touching) UI to show any place on 7500 in the time before meeting, can call described one group of optional UI project.Replace this call operation, or be combined with this call operation, some embodiment also allow user to pass through other operation, such as the touch screen operation by different, or utilize one or more other physics of equipment to input, and call described one group of optional UI project.

Phase IIIs 7506 graphic extension has one group of user that supplies who calls and selects the UI 7500 of the optional UI project 7526 of video.In this example, the one group of optional UI project 7526 that is pop-up menu form is presented in PIP viewing area 7542, and overlaps in PIP demonstration.In this example, described one group of optional UI project comprises: " Transmit L1 " project 7528 (for example, button 7528); " Transmit L2 " project 7530 (for example, button 7530); " Transmit Both " project 7532 (for example, button 7532); " Cancel " project 7534 (for example, button 7534).In this example, select " Transmit L1 " button 7528 can make UI 7500 during video conference, only the video of the back side camera by equipment is sent to remote equipment.Select " Transmit L2 " button 7530 can make UI 7500 during video conference, only the video of the positive camera by equipment is sent to remote equipment.Select " Transmit Both " button 7532 can make UI 7,500 two videos of the front and back camera by equipment be sent to the long-distance user of video conference, and select " Cancel " button 7534 can cancel this operation.

Fourth stage 7508 is illustrated in user and has selected " Transmit L1 " button 7528 (for example, dubbing button 7528 by the finger 7524 with him) UI 7500 afterwards.By highlighting the button 7528 on PIP viewing area 7542, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the text in frame or the selected item of selected item).

Five-stage 7510 is illustrated in user has selected the video of the back side camera of equipment, to send the animation of the UI 7500 after remote equipment to.In this example, by as shown in arrow 756, unwanted prospect is inserted and shown that 7522 skid off the right hand edge of PIP demonstration 7542, UI 7500 removes unwanted prospect and inserts demonstration 7522.In the 6th stage 7512, insert and show that 7522 have been completely removed from PIP viewing area 7542.Different embodiment utilizes different animations to remove unwanted viewing area, such as making this viewing area fade out or fade out, move this viewing area or eliminate immediately simply this viewing area along different directions.

The 7th stage 7514 was illustrated in long-distance user has accepted the animation of the UI 7500 after video conference request.By eliminating " Preview, the Waiting forresponse... " annotation in display frame, highlight the acceptance of video conference request.In this stage, narrow down to gradually the lower left corner of PIP viewing area 7542 as the background viewing area 7520 of the video of the back side camera from equipment, as shown in arrow 7538.Background shows 7520 dwindles, and UI 7500 can be shown after viewing area 7520 comprise the viewing area 7544 from the video of long-distance user's camera.Some embodiment narrow down to different positions local camera, utilize the tiling compound display of two videos that show, or make remote view become the insertion viewing area that PIP shows.

The 8th stage 7516 was illustrated in video and has selected operation UI 7500 afterwards.UI 7500 presents new PIP and shows that 7546, PIP shows that 7546 comprise that the insertion of the video of taking from local device shows 7520, and the background of the video sending from remote equipment shows 7544.

B. bandwidth & frame rate

In certain embodiments, the big or small adjustment of the viewing area to remote-moving apparatus during video conference can make local mobile device redistribute to distribute to the bandwidth of each video of being taken by two cameras (, front camera and back side camera) of local mobile device.Figure 76 is illustrated in two examples that this bandwidth between two cameras of local device is redistributed.

Each example in Figure 76 relates to the back side camera sensor 7605 of local device, positive camera sensor 7610, the video conference module 7615 of local device and the UI 7635 of remote-moving apparatus 7620 of local device.Back side camera sensor 7605 and positive camera sensor 7610 are caught from the respective back surfaces camera of local device and the video of front camera.The video of catching is sent to video conference module 7615, and video conference module 7615 is processed the video of catching, and they are sent to remote equipment, to show in UI 7635.

In Figure 76, the UI 7635 of remote equipment presents compound display.Compound display represents by the front camera of local device and the video of back side camera.From the video capture tree of front camera and the man who wears a hat, and from the video capture mountain range landscape of back side camera.As shown in Figure 76, according to the size of the arrangement of the viewing area for display video and viewing area, available many different modes show this two videos in UI 7635.In each example, video conference module 7615 is initial according to the relative size of the viewing area in remote equipment, distributes total output bandwidth between each video.Specifically, the video being presented in the larger viewing area in UI 7635 is assigned with the total bandwidth of major part, and is presented at video in the less viewing area in UI7635 and is assigned with the bandwidth of smaller portions.In certain embodiments, in the time that video is displayed in the viewing area of formed objects, between these two videos, divide total output bandwidth equally.

The amount of distributing to the bandwidth of the each video in these two videos can affect the mode of processing each video most.For example, video may need the bandwidth larger than the bandwidth of distributing to this video.In this case, the frame rate of video is adjusted, or the size of video image dwindled by ratio, to adapt to less bandwidth.Reduce the frame rate of video and cause video seem " discontinuous ", and ratio is dwindled the size of video image and can reduce to show the region of this video.So in the time that video is assigned with a certain amount of bandwidth, some embodiment adjust the frame rate of this video, ratio is dwindled the size of video image, or carries out the two combination, transmits this video to guarantee in the bandwidth of distributing.Those of ordinary skill in the art will appreciate that the adjustment of frame rate and average frame size can change, and to obtain best overall video quality, still guarantees to transmit this video in the bandwidth of distributing simultaneously.

The example (1) of Figure 76, with two operational phases of UI 7635, illustrates a kind of situation that bandwidth is redistributed.The UI 7635 of the remote equipment 7620 in the first stage 7670 presents the compound display that comprises two demonstrations: one at the top of UI 7635, and another is in the bottom of UI 7635.In this example, the viewing area 7625 at top shows the video by the positive camera of local device, and the viewing area 7630 of bottom shows the video by the back side camera of local device.As shown in the first stage 7670, the viewing area 7625 at top is greater than the viewing area 7630 of bottom.Thereby, be assigned with 80% bandwidth from the video of the front camera of local device, be assigned with 20% bandwidth from the video of the back side camera of local device.In order to ensure can, in the bandwidth of distributing, the video of the back side camera from local device being sent to remote equipment from local device, adjust frame rate and/or the scaling size of this video.

The user that second stage 7675 is illustrated in remote equipment increases the size of viewing area, bottom, makes the approximate UI 7635 after identical of the size of viewing area, top 7625 and viewing area, bottom 7630.As a result, each video is redistributed 50% total bandwidth by video conference module 7615.

The example (2) of Figure 76, with two operational phases of UI 7635, illustrates the another kind of situation that bandwidth is redistributed.In the first stage 7680 of example (2), UI 7635 presents PIP and shows.This PIP demonstration comprises two demonstrations: the main viewing area 7650 of background and prospect are inserted viewing area 7655.The main viewing area 7650 of background occupies most PIP and shows, and that prospect is inserted viewing area 7655 is less and overlap on the main viewing area 7650 of background.In this example, background viewing area 7650 presents the video by the positive camera of this equipment.Insert viewing area 7655 and present the video by the back side camera of this equipment.As shown in this stage, background viewing area 7650 is greater than inserts viewing area 7655.Thereby, be assigned with 80% bandwidth from the video of the front camera of equipment, be assigned with 20% bandwidth from the video of the back side camera of equipment.In order to ensure in the bandwidth of distributing, the video of the back side camera from local device can be sent to remote equipment from local device, adjust frame rate and/or the scaling size of video.

The user that second stage 7685 is illustrated in remote equipment has exchanged the UI 7635 after the demonstration of these two videos.Specifically, background viewing area 7660 presents the video by the back side camera of equipment now, and insertion viewing area 7665 presents the video by the positive camera of equipment now.Because the size of the viewing area of these two videos changes, be therefore assigned with 80% bandwidth from the video of the back side camera of equipment, be assigned with 20% bandwidth from the video of the front camera of equipment.Thereby, be reduced from frame rate and/or the large young pathbreaker of scaling of the video of the front camera of local device.Those of ordinary skill in the art will appreciate that the bandwidth distribution illustrating in Figure 76 is example, and during video conference, other technology of distributing bandwidth between two cameras is also possible.

1. frame rate control

Be similar to the frame rate control operation in meeting described above, some embodiment may wish to adjust separately or remain on to another equipment in video conference and transmit the speed by the video image of every camera of two camera movement equipment.Some such embodiment provide similar techniques described above.For example, some embodiment, by the VBI of the sensor assembly 415 of every camera of adjustment, control the frame rate of every camera.Other embodiment also provides other technology, such as the frame that can be carried out by the sensor assembly of every camera 415 and/or general transmission buffer 3120 abandons.

2. the bandwidth control of passing ratio convergent-divergent

As mentioned above, during the video conference between two camera movement equipment and another equipment, in certain hour amount, the quantity (, network connects bandwidth) that connects the view data that can transmit by one or more networks is limited.The throughput connecting in order to maximize and maintain network, the different embodiment of two camera movement equipment provide different modes to be controlled in special time amount, connect the quantity of the view data transmitting by network.In certain embodiments, throughput is the Mean Speed for example, transmitting by the success message of communication port (, network connects).

When transmit by two camera of two camera movement equipment image time, a kind of such method adjustment is from the size of the image of one or two camera of two camera movement equipment, to control the quantity that connects the view data of transmission by network.Some embodiment ratios are dwindled the size of the image of being taken by two camera movement equipment, to reduce the quantity that connects the view data transmitting by network, and the size of other embodiment ratio enlarged image, to increase the quantity that connects the view data transmitting by network.

Some embodiment, in the time of convergent-divergent, keep the depth-width ratio (, even convergent-divergent) of image.Other embodiment zoomed image, makes the depth-width ratio of the image after convergent-divergent be different from the depth-width ratio (, distortion convergent-divergent) of original image.

In addition, can carry out proportional zoom in the different phase of image processing process.The proportional zoom of some embodiment can be undertaken by camera sensor.In such embodiments, the multirow of the discardable image of camera sensor or multi-column data (, pixel value).In some such embodiment, remaining view data is by interpolation, with the profile of smoothed image.

The proportional zoom of other embodiment is carried out by the scaler module 455 of CIPU 400.In certain embodiments, proportional zoom is carried out by video conference manager 1604, and as mentioned above, in other embodiments, proportional zoom is carried out by encoder.Thereby the different embodiment of two camera movement equipment differently carry out proportional zoom.

3. Bit-Rate Control Algorithm

Some embodiment provide different mechanism to manage the bit rate of the Video coding to the camera by two camera movement equipment.In certain embodiments, two camera movement equipment comprises the rate controller for every camera.Some embodiment provide fixed bit rate Managed Solution.According to this scheme, each rate controller is set to fixed bit rate, and it is constant making from the gross bit rate of the video of two cameras on mobile device.Other embodiment provides precedence scheme, wherein in the time requiring to reduce gross bit rate, always obtains the priority higher than another video from one of two videos of the camera of equipment.

In certain embodiments, two rate controllers of two cameras of arbiter module management.The example of this arbiter module of Figure 77 graphic extension.As shown in Figure 77, rate controller 7700 is set the bit rate of front camera, and rate controller 7705 is set the bit rate of back side camera.Rate controller is sending to encoder 7715 from the image of camera sensor.Arbiter module 7710 is connected with two rate controllers, and according to video size of the each video such as in available bandwidth, two videos etc., with the setting of the bit rate of the each rate controller 7700 and 7705 of any various ways control.To guarantee under available bandwidth, two videos all can be sent to remote equipment.In addition, arbiter 7710 can be configured to realize above mentioned fixed rate scheme, or precedence scheme.

In some other embodiment, can intercom mutually for two rate controllers of two cameras.According to this scheme, rate controller can exchange their information of video separately, and sets accordingly the bit rate of video.Several examples of rate controller speed administrative mechanism are provided.But, many other different mechanisms are also possible.

4. Video processing

Some embodiment of two camera movement equipment, different in the situation that, differently process the image by two camera of two camera movement equipment.For example, in the time that processing comprises the PIP composograph by the image of two camera of two camera movement equipment, some embodiment carry out TNR to PIP composograph selectively and process 2000.Some this embodiment only carry out TNR to the master image in PIP composograph and process 2000, and other this embodiment only carries out TNR processing 2000 to the insertion image in PIP composograph.

Another example as processing by the image of two camera of mobile device, some embodiment are according to the various changes of video conference, such as the adjustment of the viewing area of user to display video (for example, main viewing area/insertion the viewing area that amplify the insertion viewing area of PIP demonstration, define area-of-interest in the video showing, exchange PIP shows, etc.), to the change of total available bandwidth, etc., proportional zoom is by the image of two camera of two camera movement equipment.Some such embodiment proportional zoom image in the manner described above., image can be by encoder 1655, video conference manager 1604, scaler module 455 and camera sensor (, 405a or the 405b) proportional zoom with its photographic images.

5. coding

As mentioned above, some embodiment send the video from two cameras of two camera movement equipment.Thereby these embodiment can be to the Video coding by two camera, to send remote equipment to during video conference.Different embodiment provides different modes to carry out encoded video so that transmission.Figure 78 graphic extension utilizes multiplexer (MUX) 7815, coder module 7825, buffer 7830 and composite module 7835 to process video, so that the method for transmission.

According to selecting signal, MUX 7815 obtains an input signal, and the input signal of selecting is exported to encoder 7825.For example, if the input signal of selecting signal designation MUX 7815 to obtain from C1, MUX 7815 selects this input signal so, and exports this input signal.Available various ways, such as by the instruction from video conference manager 1604, provides described selection signal.By MUX 7815, encoder 7825 alternately becomes bitstream format the Image Coding receiving from MUX7815, and an Image Saving of coding is in buffer 7830.Composite module 7835 combines (that is, multiplexed) and is kept at the one or more bit streams in buffer 7830, and exports single bit stream.

, according to three phases 7860,7865 and 7870, the operation of this coding method is described now.In the first stage 7860, MUX 7815 is configured to receive the image 7805 of being taken by camera C 1, and exports to encoder 7825 so that coding.Encoder 7825 is to the Image Coding receiving and produce bit stream 7850, and bit stream 7850 is stored in buffer 7830 subsequently.Second stage 7865 is similar to the first stage 7860, except MUX 7815 is configured to receive the image 7810 of being taken by camera C 2, and exports to encoder 7825, so that outside coding.Similarly, encoder is to the Image Coding receiving and produce bit stream 7855, and bit stream 7855 is stored in buffer 7830.In the phase III 870, composite module 7835 is fetched bit stream 7850 and 7855 from buffer 7830, and they are combined into a bit stream, to send remote equipment to.

Figure 79 graphic extension is to two Video codings from two camera movement equipment, to send the another kind of method of remote equipment to during video conference.In this method, from the frame of video of the First camera of mobile device (, image) and to be synthesized from another frame of video of second camera of mobile device be a frame of video, synthetic frame of video is encoded into bit stream afterwards, to send to remote equipment.As shown in Figure 79, this method comprises synthesizer 7915, buffer 7920 and encoder 7925.

As shown in the figure, synthesizer 7915 is synthetic from the image 7905 of First camera with from the image 7910 of second camera, to form composograph 7955.Different embodiment is composograph 7905 and 7910 differently.For example, the synthesizer 7915 of some embodiment can be by being arranged in (as shown in Figure 80) adjacent one another are, composograph two images.Composograph 8030 and 8035 graphic extensions utilize two example composographs of this technology.In composograph 8030, be arranged on the image 7910 from the second camera from the image 7905 of First camera.And composograph 8035 presentation videos 7905 are arranged in the left side of image 7910.

At some in other embodiment, synthesizer 7915 can by two images 7905 and 7910 are superimposed upon larger background image above, synthetic two images 7905 and 7910.Composograph 8040 graphic extensions of Figure 80 utilize the example composograph of this technology.In composograph 8040, image 7905 and 7910 diagonal angles are arranged, and are superimposed upon (that is, image 7905 is positioned at the upper left corner of background image, and image 7910 is positioned at the lower right corner of background image) in blank image.In certain embodiments, camera sensor may vary in size, thereby takes the image with different pixels resolution.In such an embodiment, synthesizer 7915 can be according to the similar manner composograph 7905 and 7910 of composograph 8045 graphic extensions as by Figure 80.After synthetic these two images, synthesizer 7915 is kept at composograph in buffer 7920.Encoder 7925 is fetched composograph from buffer 7920, and composograph is encoded into bit stream, and sends it to the remote equipment of video conference.

With reference now to synthesizer 7915, buffer 7920 and the encoder 7925 of graphic extension in Figure 79,, each operation is described.First, First camera sends to synthesizer 7915 the image 7905 of the part as a series of images in video.Meanwhile, second camera sends to synthesizer 7915 the image 7910 of the part as a series of images in video.Synthesizer 7915 composograph 7905 and 7910 in the manner described above subsequently, to form composograph 7955.Afterwards, synthesizer 7915 sends to buffer 7920 composograph 7955.Buffer 7920 is preserved composograph subsequently, afterwards composograph is sent to encoder 7925.Finally, encoder 7925 is encoded into bit stream composograph, and sends it to the remote equipment of video conference.

Figure 81 graphic extension is to two Video codings from two camera movement equipment, to send another method of remote equipment to during video conference.In this method, shown with the form of compound display from two videos of this equipment, obtain the screenshot capture of compound display, and be encoded into bit stream, to send to remote equipment.As shown in Figure 81, this method comprises encoder 8115.In certain embodiments, encoder 8115 synthetic images codings, and send to remote equipment.

With reference now to the encoder 8115 of graphic extension in Figure 81,, each operation is described.First, be displayed on the screen of equipment with the form of compound display from the video of two cameras of two camera movement equipment.Described compound display can present video by any-mode.For example, the compound display in some embodiment can be presented on PIP two videos and show, such as the PIP of graphic extension in Figure 81 shows in 8105.In other embodiments, compound display can be presented on two viewing areas side by side two videos, or in the viewing area of two diagonal angle arrangements.Obtain PIP and show 8105 screenshot capture, such as image 8110, and send it to encoder 8115.Encoder is encoded into bit stream 8120 a series of screenshot capture subsequently, sends it to afterwards the remote equipment of video conference.Although several distinct methods to two Video codings have been described above, but other method is also possible.

6. decoding

Some embodiment of two camera movement equipment can receive the bit stream of encoding with reference to the method for figure 78-81 explanation with above.In such embodiments, two camera movement equipment can receive the information of (for example,, by video conference control channel) instruction for the method to Video coding.Figure 82 graphic extension one is to passing through communication network, and the bit stream decoding of two videos that receive from another equipment, so that during video conference, is presented at the method on two camera movement equipment.Specifically, this method is for the bit stream decoding of the coding method coding to illustrating with reference to figure 78 above.

As shown in Figure 82, this scheme is used separation module 8235, buffer 8230 and 8290 and decoder module 8225.Separation module 8235 separates bit stream (, multichannel decomposition) and becomes one or more bit streams, and bit stream is kept in buffer 8230.Decoder 8225 is fetched the bit stream of coding, and they are decoded to produce video, subsequently video is kept in buffer 8290.

With reference now to the separation module 8235 of graphic extension in Figure 82, buffer 8230 and 8290 and decoder module 8225,, the operation of this method is described.First, two camera movement equipment is by communication network, and another equipment from video conference receives bit stream 7845 (, at networking manager 1614).Separation module 8235 is separated into two bit streams 8255 and 8260 the bit stream receiving, because the bit stream receiving is the multiplexed bit stream of two bit streams.The bit stream of each coding represents the video data of taking from one of two cameras of equipment.Subsequently, separation module 8235 is kept at bit stream 8255 and 8260 in buffer 8230.

Afterwards, decoder 8225 is fetched bit stream 8250 (it is two bit streams 8255 and 8260) from buffer 8230, and decoder 8225 is decoded to bit stream 8250, to produce video 8280, and video 8280 is kept in buffer 8290.Decoder 8225 is also to another decoding in bit stream 8255 and 8260, and a video producing is kept in buffer 8290.Now, two videos can be retrieved from buffer 8290, and are saved or are presented on two camera movement equipment.

Figure 83 graphic extension is to using the method for the bit stream decoding of encoding with reference to the method for Figure 79 explanation.As shown in Figure 83, the method comprises decoder 8325, buffer 8320 and decombiner 8315.

In certain embodiments, decoder 8325 receives the bit stream with the method coding of graphic extension in Figure 79, and bit stream decoding is become to one or more composographs, and described composograph is stored in buffer 8320 subsequently.Decombiner 8315 extracts this two images from each composograph.In order to extract this two images from composograph, the information that decombiner 8315 also receives the position of the each image of instruction in composograph (for example, by video conference communication control channel, the information that the equipment of the synthetic and coded image from video conference receives).

With reference now to decoder 8325, buffer 8320 and the decombiner 8315 of graphic extension in Figure 83,, the operation of this method is described.First, decoder 325 is from the bit stream of another mobile device receiver, video of video conference, such as using the bit stream of the video producing with reference to the method for Figure 79 explanation.Decoder 8325 becomes to comprise one or more composographs of composograph 7955 this bit stream decoding, and they is kept in buffer 8320.Buffer 8320 is preserved composograph subsequently, afterwards they is sent to decombiner 8315.In the time that decombiner receives composograph 7955 from buffer 8320, it is separated into composograph 7955 and image 7905 and 7910 identical two images 7905 and 7910 in Figure 79.

In the time that the system from the system illustrating such as Figure 81 receives bit stream, decoder, such as the decoder 8325 in Figure 83 becomes a series of screenshot capture bit stream decoding.Described a series of screenshot capture is displayed on the screen of equipment as video, and does not further process.

VI. many sources

As mentioned above, video can be by two of a two camera movement equipment camera, and sends another equipment in video conference to.Some embodiment can be in company with the video of a camera from two camera movement equipment together, transmission is presented at different media contents or the arbitrary content on two camera movement equipment, instead of transmits from the video of two camera of two camera movement equipment.In other words, these embodiment can in company with the video of the camera by two camera movement equipment together, send the content from multiple sources.

Figure 84 is the video conference of two camera movement equipment of some embodiment of graphic extension and the another kind of software architecture of processing module conceptually.The video conference of Figure 84 and processing module are similar to video conference and the processing module 1600 of Figure 16, except this video conference and processing module comprise display driver 8485 and memory 8475, and exchange of media module 1620 comprises that media come outside source module 8470 and screen capture module 8480.

The media of some embodiment carry out source module 8470 route media content between video conference module 8402 and memory 8475.The example of media content comprises video, image, document and music.Other embodiment is kept at the media content of other type in memory 8475.The memory 8475 of some embodiment be internal storage (for example, RAM), for example, and the memory 8475 of other embodiment is external memory storage (, compact flash (CF) card, secure digital (SD) card, etc.).

In certain embodiments, screen capture module 8480 is by display driver 8485, and route is presented at the image of the content on the display of two camera movement equipment.In certain embodiments, display driver 8485 is responsible for catching the content on display, and content is converted to image.Different embodiment catches the different content being presented on display.For example, some embodiment catch all the elements that are presented on display.Other embodiment catch display specific viewing area (for example, the viewing area of current active window, PIP show viewing area, etc.).

With reference now to Figure 84,, some exemplary operations of video conference and processing module are described.For the video transfers media content together of the camera in company with from two camera movement equipment, the video conference module 8402 of some embodiment is carried out and above in the identical operation of video conference module 1602 illustrated in fig. 16, fetch image except replacing from CIPU 1650, video conference manager 1604 carrys out source module 8470 by media, fetches outside media content from memory 8475.In order to transmit the image of the content on the display that is presented at two camera movement equipment, some embodiment of video conference manager 1604, by display driver 8485, fetch the image of the content on the display that is presented at two camera movement equipment.Some embodiment carry out (for example similarly processing with the processing that the image of fetching from CIPU 1650 is carried out to being presented at media content on display or the image of content, perspective correction, convergent-divergent etc.), and other embodiment does not carry out any processing.

Discussion has above been described in company with the video of a camera by two camera movement equipment together, sends the several examples from the content in various sources.But, other embodiment can transmit other dissimilar content.For example, in the video conference that relates to multiple participants, the video that some embodiment receive an equipment from video conference, and send another equipment to by the video of the camera of two camera movement equipment.Thereby, in company with the video by a camera of two camera movement equipment together, can send the content from any number of different types in any number of sources.

VII. multipart video-meeting

The above-mentioned each joint that relates to video conference has been described the video conference with two participants.But, be also possible by the multipart video-meeting (, three above participants) of the mobile device of some embodiment.In certain embodiments, all participants in multipart video-meeting can mutually see and hear.Other embodiment provides one in many ways to broadcast video conference, one of them participant (for example, broadcaster) can see and hear all other participants, and all other participants can see and hear this broadcaster, but other participant can not see or hear (unless for example obtaining broadcaster's approval) mutually.

A. the user interface of multipart video-meeting

During multipart video-meeting, some embodiment provide the various participants for display video meeting, and select the different UI of the one or more particular participant that will watch.For example, some embodiment of mobile device provide all participants that simultaneously show multipart video-meeting, and allow the user of mobile device to select one of participant to watch the UI of (for example,, by amplifying selected participant's image).Figure 85 illustrates an example of this UI.

Figure 85 is with reference to five different phases 8505,8510,8515,8520 and 8525 of UI 8530, be illustrated in the UI 8530 of mobile device 8500 and show all participants of multipart video-meeting simultaneously, and select one of participant so that the sequence of operations of watching.First stage 8505 is illustrated in the UI 8530 after the multipart video-meeting between other three users that set up miscellaneous equipment.As shown in the figure, UI 8530 comprises compound display 8535 and viewing area 1155.Compound display 8535 comprises four viewing areas 8565,8570,8575 and 8580 of demonstration by the image of the participant's of multipart video-meeting camera.In this example, viewing area 8565 represents user's (, viewing area 8565 shows the image by the positive camera of mobile device 8500) of mobile device 8500.Viewing area 1155 and identical in viewing area 1155 illustrated in fig. 12 above.

Second stage 8510 represents one of the viewing area in the user by selecting compound display district 8530 of mobile device 8500, starts participant and selects operation.Especially, second stage 8510 represents that user selects viewing area 870 (for example,, by dub finger 8550 on viewing area 8570).

The phase III 8515 of UI 8530 has been illustrated in participant and has selected operation compound display 8555 afterwards.Some embodiment provide the animation (not shown) that shows second stage 8510 and the transformation between the phase III 8515.Compound display 8555 comprises that PIP shows 8560, the participant's who selects in second stage 8510 viewing area (PIP shows 8560, viewing area 8570) be shown as background viewing area, and user's viewing area 8565 is shown as to the insertion viewing area of PIP demonstration 8560.In this example, PIP shows that 8560 show the image that is stretched horizontally the selected viewing area 8570 that is applicable to transversal orientation.In certain embodiments, this image is not stretched, and selects the image of viewing area to keep its portrait orientation (, as shown in Figure 36, being filled in the redundant space of both sides, background viewing area by secret note).In addition, compound display 8555 also comprises compound display 8585, and compound display 8585 shows the image that the ratio of two unselected viewing areas 8575 and 8580 is dwindled.

Fourth stage 8520 represents that the user by selecting PIP of mobile device 8500 showed for 8560 (for example, shown on 8560 and dubbed and point 8550 by PIP), started participant and cancelled selection operation.Five-stage 8525 is illustrated in participant to be cancelled and selects operate compound display 8535 afterwards.

Figure 85 represents to show all participants of multipart video-meeting simultaneously, carries out participant and selects operation and carry out participant and cancel the exemplary operations sequence of selecting operation.Other sequence of operation is also possible.For example, after the phase III 8515, replace beginning participant to cancel and select operation, user can select to be presented at and not select one of viewing area in compound display 8585, show 8560 background viewing area (viewing area of, previously having selected) with newly select viewing area and PIP in exchange viewing area 8585.Thereby during multipart video-meeting, viewing area and PIP that user can exchange in viewing area 8585 at any time and arbitrarily in multiple times show 8560 background viewing area.In addition, when any during multipart video-meeting, user can carry out participant and cancel and select operation, to return to compound display 8535.In addition, different embodiment allows user according to different modes, such as the switch switching on mobile device 8500, by sending voice commands etc., selects specific participant.

Some embodiment provide the technology of for example automatically selecting participant according to speech detection.In such embodiments, in the time that one of participant speaks, this participant's viewing area is automatically elected as PIP and is shown 8560 background viewing area.In the time that a different participant speaks, this participant's viewing area is automatically elected as PIP and is shown 8560 background viewing area.In certain embodiments, in the time that the participant of multipart video-meeting does not speak, ormal weight mourn in silence (for example, 3 seconds) afterwards, display show compound display 8535.In certain embodiments, in the time that the user of mobile device 8500 speaks, on the UI 8530 of mobile device 8500, any operation does not also occur.

Figure 86 graphic extension shows all participants of multipart video-meeting simultaneously and selects one of participant so that the sequence of operation of another example of watching.Figure 86 is with reference to seven different phases 8505,8605,8610,8615,8620,8625 and 8630 of UI 8645, this operation in the UI 8645 of graphic extension mobile device 850.The first stage 8505 of first stage 8505 and graphic extension in Figure 85 is identical, because it is presented at the UI 8645 after multipart video-meeting between other three users that set up miscellaneous equipment.

Second stage 8605 graphic extensions for example, by selecting viewing area 8570 (, by two fingers are placed on viewing area 8570), and the user of mobile device 8500 starts participant and selects operation.Phase III 8610 represents that participant selects the conversion stage of operation.In this stage, user drags two fingers away from each other, makes viewing area 8570 become larger simultaneously, and is full of the viewing area for showing compound display 835.This example represents that viewing area 8570 is selected, but other viewing area 8565,8575 and 8580 also can be selected arbitrarily.In certain embodiments, do not allow the user of mobile device 8500 to select this user's viewing area (, the viewing area 8565 in this example).

The fourth stage 8615 of UI 8645 has been presented at participant and has selected the PIP of operation UI8645 afterwards to show 8635.Some embodiment require user to continue to drag away from each other two fingers, until being full of PIP, viewing area 8570 shows 8635 background viewing area 8640, and other embodiment only required before user removes finger, user's drag operation (is for example greater than specific threshold quantity, be greater than specific distance, or be greater than specific time quantum).In the time that user's drag operation meets or exceedes specific threshold amount, UI 8645 continues the amplification of viewing area 8570, shows 8635 background viewing area 8640 until it is full of PIP.Otherwise participant selects to have operated, UI 8645 returns to compound display 8535.As shown in the figure, the viewing area of selection (, viewing area 8570) is that PIP shows 8635 background viewing area 8640, and user's viewing area 8565 is that PIP shows 8635 insertion viewing area.Some embodiment provide the animation (not shown) that shows the transformation between phase III 8610 and fourth stage 8615.

Five-stage 8620 graphic extensions for example, by selecting PIP to show 8635 background viewing area 8640 (, by two fingers are placed on background viewing area 8640), and the user of mobile device 8500 starts participant and cancels selection operation.The 6th stage 8625 represented that participant cancelled the conversion stage of selecting operation.This stage graphic extension user floor mop close to each other starts to refer to, to dwindle the viewing area of the background viewing area 8640 that is used to PIP demonstration 8635.Be similar to the operation of explanation in the phase III 8610, some embodiment required before user removes finger, and user's drag operation is greater than specific threshold amount (for example, be greater than specific distance, or be greater than specific time quantum).Otherwise participant cancels selection and operated, UI 8645 returns to PIP and shows 8635.The 7th stage 8630 of UI 8645 has been illustrated in participant and has cancelled selection operation compound display 8535 afterwards.

Figure 86 graphic extension shows all participants of multipart video-meeting simultaneously, carries out participant and selects operation, and carry out participant and cancel another exemplary operations sequence of selecting operation.But, some embodiment allow the user of mobile device 8500 repeat participant select operation and participant cancel select operate.Such embodiment of Figure 87 graphic extension.

Specifically, Figure 87 is with reference to seven different phases 8505,8705,8615,8710,8715,8720 and 8725 of UI 8730, and graphic extension is repeatedly carried out participant and selected operation and participant to cancel the exemplary sequence of selecting operation.The first stage 8505 of first stage 8505 and above mentioned Figure 85 and 86 is identical.Second stage 8705 is similar to the second stage 8605 of Figure 86, except user is by dubbing once (instead of two fingers are placed on viewing area 8570) of viewing area 8570, selects outside viewing area 8570.The fourth stage 8615 of phase III 8615 and Figure 86 is identical, selects operation PIP afterwards to show 8635 because it has been illustrated in participant.Fourth stage 8710 is similar to the five-stage 8620 of Figure 86, except user is by dubbing once (instead of two fingers are placed on background viewing area 8640) of background viewing area 8640, selects outside the background viewing area 8640 of PIP demonstration 8645.

The 7th stage 8630 of five-stage 8715 and Figure 86 is identical, completes cancellation selection operation compound display 8535 afterwards because it is illustrated in participant.The 6th stage 8720 was similar to second stage 8510, selected operation except viewing area 8575 being carried out to participant.Similarly, the 7th stage 8725 was similar to the phase III 8705, showed 8635 background viewing area 8640 because it is expressed as PIP the viewing area (, viewing area 8575) of selecting.Although Figure 87 has only represented that a little participant selects and participant cancels selection operation, but during multipart video-meeting, can carry out any this operation repeatedly.

In addition, some embodiment provide during video conference, can show the participant's of different numbers UI.For example, in the time that mobile device is uprightly held (, portrait orientation), the UI of some embodiment only shows some participants of multipart video-meeting, in the time that mobile device is laterally held (, transversal orientation), shows other participant.When other embodiment is laterally held at mobile device, show all participants.In addition, some embodiment provide animation indicate and Figure 34,35,36 and 37 in transformation between diverse location and/or the orientation of similar mobile device of graphic extension.Other different animation is also possible.

As during video conference, show another example of the participant's of different numbers UI, some embodiment allow the user of mobile device during video conference, to select the multidigit participant that will simultaneously watch.For the purpose of illustrating, referring to the first stage 8505 of Figure 85, some such embodiment allow the user of mobile device 8500 to select two or more viewing areas (for example,, by dubbing the corresponding viewing area in compound display 8535) in viewing area 8565,8570,8575 and 8580.The viewing area of selecting subsequently can be shown according to variety of way, any one in arranging of the demonstration of graphic extension such as compound display, PIP show, in Figure 65, and other various multi-players show and arrange.In addition,, although the example of some embodiment has been described, but, those of ordinary skill will appreciate that different embodiment can, according to any various ways, select and show the multidigit participant of multipart video-meeting.

B. in many ways broadcast the user interface of video conference

As mentioned above, in many ways broadcast video conference and only allow a participant to hear and see all other participants, and other participant can not mutually hear or see.Easier in order to make in many ways to broadcast video conference, some embodiment are provided for showing and in many ways broadcast the broadcaster of video conference and other participant's numerous different UI.For example, some embodiment provide student-teacher formula UI layout, are similar to the layout of the phase III 8515 of graphic extension in Figure 85.Thereby, with reference now to the described phase III 8515, student-teacher formula UI layout of some embodiment is described.

In these embodiments, only having broadcaster to be displayed on PIP shows in 8560 whole viewing area and (, does not show and insert viewing area).In many ways other participant who broadcasts video conference is displayed on PIP and shows below 8560, is similar to the viewing area showing in compound display 8585.With described similar above, in certain embodiments, when mobile device is during in vertical pattern, in compound display 8585, show other participant of defined amount, and when mobile device is during in transverse mode, in compound display 8585, can show other or all participants.In addition, other embodiment provides broadcaster that demonstration broadcasts video conference in many ways and other participant's different UI.

C. control the audio frequency of multipart video-meeting

In addition, the mobile device of some embodiment provides the different technologies of the multiple participants' that control multipart video-meeting audio frequency.For example, the user that some embodiment of mobile device allow mobile device for example,, by being presented at the single group volume control (, volume sliding shoe) on the UI of this embodiment, controls the audio frequency of every participant in multipart video-meeting.In other embodiments, mobile device allows the user of mobile device by one group of volume control, such as the volume sliding shoe being presented in each participant's viewing area, controls respectively the volume of the audio frequency of every participant in multipart video-meeting.Some embodiment only provide mute button, instead of one group of volume control.Thereby, in some such embodiment, the user of mobile device is merely able to all participants quiet or that cancel quiet multipart video-meeting, and in other this embodiment, the user of mobile device can distinguish each participant quiet or that cancel quiet multipart video-meeting.In addition, other technology of controlling the participant's of multipart video-meeting audio frequency is also possible, such as the switch by switching mobile device, by sending voice commands etc.

VIII. electronic system

Many above-mentioned features and application are realized as software process, and described software process is defined as the one group of instruction being recorded on computer-readable recording medium (also referred to as computer-readable medium).In the time that these instructions for example, are carried out by one or more processing units (, one or more processors, the core core of processor, or other processing unit), they make processing unit carry out the action of indicating in instruction.The example of computer-readable medium includes but not limited to CD-ROM, flash drive, RAM chip, hard disk drive, EPROM etc.Computer-readable medium does not comprise carrier wave wireless or that transmit by wired connection and the signal of telecommunication.

In this manual, term " software " intention comprises the firmware residing in read-only memory, or is kept at the application in magnetic storage, and they can be read in memory, so that by processor processing.In addition, in certain embodiments, multiple software inventions can be realized as the more subdivision of large program, remain different software inventions simultaneously.In certain embodiments, multiple software inventions also can be realized as independently program.Finally, together realize software described herein invention separable programming any combination also within the scope of the invention.In certain embodiments, when being installed to while operating in one or more electronic systems, software program definition is carried out and is realized the one or more specific machine of the operation of software program and realizes.

Caller code by one or more interfaces and the mutual environment of invoked other program code in, some embodiment are realized as the software process that comprises one or more API (API).Various function calls, message or other various calling that further can comprise various parameters can be passed through API, between caller and called code, transmit.In addition, API can provide definition the data type realizing in invoked program code or the ability of classification in API of using to caller code.

At least some embodiment comprise and call component software by API and the mutual environment of invoked component software.A kind of method operating by API in this environment comprises by API, transmits one or more function calls, message, other various calling or parameter.

In certain embodiments, can use one or more API (API).For example, some embodiment of exchange of media module 310 (or 910) provide one group of API to other component software, for accessing at Fig. 3 and 9 various Video processing and the encoding functions of describing, such as the function of the TNR module 1900 of describing in Figure 19.

API is the interface of being realized by program code assembly or nextport hardware component NextPort (being called " API realizes assembly " below), allows different program code assemblies or nextport hardware component NextPort (being called " API Calls assembly " below) access and use to realize by API one or more functions, method, process, data structure, classification and/or other service that assembly provides.AIP can be defined in API Calls assembly and AIP realizes the one or more parameters that transmit between assembly.

API allows developer's (can be third party developer) of API Calls assembly to utilize and realizes by API the regulation feature that assembly provides.Can there is an API Calls assembly, or can have more than one API Calls assembly.API can be computer system or the source code interface of program library for supporting that the service request of self-application provides.Operating system (OS) can have multiple API, with the one or more such API of application call that allow to move on OS, service (such as program library) can have multiple API, to allow to use the one or more such API of application call of this service.Can utilize in the time setting up application, the programming language that can be explained or compile is specified API.

In certain embodiments, API realizes assembly can provide more than one API, and each API provides access to be realized the different views of the different situations of the different aspect of the function of assembly realization by API, or has the different views of described different situations.For example, the API that API realizes assembly can provide first group of function, and can be exposed to third party developer, another API that API realizes assembly can be hidden (not being exposed), and provide the subset of first group of function, also provide another group function, such as test or the debug function in first group of function not.In other embodiments, API realizes assembly itself can pass through one or more other assemblies of API Calls of lower floor, thereby is API Calls assembly, is again that API realizes assembly.

When API defines when access and utilizes API to realize the specific characteristic of assembly, language and parameter that API Calls assembly uses.For example, one or more API Calls that API Calls assembly exposes by API or enable (for example specializing with function or method call), access API realizes the specific characteristic of assembly, and via API Calls or enable, utilizes parameter to transmit data and control information.API realizes assembly can respond the API Calls from API Calls assembly, returns to numerical value by API.For example, although grammer and the result (, how to enable API Calls, and what API Calls doing) of API definition API Calls, call how to complete the function of being specified by API Calls but API can not represent AIP.Each API Calls is that the one or more API that realize between assembly by called side (API Calls assembly) and API transmit.Transmitting API Calls can comprise and send, just open, enable, call, receive, return or response function calls or message; In other words, transmission can be described API Calls assembly or API and realize the action of assembly.The function call of API or other are enabled and can be passed through parameter list or other structure, send or receive one or more parameters.Parameter can be the pointer of constant, key (key), data structure, object, object type, variable, data type, pointer, array, list or function or method or quote data or other modes of another object that will transmit by API.

In addition, data type or classification can be provided by API, and realize assembly by API and realize.Thereby the definition providing in API can be provided API Calls assembly, explanatory variable, the pointer that uses this type or classification, use or illustrate this type or the constant of classification.

Conventionally, API can be used to access and realize by API service or the data that assembly provides, or initiates to realize operation that assembly provides or the execution of calculating by API.For example, it can be operating system, program library, device driver, API, application program or other module one of any (should understand that API realizes assembly and API Calls assembly can be the module of same type, or the module that differs from one another of type) that API realizes assembly and API Calls assembly.In some cases, can carry out imbody API with firmware, microcode or other hardware logic at least partly and realize assembly.In certain embodiments, API can allow client-side program that the service being provided by software development kit (SDK) storehouse is provided.In other embodiments, the API being provided by application architecture can be provided for application or other client-side program.In these embodiments, calling of function to being provided by SDK and being provided by API or method can be provided for application or client-side program, or the data type or the object that use in SDK definition and provided by API.In these embodiments, application architecture can be program the main event loop of response by the variety of event of described architecture definition is provided.API allows applications exploiting application architecture, allocate event and the response to event.In some implementations, API Calls can be to the ability of application report hardware device or state, comprise the ability relevant to various aspects or state, such as input capability and state, fan-out capability and state, disposal ability, power supply status, memory capacity and state, communication capacity etc., and other the rudimentary logic realization that can partly move on nextport hardware component NextPort by firmware, microcode or part of API.

API Calls assembly can be local component (, realize assembly in identical data handling system with API), or realize the remote component (, realizing assembly in different data handling systems from API) of component communication through API and API by network.Should understand that API realizes assembly and also can serve as API Calls assembly (, it can carry out API tune to the AIP that is realized assembly exposure by different API), be exposed to the API of different API Calls assemblies by realization, API Calls assembly also can serve as API and realize assembly.

API also allows the multiple API Calls assemblies and the API that write with different programming languages to realize component communication (thereby API can be included in API and realize the feature that between assembly and API Calls assembly, conversion is called and replied); But, can utilize specific programming language to realize API.In one embodiment, API Calls assembly is adjustable uses the API from different providers, such as one group of API from OS provider, from another group API of plug-in unit provider with for example, from another group API of the founder of another provider (, the provider of software library) or another group API.

Figure 88 is the block diagram that graphic extension can be used example A PI architecture in some embodiments of the invention.As shown in Figure 88, API architecture 8800 comprises that the API that realizes API 8820 realizes assembly 8810 (for example, operating system, program library, device driver, API, application program, software or other module).API 8820 specifies one or more functions, method, classification, object, agreement, data structure, form and/or the API that can be used by API Calls assembly 8830 to realize the further feature of assembly.API 8820 can specify at least one calling convention, and described calling convention specifies how the function that API realizes in assembly 8810 receives the parameter from API Calls assembly 8830, and how this function returns to API Calls assembly result.API Calls assembly 8830 (for example, operating system, program library, device driver, API, application, software or other module) send API Calls by API 8820, to access and to utilize the API being specified by API 8820 to realize the feature of assembly 8810.API realizes assembly 8810 can respond API Calls, returns to numerical value by API 8820 to API Calls assembly 830.

Recognize that API realizes assembly 8810 and can comprise by API 8820 and specifying and can not be for other function, method, classification, data structure and/or the further feature of API Calls assembly 8830.Should understand that API Calls assembly 8830 can realize assembly 8810 in identical system with API, or can be positioned at remote location, and access API by network utilisation API 8820 and realize assembly 8810.Although the single API Calls assembly 8830 that Figure 88 graphic extension and API 8820 are mutual, but should understand that other API Calls assembly can use API8820, described other API Calls assembly can use the language different from API Calls assembly 8830 (or with identical language) to write.

API realizes assembly 8810, API 8820 and API Calls assembly 8830 and can be stored in machine readable media, and described machine readable media for example comprises, with the readable form of machine (computer or other data handling system) protects stored any mechanism.For example, machine readable media comprises disk, CD, random access memory, read-only memory, flash memory device etc.

Figure 89 is an example of two camera movement computing equipment architectures 8900.The realization of mobile computing device can comprise one or more processing units 8905, memory interface 8910 and Peripheral Interface 8915.These assemblies that form computing equipment architecture can be independent assemblies, or are integrated in one or more integrated circuits.These each assemblies also available one or more communication bus or holding wire are coupled in together.

Peripheral Interface 8915 can with various transducers and subsystem, comprise that camera subsystem 8920, radio communication subsystem 8925, audio subsystem 8930, I/O subsystem 8935 etc. couple.Peripheral Interface 8915 can be realized the communication between processor and ancillary equipment.Ancillary equipment such as orientation sensor 8945 or acceleration transducer 8950 can couple with Peripheral Interface 8915, with convenient directed and acceleration function.

Camera subsystem 8920 can with one or more optical pickocffs 8940, for example charge coupled device (CCD) optical pickocff, complementary metal oxide semiconductors (CMOS) (CMOS) optical pickocff couple.The camera subsystem 8920 coupling with transducer can convenient camera-enabled, such as image and/or video data capture.Radio communication subsystem 8925 can be used for convenient communication function.Radio communication subsystem 8925 can comprise radio frequency receiver and reflector, and optical receiver and reflector.They can be realized as by one or more communication networks, such as work such as GSM network, Wi-Fi network, blueteeth networks.Audio subsystem 8930 couples with loud speaker and microphone, allows the function of speech, such as speech recognition, digital record etc. with facility.

I/O subsystem 8935 relates between I/O ancillary equipment such as display, touch-screen and the data/address bus of CPU by the transmission of Peripheral Interface.I/O subsystem 8935 can comprise touch screen controller 8955 and other input control device 8960, with convenient these functions.Touch screen controller 8955 can couple with touch-screen 8965, and utilizes any one in multiple touch-sensitive technology, detects contact and movement on screen.Other input control device 8960 can with other input-output apparatus, such as one or more buttons couple.

Memory interface 8910 can couple with memory 8970, and memory 8970 can comprise high-speed random access memory and/or nonvolatile memory, such as flash memory.Memory can be preserved operating system (OS) 8972.OS 8972 can comprise the instruction of processing basic system services and carrying out hardware dependent tasks.

Memory also can comprise to be made and the easier communication instruction 8974 of communicating by letter of one or more other equipment; The graphic user interface instruction 8976 of convenient graphic user interface processing; Facility relates to the processing of image/video and the image/video processing instruction 8978 of function; Facility relates to the processing of phone and the telephone order of function 8980; The convenient processing relevant with processing to media communication and exchange of media and the processing instruction 8982 of function; Facility relates to the processing of camera and the camera instruction 8984 of function; Video conference instruction 8986 with convenient video conference processing and function.Above-mentioned instruction does not need to be realized as independently software program or module.Available hardware and/or software, comprise the various functions that realize mobile computing device with one or more signals processing and/or application-specific integrated circuit (ASIC).

Above-described embodiment can comprise receiving and touches input, to pass through wired or wireless communication passage 9002 as Figure 90 as shown in, with the mutual touch I/O equipment 9001 of computing system 9003.Touch I/O equipment 9001 can be used to replace or in conjunction with other input equipment, such as keyboard, mouse etc., user's input be offered to computing system 9003.One or more touch I/O equipment 9001 can be used for user's input to offer computing system 9003.Touching I/O equipment 9001 can be the part (for example, the touch-screen of laptop computer) of computing system 9003, or is independent of computing system 9003.

Touch I/O equipment 9001 and can comprise transparent, translucent, opaque touch sensitive panel wholly or in part, or their combination in any.Touch that I/O equipment 9001 can be embodied as touch-screen, touch pad, the touch-screen that plays touch pad effect (for example, replace the touch-screen of touch pad of laptop computer), be combined with any other input equipment or the touch-screen or the touch pad (for example, being placed in touch-screen or the touch pad on keyboard) that merge or there is any multidimensional object for receiving the touch-sensitive surface that touch inputs.

In an example, the touch I/O equipment 9001 that is embodied as touch-screen can comprise the transparent and/or translucent touch sensitive panel on the display that is partly or entirely placed at least a portion.According to this embodiment, touch I/O equipment 9001 for showing the graph data transmitting from computing system 9003 (and/or another source), and for receiving user's input.In other embodiments, touch I/O equipment 9001 and can be embodied as the integrated touch screen that wherein makes touch-sensitive assembly/equipment be combined with display module/equipment.In other embodiment, touch-screen can be used as showing assistant images data, or the auxiliary or additional display of the graph data identical with basic display unit and reception touch input.

Touching I/O equipment 9001 can be configured to according to electric capacity, resistance, optics, acoustics, induction, machinery or chemical measurement result, or touch or approach about the one or many of nearby device 9001 the measurable any phenomenon of generation touching, the one or many on checkout equipment 9001 touches or approaches the position touching.Software, hardware, firmware or their combination in any can be used to process the measurement result of the touch detecting, to identify and to follow the tracks of one or more gestures.Gesture can touch or approach touch corresponding to the fixing or revocable one or many touching on I/O equipment 9001.By substantially side by side, incessantly or one after the other according to predetermined way,, pressure directed such as dubbing, press, swing, wipe, twist, changing changes to be pressed etc., touching mobile one or more finger or other object on I/O equipment 9001, can make gesture.Gesture can be with between finger, or the pinching of any other one or more finger, slide, hit, rotate, bending, drag or dub action and characterize (but being not limited to this).Single gesture can be with a hand or two hands, realized by one or more users or their combination in any.

Computing system 9003 can be used graph data driving display, with display graphics user interface (GUI).GUI can be configured to receive touch input by touching I/O equipment 9001.The touch I/O equipment 9001 that is embodied as touch-screen can show GUI.On the other hand, GUI can be displayed on and touch on the display that I/O equipment 9001 separates.GUI can comprise the graphic elements that is presented at the ad-hoc location in interface.Graphic elements can include but not limited to the virtual input device of various demonstrations, comprises virtual scroll wheel, dummy keyboard, virtual knob, virtual push button, any virtual UI etc.User can do gesture at the touch I/O equipment one or more ad-hoc locations associated with graphic elements GUI 9001.In other embodiments, user can do gesture with location-independent one or more positions of the graphic elements of GUI.Can handle directly or indirectly, control, revise, move, encourage, start or usually affect the graphic elements in GUI touching the gesture of making on I/O equipment 9001, such as cursor, icon, media file, list, text, all or part of image etc.For example, with regard to touch-screen, user can be by doing gesture on the graphic elements on touch-screen, directly and graphic elements mutual.On the other hand, touch pad generally provides indirectly mutual.Gesture also affects the GUI element (for example, causing user interface to occur) not showing, or can affect other action (for example, affecting state or the pattern of GUI, application or operating system) in computing system 9003.Can, in conjunction with the cursor showing, on touch I/O equipment 9001, do or not do gesture.For example, just on touch pad, do the situation of gesture, cursor (or designator) can be displayed on display screen or touch-screen, by the touch input controllable cursor on touch pad, with display screen on Drawing Object mutual.On touch-screen, be directly therein in other embodiment of gesture, the in the situation that user can or not being presented on touch-screen in cursor or designator demonstration, the object interaction directly and on touch-screen.

Respond or touch according to touching the touch on I/O equipment 9001 or approaching, can provide feedback to user by communication port 9002.Feedback can be changeably or immutablely by modes such as optics, machinery, electric, sense of smell, acoustics, or their combination in any transmits.

Above-mentioned these function available digital electronic circuits, realize with computer software, firmware or hardware.Various technology can utilize one or more computer programs to realize.Programmable processor and computer can be included in or be packaged into mobile device.Processing and logic flow can be carried out by one or more programmable processors with by one or more Programmable Logic Device.General and dedicated computing equipment and memory device can pass through interconnection of telecommunication network.

Some embodiment comprise computer program instructions are kept to the electronic building brick in machine readable or computer-readable medium (calling on the other hand computer-readable recording medium, machine readable media or machinable medium), such as microprocessor, memory and memory storage.The example of this computer-readable medium (for example comprises RAM, ROM, read-only optical disc (CD-ROM), CD-R (CD-R), CD-RW (CD-RW), read-only digital versatile disc, DVD-ROM, DVD-dual layer-ROM), various can record/rewritable DVD (for example, DVD-RAM, DVD-RW, DVD+RW etc.), flash memory (such as SD card, small-sized SD card, miniature SD card etc.), magnetic and/or solid-state hard drive, read-only and can record blue light cD, super-high density CD, other optics or magnetic medium and floppy disk arbitrarily.Computer-readable medium can be preserved the computer program that can be carried out by least one processing unit, comprises the many groups instruction for realizing various operations.The example of computer program or computer code comprises machine code, such as the machine code being produced by compiler, and comprises by computer, electronic building brick or the file of high-level code that utilizes the microprocessor of interpreter to carry out.

Although discussion above relates generally to microprocessor or the multi-core processor of executive software, but some embodiment are by one or more integrated circuits, such as application-specific integrated circuit (ASIC) (ASIC) or field programmable gate array (FPGA) execution.In certain embodiments, such integrated circuit is carried out and is kept at originally instruction with it of circuit.

The term " computer ", " server ", " processor " and " memory " that in any claim of specification and the application, use all refer to electronic equipment or other technical equipment.People or crowd got rid of in these terms.Concerning specification, term " demonstration " meaning shows on electronic equipment.The term " computer-readable medium " using in any claim of specification and the application is confined to protect stored tangible entity with computer-readable form completely.Any wireless signal, wired download signal and any other instant signal got rid of in these terms.

Figure 91 conceptually graphic extension according to some embodiment, for connecting some participants' the example communication system 9100 of video conference.As shown in the figure, communication system 9100 comprises some mobile devices 9115, some cellular basestations (Node B in other words) 9110, some radio network controllers (RNC) 9105, and core network 9125.Cellular basestation and RNC are called Universal Mobile Telecommunications System (UMTS) grounding wireless access network network (UTRAN) 9130 by collective.Each RNC9105 with together be called as Radio Access Network (RAN) one or more cellular basestations 9110 connect.

Each cellular basestation 9110 covers a service area 9120.As shown in the figure, the mobile device 9115 in each service area is by Uu interface, with service cellular basestation 9110 wireless connections of service area 9120.Uu interface uses the dihedral protocol stack of tool: control plane and user plane.User plane support circuit-switched, packet switching and broadcast data stream.Control plane carries the signaling message of network.

Each cellular basestation is connected with RNC by Iub interface.Each RNC 9105 is connected with core network 9125 with Iu-ps interface by Iu-cs.Iu-cs interface for example, for circuit switched service (, speech), and Iu-ps interface for example, for packet-switched services (, data).Iur interface is for linking together two RNC.

Therefore, communication system 9100 support circuit-switched service and packet-switched services.For example, circuit switched service allows to transmit communicating data (for example, speech) by the circuit switching equipment via communication system 9100, converses.Packet-switched services allows, by utilizing the transportation protocol layer (such as UDP or TCP) on Internet protocol layer (such as IP), to transmit videoconference data via the PSE of communication system 9100, carries out video conference.In certain embodiments, before, in video conference, the call-video conference illustrating in chapters and sections is set and changes circuit switched service and the packet-switched services that (for example, switching) utilizes the communication system such as communication system 9100 to support.That is, in such embodiments, call is to be undertaken by the circuit switching equipment of communication system 9100, and video conference is to be undertaken by the PSE of communication system 9100.

Although the example communication system in Figure 91 illustrates the third generation (3G) technology UTRAN mobile radio system, but should note in certain embodiments, the second generation (2G) communication system, other 3G communication system, such as 3GPP2 Evolution-Data Optimized or evolution-just data (EV-DO) and third generation partner program 2 (3GPP2) code division multiple access access 1X (CDMA1X), the 4th generation (4G) communication system, WLAN (wireless local area network) (WLAN), can be used to be connected some participants of meeting with micro-wave access global inter communication (WiMAX) communication system.The example of 2G system comprises global system for mobile communications (GSM), general packet radio service (GPRS) and enhancing data rate GSM evolution (EDGE).2G communication system architecture is similar to the architecture shown in Figure 91, replace Node B 9110, and base station controller (BSC) replaces outside RNC 9105 except 2G communication system architecture uses base station transceiver (BTS).In 2G communication system, the A interface between BSC and core network is for circuit switched service, and Gb Interface between BSC and core network is for packet-switched services.

In certain embodiments, communication system 9100 is by supplying at first mobile device 9115 to allow mobile device 9115 to utilize the service provider of communication system 9100 to move.Some embodiment, by configuration and registered user's identification module (SIM) in mobile device 9115, provide mobile device 9115.In other embodiments, change the memory configuration and the registration mobile device 9115 that utilize mobile device 9115 into.In addition, can provide other service (after client buys mobile device 9115), such as the data, services of similar GPRS, Multimedia Message sending and receiving services (MMS) and instant messaging.Once be supplied, mobile device 9115 activates with regard to serviced operator, thereby allows to use communication system 9100.

In certain embodiments, communication system 9100 is dedicated communications networks.In such embodiments, mobile device 9115 can for example, for example, in (mobile device 9115, providing for communication system 9100) communication (, carrying out voice call, swap data) each other.In other embodiments, communication system 9100 is public communication networks.Thereby except the mobile device 9115 providing for communication system 9110, mobile device 9115 can also be communicated by letter with the miscellaneous equipment outside communication system 9100.Some in miscellaneous equipment outside communication system 9100 comprise by other network, such as public switch telephone network or another kind of cordless communication network, and the telephone set, computer and the miscellaneous equipment that are connected with communication system 9100.

Long Term Evolution (LTE) specification is used for defining 4G communication system.Figure 92 conceptually graphic extension in certain embodiments, for connecting some participants' the example of 4G communication system 9200 of video conference.As shown in the figure, communication system 9200 comprises some mobile devices 9115, some evolved Node B (eNB) 9205, mobile management entity (MME) 9215, gateway (S-GW) 9220, packet data network (PDN) gateway 9225 and home subscriber server (HSS) 9235.In certain embodiments, communication system 9200 comprises one or more MME 9215, one or more S-GW 9220, one or more PDN Gateway 9225 and one or more HSS 9235.

ENB 9205 provides air interface for mobile device 9115.As shown in the figure, each eNB9205 covers service area 9210.Mobile device equipment 9115 in each service area 9210 is by LTE-Uu interface, with eNB 9205 wireless connections of service area 9210.Figure 92 also represents that eNB 9205 interconnects by X2 interface.In addition, eNB 9205 is connected with MME 9215 by S1-MME interface, is connected with S-GW 9220 by S1-U interface.ENB 9205 collectives are called as the UTRAN (E-TRAN) 9230 of evolution.

ENB 9205 provides various functions, for example, such as provided for radio resources management (, radio bearer control, the control of connection mobility etc.), user plane data is towards the route of S-GW 9220, signal measurement and measurement report, the MME in the time that mobile device connects selects, etc.The function of MME9215 comprises idle pulley mobile device tracking and paging, the enabling and stopping using, the selection of the S-GW 9220 in the time that mobile device connects, non access stratum (NAS) signaling termination of radio bearer, by with the mutual subscription authentication of HSS 9235, etc.

The function of S-GW 9220 comprises (1) route and forwards user data packets, and (2) management and preservation mobile device context, such as parameter and the network internal routing iinformation of IP carrying service.The function of PDN Gateway 9225 comprises exit point and the entrance of the traffic by becoming mobile device, and the connectedness from mobile device to external packet data network (not shown) is provided.Connectedness when mobile radio station can have more than one PDN Gateway, to access multiple packet data network.PDN Gateway 9225 also serves as 3GPP and non-3GPP technology, for example, such as the ambulant anchor point (anchor) between WiMAXt 3GPP2 (, CDMA 1X and EV-DO).

As shown in the figure, MME 9215 is connected with S-GW 9220 by S11 interface, is connected with HSS 9235 by S6a interface.S-GW 9220 is connected by S8 interface with PDN Gateway 9220.MME 9215, S-GW 9220 and PDN Gateway 9225 collectives are called as block core evolution (EPC).EPC is the primary clustering of system architecture evolution (SAE) architecture, and system architecture evolution (SAE) architecture is the Core Network Architecture of 3GPP LTE wireless communication standard.EPC is pure grouping system.For example, EPC does not have speech media gateway.Service such as speech and SMS is packet switching route, and is provided by the application function that utilizes EPC service.So utilize the call-video conference illustrating to change as an example above, in certain embodiments, call and video conference are all to be undertaken by the PSE of communication system 9200.In some such embodiment, after end of conversation, continue to be used to the voice data of video conference for the packet switched channels of conversing.But, in other this embodiment, create different packet switched channels (for example, in the time setting up video conference), by the packet switched channels of new establishment instead of when the end of conversation, utilize the packet switched channels of call to transmit voice data.

The quantity of the bandwidth that in addition, these different technologies provide exceedes 10 mbit (Mbps) from 44 kilobits/second (kbps) of GPRS to LTE.Following concerning LTE, expectation downloading rate is 100Mbps, and uploading rate is 50Mbps.

Although the present invention has been described about numerous details, but those of ordinary skill in the art will appreciate that imbody the present invention in other specific forms, and does not depart from spirit of the present invention.In addition, numerous accompanying drawings illustrates various processing conceptually.Can carry out not according to the strict order of described expression and description the concrete operations of these processing.In continuous sequence of operations, specific operation can be do not carried out, and different specific operations can be in different embodiment, carried out.In addition, processing can utilize a few height processing to realize, or is realized as a part for larger grand processing.

In addition, about the video conference between two camera movement equipment, many embodiment have been described above.But, those of ordinary skill in the art will appreciate that many embodiment in these embodiments can be used on and relates to two camera movement equipment and another equipment, such as single camera movement equipment, computer, have in the situation of the video conference between the telephone set etc. of video conference capabilities.In addition,, at single camera movement equipment and having in other computing equipment of video conference capabilities, can use many embodiment described above.Thereby those of ordinary skill in the art understands the present invention and not limited by example details above, but is limited by the claim of adding.

Claims

1. a method for operation the first mobile device, described method comprises:

During the voice-frequency telephony by cordless communication network and the second equipment, on the first mobile device, present the selectable user interface UI project for be switched to the video conference between the first mobile device and the second equipment from voice-frequency telephony;

After receiving the selection of described optional UI project, show the first video of being taken by the first mobile device; With

Allowing before the first mobile device and the second equipment presents the Voice & Video data that exchange by video conference on the first mobile device by (1), the animation that stops voice-frequency telephony and the switching of (2) display of visually ground instruction from voice-frequency telephony to video conference is initiated video conference, wherein, described animation (i) starts from reducing the size of the first video in the first video and (ii) ending to show that the first video overlay is at least a portion of the second video of being taken by the second equipment showing.

2. in accordance with the method for claim 1, wherein, during presenting optional UI project and being included in voice-frequency telephony, in the viewing area on the display screen that can be presented at the first mobile device, continue to present described optional UI project.

3. also comprise in accordance with the method for claim 1:

In waiting for that the second video arrives the first mobile device, show the first video;

When in the time that the first mobile device has received the second video, show described animation, so that on present the second videometer the first mobile device.

4. in accordance with the method for claim 3, wherein, in reducing the first video big or small, the first video overlay is on the second video.

5. in accordance with the method for claim 3, wherein, once described animation finishes, the first video and the second video just form picture-in-picture and show, wherein, the first video serves as the insertion picture overlapping on the second video.

6. in accordance with the method for claim 1, wherein, allowing before the first mobile device and the second equipment exchange audio frequency and video data by video conference termination voice-frequency telephony.

7. in accordance with the method for claim 1, wherein, after the first mobile device and the second equipment start to exchange audio frequency and video data by video conference, but before the first mobile device and the second equipment start to present the Voice & Video data of exchange, stop voice-frequency telephony.

8. in accordance with the method for claim 1, also comprise the communication port with the second equipment from the first mobile device foundation, to set up the voice-frequency telephony between the first mobile device and the second equipment.

9. in accordance with the method for claim 1, wherein, the first mobile device be with cordless communication network in the mobile communication equipment of another devices communicating.

10. in accordance with the method for claim 1, wherein, the first mobile device is smart phone.

11. in accordance with the method for claim 1, and wherein, the first mobile device provides together with described cordless communication network with the second equipment, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

12. in accordance with the method for claim 1, wherein, the first mobile device provides together with described cordless communication network, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment, and the second equipment is not provided for carrying out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

13. in accordance with the method for claim 12, wherein, described cordless communication network is the first cordless communication network, wherein, the second equipment is provided for by the second cordless communication network that is different from the first cordless communication network, or carries out voice-frequency telephony by another communication network and miscellaneous equipment.

14. in accordance with the method for claim 1, and wherein, communication network is private radio communication network.

15. in accordance with the method for claim 1, and wherein, communication network is public wireless communication network.

16. in accordance with the method for claim 1, wherein

Cordless communication network comprises the circuit switching equipment for route voice-frequency telephony, and for the PSE of route data,

Voice-frequency telephony is by the circuit switching equipment route of communication network, and the Voice & Video data that exchange by video conference be by PSE with the form exchange of grouping,

Grouping, by Internet protocol IP communication link, transmits between the first mobile device and the second equipment.

The method of 17. 1 kinds of operation first mobile devices, comprising:

During the telephone relation between the first mobile device and the second equipment, on the first mobile device, show first user interface UI layout, a UI layout comprises the optional UI project for be switched to video conference from telephone relation;

After receiving the selection of optional UI project, present the 2nd UI layout, the 2nd UI layout comprises the first video of being taken by the first mobile device; And

After receiving the acceptance of video conference request from the second equipment:

Set up the video conference between the first mobile device and the second equipment; And

Present from showing that the 2nd UI layout is switched to the animation that shows the 3rd UI layout by reduce iteratively the size of described the first video in showing the first video, described the 3rd UI layout shows the first video overlay at least a portion of second video of being caught by the second equipment.

18. in accordance with the method for claim 17, and wherein, the first mobile device and the second equipment are to carry out at least one public wireless communication network by a wireless service operators mobile device that telephone relation is supplied with.

19. in accordance with the method for claim 17, and wherein, the 3rd UI layout comprises picture-in-picture PIP layout, and this picture-in-picture layout comprises the main viewing area that shows the second video, and the insertion viewing area of performance the first video, inserts viewing area and be less than main viewing area.

20. in accordance with the method for claim 17, wherein, present from showing that the 2nd UI layout is switched to and show that the animation of the 3rd UI layout comprises the size that reduces the first viewing area that shows the first video, to be emerging in the second viewing area of the first viewing area demonstration the second video afterwards.

21. in accordance with the method for claim 17, and wherein, a UI layout, the 2nd UI layout and the 3rd UI layout are in the identical viewing area of the UI of the first mobile device.

The equipment of 22. 1 kinds of operation first mobile devices, comprising:

For during the telephone relation between the first mobile device and the second equipment, on the first mobile device, show the device of first user interface UI layout, a UI layout comprises the optional UI project for be switched to video conference from telephone relation;

For present the device of the 2nd UI layout after receiving the selection of optional UI project, the 2nd UI layout comprises the first video of being taken by the first mobile device; And

For after receiving the acceptance of video conference request from the second equipment: set up the video conference between the first mobile device and the second equipment; And present from showing that the 2nd UI layout is switched to the device of the animation that shows the 3rd UI layout by reduce iteratively the size of described the first video in showing the first video, described the 3rd UI layout shows the first video overlay at least a portion of second video of being caught by the second equipment.

23. according to the equipment described in claim 22, and wherein, the first mobile device and the second equipment are to carry out at least one public wireless communication network by a wireless service operators mobile device that telephone relation is supplied with.

24. according to the equipment described in claim 22, and wherein, the 3rd UI layout comprises picture-in-picture PIP layout, and this picture-in-picture layout comprises the main viewing area that shows the second video, and the insertion viewing area of performance the first video, inserts viewing area and be less than main viewing area.

25. according to the equipment described in claim 22, wherein, present from showing that the 2nd UI layout is switched to and show that the animation of the 3rd UI layout comprises the size for reducing the first viewing area that shows the first video, to be emerging in the second viewing area of the first viewing area demonstration the second video afterwards.

The equipment of 26. 1 kinds of operation first mobile devices, described equipment comprises:

For during the voice-frequency telephony by cordless communication network and the second equipment, on the first mobile device, present the selectable user interface UI item destination device for be switched to the video conference between the first mobile device and the second equipment from voice-frequency telephony;

For after receiving the selection of described optional UI project, show the device of the first video of being taken by the first mobile device; With

For by (1) before permission the first mobile device and the second equipment present the Voice & Video data that exchange by video conference on the first mobile device, stop the device that voice-frequency telephony and (2) display of visually ground indicate the animation of the switching from voice-frequency telephony to video conference to initiate video conference, wherein, described animation (i) starts from reducing the size of the first video in the first video and (ii) ending to show that the first video overlay is at least a portion of the second video of being taken by the second equipment showing.

27. according to the equipment described in claim 26, wherein, comprises for during voice-frequency telephony for presenting optional UI item destination device, in the viewing area on the display screen that can be presented at the first mobile device, continues to present described optional UI item destination device.

28. according to the equipment described in claim 26, also comprises:

For in waiting for that the second video arrives the first mobile device, show the device of the first video;

For when in the time that the first mobile device has received the second video, show described animation, so that the device on present the second videometer the first mobile device.

29. according to the equipment described in claim 28, and wherein, in reducing the first video big or small, the first video overlay is on the second video.

30. according to the equipment described in claim 28, and wherein, once described animation finishes, the first video and the second video just form picture-in-picture and show, wherein, the first video serves as the insertion picture overlapping on the second video.

31. according to the equipment described in claim 26, wherein, is allowing before the first mobile device and the second equipment exchange audio frequency and video data by video conference termination voice-frequency telephony.

32. according to the equipment described in claim 26, wherein, after the first mobile device and the second equipment start to exchange audio frequency and video data by video conference, but before the first mobile device and the second equipment start to present the Voice & Video data of exchange, stop voice-frequency telephony.

33. according to the equipment described in claim 26, and wherein, the first mobile device provides together with described cordless communication network with the second equipment, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

34. according to the equipment described in claim 26, wherein, the first mobile device provides together with described cordless communication network, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment, and the second equipment is not provided for carrying out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

35. according to the equipment described in claim 34, wherein, described cordless communication network is the first cordless communication network, wherein, the second equipment is provided for by the second cordless communication network that is different from the first cordless communication network, or carries out voice-frequency telephony by another communication network and miscellaneous equipment.

36. according to the equipment described in claim 26, wherein