CN102215373A - In conference display adjustments - Google Patents

In conference display adjustments Download PDF

Info

Publication number
CN102215373A
CN102215373A CN2010106026539A CN201010602653A CN102215373A CN 102215373 A CN102215373 A CN 102215373A CN 2010106026539 A CN2010106026539 A CN 2010106026539A CN 201010602653 A CN201010602653 A CN 201010602653A CN 102215373 A CN102215373 A CN 102215373A
Authority
CN
China
Prior art keywords
video
image
equipment
demonstration
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN2010106026539A
Other languages
Chinese (zh)
Other versions
CN102215373B (en
Inventor
E·C·克兰菲尔
S·O·勒梅
吴锡荣
周小松
J·S·阿布安
郑铉国
小R·加西亚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Apple Inc
Original Assignee
Apple Computer Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/794,768 external-priority patent/US8502856B2/en
Application filed by Apple Computer Inc filed Critical Apple Computer Inc
Publication of CN102215373A publication Critical patent/CN102215373A/en
Application granted granted Critical
Publication of CN102215373B publication Critical patent/CN102215373B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Abstract

Some embodiments provide a method for modifying a composite display of a first mobile device that is engaged in a video conference with a second device. The method presents, on the first device, the composite display having a first video captured by the first device and a second video captured by the second device. The method receives, at the first device, an input for modifying the composite display during the video conference. The method modifies the composite display based on the received input.

Description

Demonstration adjustment in the meeting
When the model machine of the iPhone 4 of Apple on March 25th, 2010 when the engineer of Apple is stolen, the invention that will disclose and advocate among the application is disclosed to the public in advance and without Apple with authorizing.At this obviously before the theft, do not submit to as yet the application based on U.S.'s priority application.
Background technology
Many present portable sets possess video capture function such as intelligent telephone set.By the camera on the telephone set, the user of portable set can take rest image and video.But, after finishing video capture, for the video of taking is sent to the opposing party, the user must directly send to described the opposing party to video usually, perhaps video is uploaded to another location (for example, internet video trustship website (hosting site)).Unfortunately, this does not allow described the opposing party in the portable set capture video, checks live video stream.
In addition, the portable set of standard only is equipped with a camera, and handles from the information of this camera quite difficult.Desirable equipment ought to have the multi-section camera, and can send real-time video, and described real-time video is synthesizing from the video of at least two cameras.Just handle the equipment of the video flowing of a plurality of shootings, with the network of the transmission that is connected, is responsible for the processing live video stream with described equipment, in view of the resource-constrained that can supply portable set to utilize, this is an especially problem of difficulty.
Summary of the invention
Some embodiments of the present invention provide to be had two ones and can take pictures and the mobile device of the camera of video.The mobile device of some embodiment has the photograph image of demonstration shooting and the display screen of video image.It comprises that also the image of preserving shooting is so that send the memory of another equipment after a while to.Described equipment also has network interface, and described network interface allows this equipment during the real time communication session between the user of a plurality of equipment, and the image of taking is sent to one or more equipment.Described equipment also comprises encoder, and it can use the image encoding of described encoder to taking, so that another equipment is stored or sent in this locality.Described mobile device also comprises decoder, and described decoder allows the diagram decoding of this equipment to being taken by another equipment, the perhaps picture decoding that this locality is preserved during the real time communication session.
The example of real time communication session of transmission that relates to the video image of shooting is video conference.In certain embodiments, any specific time of mobile device during video conference, be merely able to transmit the video image of a camera.But, in other embodiments, mobile device can send the video image from its two camera simultaneously during video conference or other real time communication session.
With the video conference of another equipment during, the mobile device of some embodiment transmits the content of other type by together with the video by one portion or two camera.When the camera that an example of this other content is included in this equipment is used for the video of video conference, by the low resolution or the high resolution picture image of another camera of equipment.Other example of this other content comprises that (1) is kept at file and other content on the equipment, (2) screen display of equipment (that is, being presented at the content on the screen of equipment), (3) are during video conference or other real time communication session, from the content of another equipment reception, or the like.
The mobile device of some embodiment adopts (in-conference) adjustment technology in the novel meeting, adjusts during video conference.For example, when only transmitting the video of a camera during video conference, the mobile device of some embodiment can dynamically switch to the video of transmission by its another camera.In this case, the mobile device of some embodiment makes any miscellaneous equipment of this switching notice participation video conference described miscellaneous equipment to provide by the level and smooth conversion between the video of these two camera at its that end.
In certain embodiments, the request of switching camera not only can originate from during video conference, " this locality " equipment that between its camera, switches, but also can originate from another " long-range " equipment that is just receiving the video that local device takes.In addition, allow another equipment of a device directive to switch the example of remote control ability that camera is the equipment of some embodiment.In certain embodiments, can be long-range comprise that to the example of other operation of device directive exposure adjusts operation (for example, automatic exposure), focal length adjustment operation (for example, automatic focus) or the like.Another example of adjusting in the meeting of novelty that can the Local or Remote appointment is the area-of-interest (ROI) in the identification capture video, with the behavior of using described ROI identification modification to take camera, modification has the image processing operations of the equipment of taking camera, perhaps revises the encoding operation that has the equipment of taking camera.
Another example of adjusting in the meeting of the novelty of some embodiment relates to the real time modifying of the synthetic video demonstration of equipment generation.Specifically, in certain embodiments, the mobile device generation shows the synthetic demonstration by a plurality of videos of the multi-section camera of one or more equipment simultaneously.In some cases, synthetic demonstration is placed on video in the adjacent viewing area and (for example, is placed in the adjacent window).In other cases, synthetic demonstration is that picture-in-picture (PIP) shows, described PIP demonstration comprises that at least two viewing areas that show two different videos, one of them viewing area are background master viewing areas, and another viewing area is that the prospect that overlaps on the background master viewing area is inserted the viewing area.
The real time modifying that synthetic video among some embodiment shows relates in response to the user to the selection of viewing area with move mobile one or more viewing areas in synthetic the demonstration.When the screen rotation of the synthetic equipment that shows was provided, some embodiment also rotated during video conference and should syntheticly show.In addition, the user of the mobile device of some embodiment permission equipment exchanges the video (that is, the video in the prospect insertion demonstration is appeared in the background master demonstration, and the video in the background master demonstration is appeared in the prospect insertion demonstration) in (swap) PIP demonstration.
The summary of the invention intention of front is simply introduced some embodiments of the present invention.Do not mean it is the introduction or the summary of disclosed all subject matters in this document.Following embodiment and the accompanying drawing that relates in embodiment will further specify the embodiment that describes in described summary of the invention, and other embodiment.Therefore, be appreciated that all embodiment that this document is described, need intactly investigate summary of the invention, embodiment and accompanying drawing.
Description of drawings
Stated novel feature of the present invention in the accessory claim.But, in order to illustrate, in following accompanying drawing, several embodiments of the present invention have been represented.
The synthetic demonstration of some embodiment of Fig. 1 graphic extension.
The another kind of some embodiment of Fig. 2 graphic extension is synthetic to be shown.
Fig. 3 is the Video processing of two camera movement equipment of some embodiment of graphic extension and the software architecture of coding module conceptually.
Fig. 4 is the photographic images processing unit of some embodiment of graphic extension conceptually.
Fig. 5 conceptually graphic extension based on the example of the different frame rates of different vertical blanking interval (VBI).
Fig. 6 conceptually graphic extension based on the example of the different interlacing scan frame rate of different VBI.
Fig. 7 is another photographic images processing unit of some embodiment of graphic extension conceptually.
Fig. 8 is another photographic images processing unit of some embodiment of graphic extension conceptually.
Fig. 9 is the video conference of two camera movement equipment of some embodiment of graphic extension and the software architecture of processing module conceptually.
Figure 10 is the example video conference request sending and receiving sequence of some embodiment of graphic extension conceptually.
The user interface of the video conference setting operation of some embodiment of Figure 11 graphic extension.
The user interface of accepting video conference invitation of some embodiment of Figure 12 graphic extension.
Another user interface of accepting video conference invitation of some embodiment of Figure 13 graphic extension.
Another user interface of the video conference setting operation of some embodiment of Figure 14 graphic extension.
Figure 15 is the processing of the bit rate of the setting video conference of some embodiment of graphic extension conceptually.
Figure 16 is the video conference of two camera movement equipment of some embodiment of graphic extension and another software architecture of processing module conceptually.
Figure 17 is another software architecture of two camera movement equipment of some embodiment of graphic extension conceptually.
Figure 18 conceptually graphic extension by the processing of carrying out as the video conference manager of some embodiment of graphic extension among Figure 16.
Figure 19 is the software architecture of the time noise reduction module of some embodiment of graphic extension conceptually.
Figure 20 is the processing of the time noise of the reduction video image of some embodiment of graphic extension conceptually.
Figure 21 conceptually graphic extension by the processing of carrying out as the image processing manager of some embodiment of graphic extension among Fig. 9.
The user interface of the exposure adjustment operation of some embodiment of Figure 22 graphic extension.
The user interface of the focal length adjustment operation of some embodiment of Figure 23 graphic extension.
Figure 24 graphic extension is conceptually handled by the perspective correction of carrying out as the image processing manager of some embodiment of graphic extension among Figure 16.
Figure 25 is the example perspective correction operation of some embodiment of graphic extension conceptually.
Figure 26 is the software architecture of the encoder-driven device of some embodiment of graphic extension among graphic extension such as Figure 16 conceptually.
Figure 27 graphic extension is conceptually handled by the image zoom of carrying out as the encoder-driven device of some embodiment of graphic extension among Figure 26.
Figure 28 conceptually graphic extension by software architecture as the decoder driver of some embodiment of graphic extension among Figure 16.
Figure 29 graphic extension is conceptually extracted by the image of carrying out as the decoder driver of some embodiment of graphic extension among Figure 28 and handles.
The encoder-driven device that comprises two rate controllers of some embodiment of Figure 30 graphic extension.
Figure 31 is the software architecture of the networking manager of some embodiment of graphic extension among graphic extension such as Figure 16 conceptually.
The user interface that locks onto corner (snap to corner) operation of some embodiment of Figure 32 graphic extension.
Another user interface that locks onto the corner operation of some embodiment of Figure 33 graphic extension.
The PIP of some embodiment of Figure 34 graphic extension shows the user interface of rotary manipulation.
The PIP of some embodiment of Figure 35 graphic extension shows another user interface of rotary manipulation.
The PIP of some embodiment of Figure 36 graphic extension shows another user interface of rotary manipulation.
The PIP of some embodiment of Figure 37 graphic extension shows another user interface of rotary manipulation.
Prospect during the convergent-divergent PIP of some embodiment of Figure 38 graphic extension shows is inserted the user interface of viewing area.
Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 39 graphic extension shows.
Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 40 graphic extension shows.
Another user interface of insertion viewing area during the convergent-divergent PIP of some embodiment of Figure 41 graphic extension shows.
The user interface of the area-of-interest in the identification display frame of some embodiment of Figure 42 graphic extension.
Another user interface of area-of-interest in the identification display frame of some embodiment of Figure 43 graphic extension.
Another user interface of area-of-interest in the identification display frame of some embodiment of Figure 44 graphic extension.
The processing of on two camera movement equipment, carrying out local switching camera operation of some embodiment of Figure 45 graphic extension.
The user interface of the switching camera operation of some embodiment of Figure 46 graphic extension.
Another user interface of the switching camera operation of some embodiment of Figure 47 graphic extension.
Another user interface of the switching camera operation of some embodiment of Figure 48 graphic extension.
Another user interface of the switching camera operation of some embodiment of Figure 49 graphic extension.
The processing of on two camera movement equipment, carrying out long-range switching camera operation of some embodiment of Figure 50 graphic extension.
The Long-distance Control of some embodiment of Figure 51 graphic extension is switched the user interface of camera operation.
The Long-distance Control of some embodiment of Figure 52 graphic extension is switched another user interface of camera operation.
The Long-distance Control of some embodiment of Figure 53 graphic extension is switched another user interface of camera operation.
The Long-distance Control of some embodiment of Figure 54 graphic extension is switched another user interface of camera operation.
Figure 55 exposing of some embodiment of graphic extension conceptually adjusts the processing of operation.
Exposing of some embodiment of Figure 56 graphic extension adjusted the user interface of operation.
Exposing of some embodiment of Figure 57 graphic extension adjusted another user interface of operation.
Exposing of some embodiment of Figure 58 graphic extension adjusted another user interface of operation.
Figure 59 conceptually graphic extension by the exposure adjustment processing of carrying out as the image processing manager of some embodiment of graphic extension among Figure 16.
Figure 60 is the exposure adjustment operation of some embodiment of graphic extension conceptually.
Figure 61 is the processing of the execution focal length adjustment operation of some embodiment of graphic extension conceptually.
The user interface of the focal length adjustment operation of some embodiment of Figure 62 graphic extension.
Another user interface of the focal length adjustment operation of some embodiment of Figure 63 graphic extension.
Another user interface of the focal length adjustment operation of some embodiment of Figure 64 graphic extension.
The difference about the video taken from the one or more pairs of camera movement equipment of some embodiment of Figure 65 graphic extension shows arranges.
The prospect inserting video of some embodiment of Figure 66 graphic extension is superimposed upon the user interface on the background video of PIP in showing.
The technology of the prospect of definite video image of some embodiment of Figure 67 graphic extension.
Some embodiment of Figure 68 graphic extension during video conference, the user interface that insertion demonstration during exchange PIP shows and background show.
The user interface that locks onto the corner operation of some embodiment of Figure 69 graphic extension.
The user interface that locks onto corner and push operation of some embodiment of Figure 70 graphic extension.
The PIP of some embodiment of Figure 71 graphic extension shows the user interface of rotary manipulation.
The PIP of some embodiment of Figure 72 graphic extension shows another user interface of rotary manipulation.
Some embodiment of Figure 73 graphic extension during video conference, from two long-distance videos, select the user interface of a video.
Some embodiment of Figure 74 graphic extension during video conference, from two local videos, select the user interface of a video.
Selection is used for the user interface of the video of video conference before the meeting of some embodiment of Figure 75 graphic extension.
The example of some embodiment of Figure 76 graphic extension by the allocated bandwidth between two videos of two camera movement equipment shootings.
Figure 77 is the arbiter module of the rate controller of the two camera movement equipment of management of some embodiment of graphic extension conceptually.
Figure 78 image to by the camera of two camera movement equipment of some embodiment of graphic extension conceptually carries out Methods for Coding.
Figure 79 is the other method to being encoded by the image of the camera of two camera movement equipment of some embodiment of graphic extension conceptually.
Figure 80 is illustrated in the example image of the method for graphic extension among Figure 79 and synthesizes.
Figure 81 is the other method to being encoded by the image of the camera of two camera movement equipment of some embodiment of graphic extension conceptually.
Figure 82 is the method to being decoded by the image of the camera of two camera movement equipment of some embodiment of graphic extension conceptually.
Figure 83 is the other method to being decoded by the image of the camera of two camera movement equipment of some embodiment of graphic extension conceptually.
Figure 84 is the video conference of two camera movement equipment of some embodiment of graphic extension and another software architecture of processing module conceptually.
The user interface of the multipart video-meeting of some embodiment of Figure 85 graphic extension.
Another user interface of the multipart video-meeting of some embodiment of Figure 86 graphic extension.
Another user interface of the multipart video-meeting of some embodiment of Figure 87 graphic extension.
Figure 88 is the API of some embodiment of graphic extension (API) architecture conceptually.
The architecture of two camera movement computing equipments of some embodiment of Figure 89 graphic extension.
Figure 90 is touch I/O (I/O) equipment of some embodiment of graphic extension conceptually.
Figure 91 is the example communication system of some embodiment of graphic extension conceptually.
Figure 92 is another example communication system of some embodiment of graphic extension conceptually.
Embodiment
In the following description, for purpose of explanation, stated numerous details.But, those of ordinary skill in the art will appreciate that and can put into practice the present invention under the situation of not utilizing these details.In other cases, in order to make description of the invention smudgy, known structure and equipment have been represented with the form of block diagram because of unnecessary details.
Some embodiments of the present invention provide a kind of to be had two ones and can take pictures and the mobile device of the camera of video.The example of mobile device comprises the mobile computing device of mobile phone, intelligent telephone set, PDA(Personal Digital Assistant), laptop computer, tablet personal computer or any other type.Photo used herein refers to by the each ground of individual screening-mode, perhaps by the picture image of each several the ground of quick screening-mode by camera.On the other hand, video refers to a series of video images that camera is taken with special speed (being commonly referred to frame rate).The typical frame speed of capture video is 25 frame/seconds (fps), 30fps and 60fps.The camera of the mobile device of some embodiment can be with these and other frame rate capture video image (that is frame of video).
The mobile device of some embodiment (1) can show the photograph image and the video image of shooting, (2) can preserve the image of shooting, so that send another equipment after a while to, (3) can be during the real time communication session between the multidigit user of a plurality of equipment, the image of taking is sent to one or more equipment, (4) can be to the image encoding of taking, so that another equipment is stored or sent in this locality.
The example of real time communication session of transmission that relates to the video image of shooting is video conference.In certain embodiments, any specific time during video conference, mobile device is merely able to transmit the video image of a camera.But, in other embodiments, during video conference or other real time communication session, mobile device can transmit the video image of its two camera simultaneously.
The mobile device of some embodiment produces synthetic the demonstration, shows when described synthetic demonstration comprises by a plurality of video of the multi-section camera of one or more equipment.In some cases, synthetic demonstration is placed on video in the adjacent viewing area and (for example, is placed in the adjacent window).Synthetic 100 this example that shows of Fig. 1 graphic extension, synthetic demonstration 100 comprises two adjacent viewing areas 105 and 110, these two viewing areas 105 and 110 show by two camera of an equipment simultaneously, perhaps by two videos of two camera of two distinct devices participating in video conference.
In other cases, synthetic demonstration is to comprise that the PIP of at least two viewing areas that show two different videos shows that one of them viewing area is background master viewing area, and another viewing area is that the prospect that overlaps on the background master viewing area is inserted the viewing area.The synthetic PIP of Fig. 2 graphic extension shows a this example of 200.Synthetic PIP shows that 200 comprise background master viewing area 205 and the prospect insertion viewing area 210 that overlaps on the background master viewing area.These two viewing areas 205 and 210 show by two camera of an equipment simultaneously, perhaps by two videos of two camera of two distinct devices participating in video conference.Although the synthetic PIP of the example that illustrates in this article and discuss shows that being similar to the synthetic PIP that shows whole prospect insertion viewing area 210 in background master viewing area 205 shows 200, but have on the background of the overlapping master viewing area 205, but other synthetic PIP of also incomplete prospect insertion viewing area 210 in background master viewing area 205 shows it also is possible.
Except with the video conference of another equipment during transmit the video content, the mobile device of some embodiment can transmit the content of other type together in company with the video content of meeting.When one of camera that an example of this other content is included in equipment is taken the video that is used for video conference, by the low resolution or the high resolution picture image of another camera of this equipment.Other example of this other content comprises that (1) is kept at file and other content on the equipment, (2) screen display of equipment (that is, being presented at the content on the screen of equipment), (3) are during video conference or other real time communication session, from the content of another equipment reception, or the like.
The mobile device of some embodiment adopts adjustment technology in the novel meeting, adjusts during video conference.For example, when only transmitting the video of a camera during video conference, the mobile device of some embodiment can dynamically switch to the video of transmission by its another camera.In this case, the mobile device of some embodiment is participated in any another equipment of video conference to this switching notice, makes described another equipment to provide level and smooth conversion between the video of these two camera at its that end.
In certain embodiments, the request of switching camera not only can originate from during video conference, " this locality " equipment that between its camera, switches, but also can originate from another " long-range " equipment that is just receiving the video that local device takes.In addition, allow another equipment of a device directive to switch the example of remote control ability that camera is the equipment of some embodiment.In certain embodiments, can be long-range comprise that to the example of other operation of device directive exposure adjusts operation (for example, automatic exposure), focal length adjustment operation (for example, automatic focus) or the like.Another example of adjusting in the meeting of novelty that can the Local or Remote appointment is the area-of-interest (ROI) in the identification capture video, with the behavior of using described ROI identification modification to take camera, modification has the image processing operations of the equipment of taking camera, perhaps revises the encoding operation that has the equipment of taking camera.
Another example of adjusting in the meeting of the novelty of some embodiment relates to the real time modifying of the synthetic video demonstration of equipment generation.Specifically, in certain embodiments, the real time modifying that synthetic video shows relates in response to the user to the selection of viewing area with move mobile one or more viewing areas in synthetic the demonstration.When the screen rotation of the synthetic equipment that shows was provided, some embodiment also rotated during video conference and should syntheticly show.In addition, the order of the video during user's upset (flip) PIP of the mobile device permission equipment of some embodiment shows (promptly, video in the prospect insertion demonstration is appeared in the background master demonstration, and the video in the background master demonstration is appeared in the prospect insertion demonstration).
The following describes several more detailed embodiment.The I joint provides the explanation of the video processing architecture of some embodiment.Subsequently, the II joint is described the photographic images processing unit of some embodiment.In certain embodiments, the photographic images processing unit is the assembly of the responsible processing of equipment by the original image of the camera of equipment.
Next, the III joint is described the video conference architecture of some embodiment.The III joint is also described the video conference module of some embodiment, and several modes that single camera video conference is set.Afterwards, adjust and control operation in the meeting of some embodiment of IV joint description.V joint describe subsequently some embodiment during video conference, transmit and show video conference feature from a plurality of videos of each equipment.Next, the VI joint is described in during the video conference, in company with the transmission of non real-time content real-time video together.At last, the VII joint is described the hardware architecture of two camera apparatus of some embodiment.
I. video capture and processing
Fig. 3 is the Video processing and the coding module 300 of two camera movement equipment of some embodiment of graphic extension conceptually.In certain embodiments, 300 pairs of images by the camera of two camera movement equipment of module are handled and video are encoded.As shown in Figure 3, module 300 comprises photographic images processing unit (CIPU) driver 305, exchange of media module 310, encoder-driven device 320 and video processing module 325.
In certain embodiments, program (user of media content and producer) the switched-media content on the exchange of media module 310 permission equipment and about the instruction of the processing of media content.In Video processing and coding module 300, the exchange of media module 310 of some embodiment between video processing module 325 and CIPU driver 305, and between video processing module 325 and encoder-driven device 320 these instructions of route and media content.Easier for the route that makes described instruction and media content, the exchange of media module 310 of some embodiment provides the one group of API (API) for the user of media content and producer's use.In some such embodiment, exchange of media module 310 is one group of one or more framework as a part that runs on the operating system on two camera movement equipment.An example of this exchange of media module 310 is Core Media frameworks that Apple provides.
325 pairs of image and/or videos by the camera of equipment of video processing module carry out image processing.The example of this operation comprises that exposure adjustment operation, focal length adjustment operation, perspective correction, dynamic range adjustment, image zoom, image synthesize or the like.In certain embodiments, some image processing operations also can be carried out by exchange of media module 310.For example, as shown in Figure 3, the exchange of media module 310 of some embodiment is carried out time noise reduction (TNR) operation (for example, with TNR 315) of reduction by the noise in the video image of the camera of equipment.The other example of this image processing operations of video processing module 325 and exchange of media module 310 is provided below.
By exchange of media module 310, video processing module 325 and CIPU driver 305 and encoder-driven device 320 interfaces, as mentioned above.CIPU driver 305 serves as the communication interface between photographic images processing unit (CIPU) 330 and the exchange of media module 310.As described further below, CIPU 330 is assemblies of two camera apparatus, is responsible for processing at the image taking of the camera of equipment or the image of video capture operating period shooting.CIPU driver 305 is by exchange of media module 310, receives from the image of one one or two camera of equipment and/or the request of video from video processing module 325.CIPU driver 305 passes to CIPU 330 to described request, in response, receive the image and/or the video of asking from CIPU 330, CIPU driver 305 sends to video processing module 325 to the image and/or the video that receive subsequently by exchange of media module 310.By CIPU driver 305 and exchange of media module 310, the video processing module 325 of some embodiment also sends instruction to CIPU 330, so that revise its certain operations (for example, revising the frame rate of camera, the adjustment operation that exposes, focal length adjustment operation or the like).
Encoder-driven device 320 serves as the communication interface between exchange of media module 310 and the encoder hardware 335 (for example, encoding pack on encoder chip, the chip system or the like).In certain embodiments, encoder-driven device 320 is by exchange of media module 310, receives images and to the request of image encoding from video processing module 325.Encoder-driven device 320 is wanting image encoded to send to encoder 335, and encoder 335 carries out photo coding or video coding to image subsequently.When encoder-driven device 320 when encoder 335 receives coded image, encoder-driven device 320 is given video processing module 325 the coded image loopback by exchange of media module 310.
In certain embodiments, video processing module 325 can carry out different operations from the coded image that encoder receives to it.The example of described operation comprises coded image is kept in the memory of equipment, and the network interface by equipment transmits coded image in video conference, or the like.
In certain embodiments, some or all modules of Video processing and coding module 300 are realized as the part of operating system.For example, some embodiment are embodied as all four assemblies 305,310,320 and 325 of Video processing and coding module 300 part of the operating system of equipment.Other embodiment is embodied as the part of the operating system of equipment to exchange of media module 310, CIPU driver 305 and encoder-driven device 320, and video processing module 325 as the application that on operating system, moves.In addition, other realization of module 300 also is possible.
The operation of Video processing and coding module 300 is described now during the video capture session.In order to begin the video capture session, video processing module 325 is initialized as the required several assemblies of video capture session.In certain embodiments, these assemblies comprise (1) CIPU 330, proportional zoom of (2) video processing module 325 and synthesis module (not shown), the image processing module (not shown) of (3) video processing module 325 and (4) encoder 335.In addition, when participating in video conference, the video processing module 325 initialization network manager (not shown) of some embodiment.
By exchange of media module 310 and CIPU driver 305, video processing module sends to CIPU 330 to its initialization request, so that one one of equipment or two cameras begin video capture.In certain embodiments, described request is specified particular frame speed, exposure and the scaling size of every the camera that needs capture video.The response described request, CIPU 330 beginnings are returned the video image from the camera of being asked with frame rate, exposure and the scaling size of appointment.These video images are returned to video processing module 325 by CIPU driver 305 and exchange of media module 310, and as mentioned above, before video image was offered video processing module 325,310 pairs of video images of exchange of media module carried out the TNR operation.At video processing module 325, video image is stored in the buffer (not shown), so that carry out other image processing.
The image processing module of video processing module 325 is fetched the video image that is kept in the buffer, to carry out other Video processing.Proportional zoom and synthesis module are fetched the processed video image subsequently, so that scaled video image (if the necessary words of real-time demonstration on the display screen of equipment).In certain embodiments, the image of this module by two camera of equipment, perhaps be used in during the video conference image creation composograph by the camera of the camera of equipment and another equipment, so that the real-time demonstration of the video image of shooting is provided, perhaps create composite video image for coding on equipment.
Handle and/or synthetic after video image be provided for encoder 335 by encoder-driven device 320 and exchange of media module 310.Encoder 335 is subsequently to encoding video pictures.Coded image is returned to video processing module 325 (once more by encoder-driven device 320 and exchange of media module 310) subsequently, so that be kept on the equipment, perhaps transmits during video conference.When equipment was participated in video conference, network manager (by video processing module 325 initialization) was fetched these coded images subsequently, to the coded image subpackage, and by the network interface (not shown) of equipment coded image was sent to one or more miscellaneous equipments.
II. photographic images is handled
A. single lead
Image by the camera of two camera movement equipment of some embodiment is original raw image.Can be used to other operation at these images, such as image being sent to another equipment (for example, during video conference), preserving before image or the display image, these images need transform to the specific color space.In addition, it is processed that the image of camera needs, and with error recovery and/or distortion, and adjusts color, size of image or the like.Therefore, some embodiment carried out several processing operations to image before preserving, transmit and showing this image.The section processes of described image is carried out by CIPU 330.
Illustrate the example of described CIPU among Fig. 4.Specifically, Fig. 4 summarizes the photographic images processing unit (CIPU) 400 of some embodiment of ground graphic extension.CIPU 400 comprises or each image of only handling one of camera from equipment, perhaps handle single processing pipeline (single processing pipeline) 485 simultaneously from the image of two cameras of equipment according to time division multiplexing mode (that is, according to time-interleaved mode).Can differently dispose the processing pipeline 485 of CIPU 400, with different qualities and/or the operation setting of handling different cameras.The example of the different camera characters among some embodiment comprises different resolution, noise transducer, lens type (fixed lens or zoom lens), or the like.In addition, in certain embodiments, equipment can comprise image resolution ratio size, frame rate, zoom rank, exposure etc. according to the example of the different operating setting of its operate camera.
As shown in Figure 4, CIPU 400 comprises sensor assembly 415, OK/and frame buffer 417, bad pixel correction (BPC) module 420, lens shade (LS) module 425 is gone mosaic module 430, white balance (WB) module 435, γ module 440, color space transformation (CSC) module 445, tone, saturation and contrast (HSC) module 450, scaler module 455, filter module 460, statistics engine 465, two groups of registers 470 and controller module 475.In certain embodiments, whole modules of CIPU 400 all are to realize with hardware (for example, ASIC, FPGA, have the SOC of microcontroller, or the like), and in other embodiments, some or all modules of CIPU 400 realize with software.
As shown in Figure 4, two pel array 410a of two cameras of sensor assembly 415 and equipment and 410b, and two sensors 405a and 405b coupled in communication.In certain embodiments, the mobile industry processor interface (MIPI) by each camera sensor makes described coupled in communication easier.
By described coupled in communication, sensor assembly 415 can be transmitted instruction to camera, with the various aspects of the operation of controlling every camera, such as its power stage, zoom rank, focal length, exposure etc.In certain embodiments, every camera has four kinds of operating power patterns.Under first kind of operating power pattern, camera is de-energized.With regard to second kind of operating power pattern, camera is energized, but camera also is not configured.Under the third operating power pattern, camera is energized, and the transducer of camera is configured, and the pixel of camera sensor collection photon, and a photon conversion of collecting is become digital value.But, camera sensor does not also send image to sensor assembly 415.At last, under the 4th kind of operating power pattern, camera is under the operating power pattern identical with the third power mode, except camera sends the image to sensor assembly 415 now.
In the operating period of equipment, camera can be repeatedly from a kind of operating power mode switch to another kind of operating power pattern.When the switch operating power mode, some embodiment require camera according to said sequence switch operating power mode.So in these embodiments, the camera under first kind of operating power pattern can only switch to second kind of operating power pattern.When camera was in second kind of operating power pattern, it can switch to first kind of operating power pattern or the third operating power pattern.Similarly, camera can be from the third operating power mode switch to the second kind of operating power pattern or the 4th kind of operating power pattern.When camera was in the 4th kind of operating power pattern, it was merely able to switch back the third operating power pattern.
In addition, need a certain amount of time from a kind of operating power mode switch to following a kind of or preceding a kind of operating power pattern.Thereby, switch two or three operating power pattern and be slower than a kind of operating power pattern of switching.Different operating power patterns also consumes the power of different amounts.For example, the 4th kind of operating power pattern consumes the power of maximum, and the power of the third operating power pattern consumption is higher than first kind and second kind of operating power pattern, and the power of second kind of operating power pattern consumption is higher than first kind of operating power pattern.In certain embodiments, first kind of operating power pattern do not consume any power.
When camera is not in the 4th kind of operating power pattern of photographic images, camera is remained under one of other operating power pattern.Make no camera remain on determining under which kind of mode of operation and depend on that how much power the permission camera consumes and how soon the camera needs respond the request that begins photographic images.For example, be configured to the camera that camera ratio according to the third operating power pattern (for example, standby mode) work is configured to be in first kind of operating power pattern (that is outage) and consume more power.But, when instruction camera image, can switch to the 4th kind of operating power pattern than camera quickly according to the work of first kind of operating power pattern according to the camera of the third operating power pattern work.Thereby according to different requirements (for example, to the request responding time of photographic images, power consumption), when photographic images not, camera can be configured to according to different operating power pattern work.
As described further below, when one one of video processing module 325 request or two cameras begin photographic images, and when sensor assembly 415 receives this request by controller module 475, by the coupled in communication of it and every camera, sensor assembly 415 can instruct one group or two groups of camera sensor to begin photographic images.The Baeyer filter is superimposed on each camera sensor, thus each camera sensor output Baeyer format-pattern, in the pel array that described Baeyer format-pattern is stored in each camera sensor is associated.The Baeyer format-pattern is that wherein each pixel is only preserved a kind of color value: red, blue or green image.
By coupling of it and pel array 410a and 410b, sensor assembly 415 is fetched the primitive Bayer format-pattern that is kept among camera image pixel array 410a and the 410b.By the speed that control sensor assembly 415 is fetched image from the pel array of camera, sensor assembly 415 can be controlled the frame rate of the video image of being taken by particular camera.By controlling the speed that its image is fetched, sensor assembly 415 can also interweave by the reading of the image of different camera, so that interweave 485 pairs of image processing from the photographic images of different cameras of CIPU processing pipeline.Below further specify the control that sensor assembly 415 is fetched its image in the son of II.A.1 and II.A.2 joint.
Sensor assembly 415 is fetched it from pel array 410a and 410b image line (that is the several rows of pixel of image) preserve be expert at/frame buffer 417 in.By each image line in the CIPU processing pipeline 485 processing row/frame buffers 417.As shown in Figure 4, CIPU processing pipeline 485 by BPC module 420, LS module 425, go mosaic module 430, WB module 43, γ module 440, CSC module 445, HSC module 450, scaler module 455 and filter module 460 to constitute.In certain embodiments, (that is, by row) handle image from row/frame buffer 417, and in other embodiments, handle come voluntarily/entire image of frame buffer 417 line by line by CIPU processing pipeline 485 frame by frames for CIPU processing pipeline 485.
In the example pipeline of graphic extension in Fig. 4, BPC module 420 is the modules of fetching image from row/frame buffer 417.BPC module 420 is carried out bad pixel and is eliminated operation, this operation (is for example managed to proofread and correct in the image of fetching the bad pixel that may be caused by one or more defective camera sensor, defective photon sensor is sensor light not, mistakenly sensor light or the like).In certain embodiments, BPC module 420 detects bad pixel by specific pixel in the movement images and the one or more neighbors in the image.If the difference between the value of the value of described specific pixel and described neighbor greater than threshold quantity, uses the mean value of the value of color (that is red, green and blue) the several neighbors identical with described specific pixel to replace the value of described specific pixel so.
The operation part of BPC module 420 is controlled by the value in two groups of registers 470 that are kept at CIPU 400 for this module.Specifically, in order to handle the image by two different camera of equipment, some embodiment differently dispose CIPU processing pipeline 485 about every camera, as mentioned above.By in two groups of CIPU 400 different register 470a (Ra) and 470b (Rb), preserving two groups of different values, be two different camera arrangement CIPU processing pipeline 485.Every group of register 470 comprises a register (Ra or Rb) that is used for each the module 420-460 in the CIPU processing pipeline 485.The value of the operation of a processing pipeline module of one group of definition of each register holds in each registers group.Therefore, as shown in Figure 4, registers group 470a is used to indicate the mode of operation of each processing pipeline module of a camera (camera A) of two camera movement equipment, and registers group 470b is used to indicate the mode of operation of each processing pipeline module of another camera (camera B) of two camera movement equipment.
An example that differently disposes CIPU processing pipeline 485 for every camera is that the block configuration of CIPU processing pipeline 485 is become to handle the image that varies in size.For example, if camera sensor 405a is 640 * 480 pixels, camera sensor 405b is 2048 * 1536 pixels, be configured to the to hold instruction value of each resume module 2048 * 1536 pixel images of CIPU processing pipeline 485 of the registers group 470a value of each resume module 640 * 480 pixel images of CIPU processing pipeline 485 that is configured to hold instruction, and registers group 470b so.
In certain embodiments, different processing pipeline configuration (that is register value) is stored in during different profiles is provided with.In some such embodiment, allow the user of mobile device to select profile that (for example, by being presented at the user interface on the mobile device) one of is set, so that the operation of an one or multi-section camera to be set.For example, the user can select camera arrangement is become the profile setting of shooting high-resolution video, the profile setting that identical camera arrangement becomes to take low-resolution video, the profile setting that perhaps two cameras all is configured to take the static image of high-resolution.Various configuration all is possible, can be stored in during many different profiles are provided with.In other such embodiment, replace allowing the user to select the profile setting, according to application or the activity that the user selects, select the profile setting automatically.For example, if the user selects video conference application, the so automatic profile of selecting two camera arrangement are become capture video is used if the user selects to take a picture, the profile that so automatic selection is configured to take rest image to one of camera, or the like.
After BPS module 420, LS module 425 receives the image behind the bad pixel correction.LS module 425 is carried out the correcting lens shadow operation, to proofread and correct the image deflects that caused by the camera lens that produces optical attenuation effect (that is, towards the edge of camera sensor, light reduces gradually).This effect causes image illuminance inhomogeneous (for example, in the corner and/or edge darker).In order to proofread and correct these image deflects, the LS module 425 of some embodiment is estimated the Mathematical Modeling of the illuminance decay of lens.The model of estimating is used to the lens decay of compensating images subsequently, with the part of evenly not thrown light on of illumination image equably.For example, if the brightness in the corner of image is half of brightness of picture centre, the LS module 425 of some embodiment is on duty with 2 corner pixels so, so that produce uniform image.
Go mosaic module 430 to carry out and go the mosaic operation, so that produce full-colour image by the image of sample color.As mentioned above, camera sensor output Baeyer format-pattern, because each pixel of Baeyer format-pattern is only preserved a kind of colour, so the Baeyer format-pattern is incomplete.Go mosaic module 430 by in the Baeyer format-pattern, inserting the colour of every group of color, by Baeyer format-pattern reconstruct red, green, blue (RGB) image.
435 pairs of WB modules are carried out white balance operation from the RGB image that goes mosaic module 430 to receive, and make the color similarity of the described content that human eye perceives arrives in the color of picture material and the actual life.WB module 435 is adjusted white balance by the color of adjusting image, correctly to present neutral color (for example, grey, white etc.).For example, the image of a blank sheet of paper under incandescent lamp may be revealed as yellow, and the described a piece of paper of human eye perceives is a white.For the difference between the color of the color that solves the image that transducer takes and the image of human eye perceives, WB module 435 is adjusted the colour of images, makes the image of shooting correctly reflect the color of human eye perceives.
Statistics engine 465 is collected in the view data in each stage of CIPU processing pipeline 485.For example, Fig. 4 represents that statistics engine 465 is collected in LS module 425, goes mosaic module 430 and WB module 435 view data afterwards.Different embodiment collects data from the different phase of the arbitrary number of CIPU processing pipeline 485.Statistics engine 465 is handled the data of collecting, and according to the data after handling, adjusts the operation of camera sensor 405a and 405b by controller module 475 and sensor assembly 415.The example of this operation comprises exposure and focuses on.Although Fig. 4 shows the statistics engine 465 by controller module 475 control camera sensor 405a and 405b, but, other embodiment of statistics engine 465 is by sensor assembly 415 control camera sensor.
Data after the processing also can be used for adjusting the operation of each module of CIPU 400.For example, the statistics engine 465 of some embodiment is adjusted the operation of WB module 435 according to the data of collecting after WB module 435.In some such embodiment, statistics engine 465 provides Automatic white balance (AWB) function by utilizing the data after handling, to adjust the white balance operation of WB module 435.Other embodiment can use data after the processing of collecting from stage of the arbitrary number of CIPU processing pipeline 485 to adjust the operation of the arbitrary number module in the CIPU processing pipeline 485.In addition, statistics engine 465 can also the slave controller module 475 receives instruction, with the operation of one or more modules of adjusting CIPU processing pipeline 485.
After WB module 435 receives image, 440 pairs of images of γ module carry out the γ correct operation, with the brightness or the tristimulus values of Code And Decode camera arrangement.The γ module 440 of some embodiment, is carried out the γ value and is proofreaied and correct with the γ value of correcting image by the linear signal of 10-12 bit being converted to the non-uniform encoding of 8 bits.Some embodiment utilize look-up table to proofread and correct the γ value.
CSC module 445 the images that receive from γ module 440 from a color space transformation to another color space.Specifically, CSC module 445 image from the RGB color space transformation to brightness with colourity (YUV) color space.But, other embodiment of CSC module 445 can be to and from the color space transformation image of arbitrary number.
HSC module 450 can be adjusted from tone, saturation, contrast or their combination in any of the image of CSC module 445 receptions.For example, HSC module 450 can be adjusted these character, to reduce noise or to strengthen image.For example, can increase the saturation of the image of taking by the low noise camera sensor, so that image seems more bright-coloured.On the contrary, can reduce the saturation of the image of taking by the strong noise camera sensor, to reduce the coloured noise of this image.
After HSC module 450, scaler module 455 scalable images to adjust the pixel resolution of image, are perhaps adjusted the size of data of image.For example, scaler module 455 also can reduce the size of image, so that be fit to less display.For example, scaler module 455 can be with many different mode proportional zoom images.For example, scaler module 455 can scale up (that is, amplifying) and reduce (that is, dwindling) image in proportion.Scaler module 455 can also bi-directional scaling image, perhaps shifting ground zoomed image.
One or more filtering operations of image applications that 460 pairs of filter modules receive from scaler module 455 are with one or more attributes of some or all pixels of changing image.The example of filter comprises low pass filter, high pass filter, band pass filter, two-sided filter, Gaussian filter, or the like.Thereby filter module 460 can be to image applications multiple different filtering arbitrarily.
The controller module 475 of some embodiment is microcontrollers of the operation of control CIPU 400.In certain embodiments, controller module 475 (1) is by sensor assembly 41, the operation (for example, exposure) of control camera sensor, the operation of (2) control CIPU processing pipeline 485, (3) timing of control CIPU processing pipeline 485 (for example, when switch camera sensor, when switch register, or the like), (4) control photoflash lamp/stroboscope light (not shown), described photoflash lamp/stroboscope light is the part of two camera movement equipment of some embodiment.
Some embodiment of controller module 475 handle from the instruction of statistics engine 465 and 480 receptions of CIPU driver.In certain embodiments, the instruction that receives from CIPU driver 480 is from two camera movement equipment (promptly, be received from local device) instruction, and in other embodiments, the instruction that receives from CIPU driver 480 is the instruction (for example, the Long-distance Control during the video conference) from another equipment.According to the instruction after handling, controller module 475 can be adjusted the operation of CIPU 400 by the value of planning register 470.In addition, controller module 475 can dynamically be planned the value of register 470 again in the operating period of CIPU 400.
As shown in Figure 4, CIPU 400 comprises the number of modules in the CIPU processing pipeline 485.But, those of ordinary skill in the art will appreciate that and can perhaps realize CIPU 400 with other disparate modules only with the module of some graphic extensions.In addition, the processing of being carried out by disparate modules can be according to the be applied sequentially to image different with the order of graphic extension among Fig. 4.
With reference now to Fig. 4,, the exemplary operations of CIPU 400 is described.For the purpose of illustrating, one group of register Ra is used to handle the image that the camera sensor 405a by two camera movement equipment takes, and one group of register Rb is used to handle the image that the camera sensor 405b by two camera movement equipment takes.Controller module 475 receives the instruction of the image that one of camera of producing by two camera movement equipment takes from CIPU driver 480.
Controller module 475 is each module of initialization CIPU processing pipeline 485 subsequently, to handle the image that one of camera by two camera movement equipment is taken.In certain embodiments, this comprises whether controller module 475 inspections have used one group of correct register in the register 470.If for example CIPU driver 480 instruction control unit modules 475 produce the image of being taken by camera sensor 405a, controller module 475 checks that whether one group of register Ra is one group of register that the module of CIPU 400 reads so.If not, controller module 475 switches between two groups of registers so, makes that this group register Ra is one group of register that the module by CIPU 400 reads.
Each module in CIPU processing pipeline 485, mode of operation is by the value indication that is kept among described one group of register Ra.As previously mentioned, the value in one group of register 470 can operating period of CIPU 400 by dynamic rule again.Thereby a treatment of picture is different from next treatment of picture.Although the value of the mode of operation of indication each module of each module reading and saving in register among the CIPU 400 is described in the discussion of this exemplary operations of CIPU 400, but in the embodiment that some softwares are realized, change each module that parameter is sent to CIPU400 into.
In certain embodiments, controller module 475 postpone the special time amount after pel array 410a fetches image, comes initialization sensor assembly 415 by instruction sensor module 415.In other words, controller module 475 instruction sensor modules 415 are fetched image with special speed from pel array 410a.
Subsequently, controller module 475 is by sensor assembly 415 instruction camera sensor 405a photographic images.In certain embodiments, controller module 475 also provides exposure parameter and other camera operation parameter to camera sensor 405a.In other embodiments, camera sensor 405a uses the default value of camera sensor operating parameter.According to described parameter, camera sensor 405a takes original image, and described original image is stored among the pel array 410a.Sensor assembly 415 is fetched described original image from pel array 410a, and image is sent to row/frame buffer 417 so that preserve, and CIPU processing pipeline 485 is handled this image afterwards.
In some cases, image may be abandoned by row/frame buffer 417.As camera sensor 405a and/or 405b during with the two-forty photographic images, sensor assembly 415 comparable BPC modules 420 can be fetched image from row/frame buffer 417 and be received image quickly, and image preserved be expert at/frame buffer 417 in (for example, when taking high frame rate video), OK/frame buffer 417 can become and be full of fully.When this thing happens, row/frame buffer 417 of some embodiment abandoned image (that is frame) according to first in first out.That is, when row/frame buffer 417 abandons a two field picture, OK/frame buffer 417 abandons that two field picture of being expert at/receiving before all other images in the frame buffer 417.
The image processing of CIPU processing pipeline 485 fetches voluntarily/image of frame buffer 417 from BPC module 420, begins with any bad pixel in the correcting image.BPC module 420 sends to LS module 425 to image subsequently, with any non-uniform illumination degree in the correcting image.After the illuminance of correcting image, LS module 425 sends to mosaic module 430 to image, goes mosaic module 430 to handle original image, produces the RGB image by original image.Subsequently, WB module 435 receives the RGB image from going mosaic module 430, and adjusts the white balance of RGB image.
As mentioned above, statistics engine 465 may have been collected some data at each point of CIPU processing pipeline 485.For example, shown in diagram among Fig. 4, statistics engine 465 in LS module 425, go to collect data after mosaic module 430 and the WB module 435.According to the data of collecting, statistics engine 465 can be adjusted the operation of camera sensor 405a and/or the operation of the one or more modules in the CIPU processing pipeline 485, so that adjust from the shooting of the successive image of camera sensor 405a.For example, according to the data of collecting, statistics engine 465 can determine that the exposure of present image is low excessively, thereby increases follow-up exposure of clapping image by sensor assembly 415 instruction camera sensor 405a.Thereby the statistics engine 465 of some embodiment plays some feedback loops of handling operation.
After WB module 435 was adjusted the white balance of image, it sent to γ module 440 to image, proofreaied and correct (gamma curve of for example, adjusting image) to carry out γ.Image after CSC module 445 is proofreaied and correct from γ module 440 reception γ, the circumstances in which people get things ready for a trip of going forward side by side spatial alternation.In this example, CSC module 445 becomes the YUV image to the RGB image transform.In other words, CSC module 445 becomes the image that shows with yuv color space to the image transform with the performance of the RGB color space.HSC module 450 receives the YUV image from CSC module 445, and adjusts tone, saturation and the contrast properties of each pixel in the image.After HSC module 450, scaler module 455 zoomed images (for example, amplification or downscaled images).After scaler module 455 receives image, 460 pairs of one or more filtering of image applications of filter module.At last, filter module 460 sends to CIPU driver 480 to the image after handling.
In this example of operation of Shuo Ming CIPU 400, each module in the CIPU processing pipeline 485 is handled image in some way in the above.But, other image of CIPU 400 processing may not need the processing of all modules of CIPU processing pipeline 485.For example, image may not need white balance adjustment, γ correction, proportional zoom or filtering.Thereby CIPU 400 can be according to the various inputs that receive, and such as the instruction from CIPU driver 480, perhaps data of being collected by statistics engine 465 are with any treated in various ways image.
Different embodiment is the speed of control and treatment image (that is frame rate) differently.A kind of mode of control frame speed is the manipulation by vertical blanking interval (VBI).For fetching image line so that handle some embodiment of image line by line, VBI fetches last column by an image of the video of the camera of two camera movement equipment from pel array, and fetches from pel array the time difference between first row of next image of video.In other embodiments, VBI fetches a image by the video of the camera of two camera movement equipment from pel array, and fetches from pel array the time difference between the next image of video.
An example that wherein can use VBI is between sensor assembly 415 and pel array 410a and 410b.For example, some embodiment of sensor assembly 415 fetch image from pel array 410a and 410b line by line, and other embodiment of sensor assembly 415 fetches image from pel array 410a and 410b in image ground one by one.Thereby, by adjusting the VBI of sensor assembly 415, can control frame speed: increase VBI and can reduce frame rate, can improve frame rate and reduce VBI.
1.VBI application: frame rate control to single camera
Fig. 5 conceptually graphic extension based on the different frame rates 505,510 of different VBI and 515 example.Each sequence is presented at each 525-555 constantly along timeline 520, holds the personage's of guitar image, and described image is taken by one of camera of two camera movement equipment.In addition, the time between each moment 525-555 is all identical, is called a time quantum.For the purpose of illustrating, referring now to sensor assembly 415 and the pel array 410a of Fig. 4, key diagram 5.Thereby each image was represented along the sensor assembly 415 of timeline 520 is fetched image from pel array 410a the moment.
In the frame rate 505 of example, sensor assembly 415 is set to 3 time quantums (for example, being set by controller module 475) about the VBI of pel array 410a.That is, sensor assembly 415 is fetched a two field picture every two moment from pel array 410a along timeline 520.As shown in the frame rate 505 of example, sensor assembly 415 525,540 and 555 is being fetched image constantly.Thereby the frame rate 505 of example has the frame rate of per three time quantum one two field pictures.
Except VBI was set to 2 time quantums, the frame rate 510 of example was similar to the frame rate 505 of example.Thereby sensor assembly 415 is fetched a two field picture every a moment from pel array 410a along timeline 520.The frame rate 510 expression sensor assemblies 415 of example are in the moment 525,535,545 and 555, and 410a fetches image from pel array.Because the VBI of the frame rate 510 of example is less than the VBI of the frame rate 505 of example, so the frame rate of the frame rate 510 of example is higher than the frame rate of the frame rate 505 of example.
VBI about pel array 410a is set to 1 time quantum except sensor assembly 415, and the frame rate 515 of example also is similar to the frame rate 505 of example.So sensor assembly 415 along timeline 520, constantly from pel array 410a is fetched a two field picture at each by instruction.As shown in the figure, sensor assembly 415 is at moment 525-555, and 410a fetches image from pel array.The VBI of the frame rate 515 of example is less than the frame rate 505 of example and 510 VBI.So the frame rate of the frame rate 515 of example is greater than the frame rate 505 and 510 of example.
2.VBI application to two cameras
Some embodiment may wish to operate simultaneously two cameras (for example, during video conference, transmitting video from two cameras) of two camera movement equipment.The different embodiment that comprise two camera movement equipment of single processing pipeline provide the different mechanisms of two cameras of the two camera movement equipment of operation simultaneously.
A kind of such mechanism is by controlling the VBI of every camera, interweaving by the treatment of picture of two camera.That is, during the VBI of a camera, take and handle the one or more images by another camera, vice versa.Because CIPU 400 described above has single processing pipeline 485, therefore this mechanism can realize in the CIPU 400 of some embodiment.In such embodiments, sensor assembly 415 is fetched image from one of pel array 410a and 410b, and the image of fetching is handled during the VBI of sensor assembly 415 about the one other pixel array by CIPU 400.
Sensor assembly 415 can be set to particular value about the VBI of each pel array.But, in certain embodiments, VBI is not configured to fetch and handle little value of used time of a two field picture than CIPU 400.Some embodiment set sensor assembly 415 for identical value about the VBI of each pel array.For example, when sensor assembly 415 was configured to identical value about the VBI of each pel array, sensor assembly 415 was alternately fetched image from pel array 410a and 410b.Other embodiment sets sensor assembly 415 for different value about the VBI of each pel array.In some such embodiment, sensor assembly 415 is configured to the multiple of sensor assembly 415 about the VBI of one other pixel array about the VBI of a pel array.For example, sensor assembly 415 is configured to 2 time quantums about the VBI of a pel array, and sensor assembly 415 is configured to 4 time quantums about the VBI of one other pixel array.In this example, with respect to each two field picture that sensor assembly 415 is fetched from described one other pixel array, sensor assembly 415 is fetched two two field pictures from a described pel array.
Fig. 6 conceptually graphic extension based on the different interleaving frame rate 605,610 of different VBI and 615 example.Except Fig. 6 comprises that Fig. 6 is similar to Fig. 5 along 13 moment 625-685 of timeline 620.In addition, hold the personage's of guitar image representative and fetch the moment of this image from a pel array, and personage's the image representative that has on trencher cap is along timeline 620 is fetched image from the one other pixel array moment along timeline 620.
For the purpose of illustrating, hold the personage's of guitar image supposition and take by the camera sensor 405a of two camera movement equipment, and the image supposition that has on the personage of trencher cap is taken by the camera sensor 405b of two camera movement equipment.In addition, referring now to sensor assembly 415 and pel array 410a and the 401b key diagram 6 of Fig. 4.
In the interlacing frames speed 605 of example, sensor assembly 415 is configured to 2 time quantums about the VBI of pel array 410a and pel array 401b.As shown in the interlacing frames speed 605 of example, sensor assembly 415 is in the moment 625,635,645,655,665,675 and 685 along timeline 620,410a fetches image from pel array, and sensor assembly 415 is in the moment 630,640,650,660,670 and 680 along timeline 620, and 410b fetches image from pel array.That is, at each time quantum, sensor assembly 415 is alternately fetched image from pel array.
VBI about pel array 410a and pel array 401b is configured to 4 time quantums except sensor assembly 415, and the interlacing frames speed 610 of example is similar to the interlacing frames speed 605 of example.The interlacing frames speed 610 of example shows that sensor assembly 415 is in the moment 625,645,665 and 685 along timeline 620,410a fetches image from pel array, and sensor assembly 415 is in the moment 635,655 and 675 along timeline 620, and 410b fetches image from pel array.Because the VBI of the interlacing frames speed 610 of example is greater than the VBI of the interlacing frames speed 605 of example, so the frame rate of the interlacing frames speed 610 of example is less than the frame rate of the interlacing frames speed 605 of example.
VBI about pel array 410a and pel array 401b is configured to 6 time quantums except sensor assembly 415, and the interlacing frames speed 615 of example also is similar to the interlacing frames speed 605 of example.As shown in Figure 6, sensor assembly 415 is in the moment 625,655 and 685 along timeline 620, and 410a fetches image from pel array, and sensor assembly 415 is in the moment 640 and 670 along timeline 620, and 410b fetches image from pel array.The VBI of the interlacing frames speed 615 of example is greater than the interlacing frames speed 605 of example and 610 VBI.Thereby the frame rate of the interlacing frames speed 615 of example is less than the interlacing frames speed 605 of example and 610 frame rate.
B. multi-line
Fig. 7 is another photographic images processing unit (CIPU) 700 of some embodiment of graphic extension conceptually.Except CIPU 700 by two front-end processing pipelines, memory and a back-end processing pipeline realize, rather than outside being realized by single processing pipeline, CIPU 700 realizes and CIPU described above 400 identical functions.Therefore, will the description of the function of CIPU 700 be described with reference to the module of CIPU 400.
As shown in the figure, CIPU 700 comprises the front-end processing pipeline 715 that is used for camera sensor 405a and pel array 410a, the front-end processing pipeline 720 that is used for camera sensor 405b and pel array 410b, memory 725, controller module 730 and back-end processing pipeline 735.The camera sensor 405a of some embodiment and 405b are the transducers of the camera of two camera movement equipment.
The front-end processing pipeline 715 of some embodiment and 720 is carried out the parts of images of CIPU 400 and is handled.Thereby different embodiment can comprise the module of the different numbers of CIPU 400.For example, each front- end processing pipeline 715 and 720 of some embodiment comprise CIPU 400 sensor assembly 415, BPC module 420, LS module 425, remove mosaic module 430, WB module 435 and statistics engine 465.
Although owing to have identical module, front- end processing pipeline 715 and 720 is carried out the image processing of same type, but the different register value by as above illustrating about CIPU 400 can differently dispose each module in each front-end processing pipeline 715 and 720.In addition, because each camera sensor 405a and 405b have its front-end processing pipeline, so front- end processing pipeline 715 and 720 can be handled image independently of each other.For example, front- end processing pipeline 715 and 720 concurrently (that is, in the identical time), the different time with different rate processing images.
In certain embodiments, each front- end processing pipeline 715 and 720 can be fetched image from its corresponding camera sensor and pel array.For example, front-end processing pipeline 715 is fetched the image of being taken by camera sensor 405a from pel array 410a, and front-end processing pipeline 720 is fetched the image of being taken by camera sensor 405b from pel array 410b.When one of front end processing pipeline 715 and 720 when its corresponding camera sensor and pel array are fetched image, this this image of front-end processing pipeline, and the image after handling sent to memory 725.In addition, each front-end processing pipeline 715 communicate by letter with controller module 730 as mentioned above with 720 statistics engine of each front-end processing pipeline (for example, by).
The memory 725 of some embodiment is preserved the image after the section processes, finishes processing for back-end processing pipeline 735.In these embodiments, the image after memory 725 is handled from front end processing pipeline 715 and 720 receiving units, and the image after the section processes sent to back-end processing pipeline 735.Some embodiment are embodied as volatile memory (for example, random-access memory (ram)) to memory 725, and other embodiment is embodied as nonvolatile memory (for example, flash memory, hard disk, CD etc.) to memory 725.In addition, the memory 725 of some embodiment be internal storage (for example, RAM), and the memory 725 of other embodiment be external memory storage (for example, compact flash (CF) card, secure digital (SD) card, or the like).
Some embodiment of back-end processing pipeline 735 carry out the parts of images of CIPU 700 and handle.In certain embodiments, back-end processing pipeline 735 comprises the module of front- end processing pipeline 715 and 720 CIPU 400 not to be covered.For example, referring to top example, back-end processing pipeline 735 should comprise CSC module 445, γ module 440, HSC module 450, scaler module 455 and the filter module 460 of CIPU 400.Thereby the residual image of the CIPU 400 that the back-end processing pipeline 735 execution front- end processing pipelines 715 and 720 of this embodiment are not carried out is handled.Therefore, back-end processing pipeline 735 is fetched image after the section processes from memory 725, and the image after the section processes is carried out remaining image processing.After handling image, back-end processing pipeline 735 sends to CIPU driver 480 to the image after handling.
The identical function that illustrates with reference to figure 4 above controller module 730 is carried out.As shown in Figure 7, controller module 730 and front- end processing pipeline 715 and 720 and back-end processing pipeline 735 mutual.In certain embodiments, controller module 730 is included in the back-end processing pipeline 735, and in other embodiments, controller module 730 is included in one of front- end processing pipeline 715 and 720.
With reference now to graphic extension in Fig. 7, camera sensor 405a and 405b, pel array 401a and 410b, front- end processing pipeline 715 and 720, memory 725 and back-end processing pipeline 735 illustrate the operation of CIPU 700.When one of front end processing pipeline 715 and 720 when its corresponding camera sensor and pel array are fetched image, this image of front-end processing pipeline, and the image after the section processes sent to memory 725.For example, front-end processing pipeline 715 can be fetched the image of being taken by camera sensor 405a from pel array 410a, and perhaps front-end processing pipeline 720 can be fetched the image of being taken by camera sensor 405b from pel array 410b.As mentioned above, front- end processing pipeline 715 and 720 can both the parallel processing image.
Back-end processing pipeline 735 is fetched image after the section processes from memory 725, and the image of processing section after handling is to finish the image processing of image.In certain embodiments, the image that is kept in the memory 725 fetched and handles by back-end processing pipeline 735 based on the principle of first in first out.In other words, specific image in the memory 725 will handled prior to the reception of this specific image and all images that is kept in the memory 725 afterwards, but, this specific image will be handled before will and being kept at image in the memory 725 in the reception that is later than this specific image.After back-end processing pipeline 735 was handled image, it sent to CIPU driver 480 to the image after handling.
Fig. 8 is another photographic images processing unit (CIPU) 800 of some embodiment of graphic extension conceptually.Except CIPU 800 realizes that by two independent processing pipeline each camera sensor has outside its independent processing pipeline, CIPU 800 carries out and CIPU described above 400 identical functions.Thereby, will the description of the function of CIPU 800 be described with reference to the module of CIPU 400.
As shown in the figure, CIPU 800 comprises the processing pipeline 815 that is used for camera sensor 405a and pel array 410a and is used for camera sensor 405b and the processing pipeline 820 of pel array 410b.Each processing pipeline 815 and 820 of some embodiment all comprises all modules that are included among the CIPU 400.So each processing pipeline 815 of these embodiment is all identical with the operation of CIPU 400 with 820 operation.
Because each camera sensor 405a and 405b have its processing pipeline, so processing pipeline 815 and 820 can be handled image independently of each other.For example, processing pipeline 815 and 820 concurrently (that is, in the identical time), the different time with different rate processing images.In addition, by top different register value, can differently dispose each processing pipeline 815 and 820 of some embodiment with reference to CIPU 400 explanations.
In certain embodiments, the number of modules of CIPU 400 comprises one or more row/frame buffers of some or all operations that are used to carry out this module.For example, the filtration module 460 of some embodiment is realized as and carries out 3 * 3 low-pass filtering.In such embodiments, 3 * 3 low pass filters are handled the continuous triplex row in the image, so that row in the middle of the described continuous triplex row is used 3 * 3 low-pass filtering.Thereby the filtration module 460 of this embodiment needs three row/frame buffers at least, so that realize 3 * 3 low-pass filtering.Other module among the CIPU 400 also comprises one or more row/frame buffers, such as BPC module 420 and LS module 425.
The processing pipeline of CIPU 800 all can have different row/frame buffer size, so that according to the characteristic customization image processing of its corresponding camera.For example, if a camera of two camera movement equipment has 2048 * 1500 element sensors, the processing pipeline of this 2048 * 1500 element sensor can comprise row/frame buffer that 2048 pixels are wide so.Similarly, if another camera of two camera movement equipment has 640 * 480 element sensors, the processing pipeline of 640 * 480 element sensors can be included as the wide row/frame buffer of 640 pixels so.That is, the big I that is included in the row/frame buffer in each module of a processing pipeline is different from the size of the row/frame buffer in each module that is included in another processing pipeline.
III. video conference
A. video conference architecture
Fig. 9 is the video conference of two camera movement equipment of some embodiment of graphic extension and the software architecture of processing module 900 conceptually.Video conference and processing module 900 comprise and top respective modules and driver 305,301 and 320 similar CIPU drivers 905, exchange of media module 910 and encoder-driven device 920 with reference to figure 3 explanations.Video conference and processing module 900 also comprise video conference module 925, video conference client 945 and the network interface 950 that is used to realize various video conference functions.Be similar to Video processing and coding module 300, video conference and processing module 900 processing and coding are from the image of the camera of two camera movement equipment.
As above described with reference to figure 3, the user of the media content in the exchange of media module 910 permission equipment and producer's switched-media content are with the instruction relevant with the processing of media content.CIPU driver 905 serves as the communication interface with photographic images processing unit (CIPU) 955, and encoder-driven device 920 serves as the communication interface with encoder hardware 960 (for example, the encoding pack on encoder chip, the SOC (system on a chip), or the like).
The video conference module 925 of some embodiment is responsible for various video conference functions, such as image processing, video conference management and networking.As shown in the figure, video conference module 925 is mutual with exchange of media module 910, video conference client 945 and network interface 950.In certain embodiments, video conference module 925 receives instruction and sends instruction to video conference client 945 from video conference client 945.The video conference module 925 of some embodiment is also by network interface 950, send data and receive data to network from network, for example, described network is Local Area Network, wireless lan (wlan), wide area network (WAN), network of network (a network of networks), code division multiple access access (CDMA) network, GSM network, or the like.
Video conference module 925 comprises image processing layer 930, management level 935 and network layer 940.In certain embodiments, 930 pairs of image of image processing layer carry out image processing operations, for use in video conference.For example, the image processing layer 930 of some embodiment expose adjustment, image zoom, perspective correction and dynamic range adjustment, following further described in detail.The image processing layer 930 of some embodiment sends the request from the image of CIPU955 by exchange of media module 910.
The operation of the management level 935 control of video meeting modules 925 of some embodiment.For example, in certain embodiments, one one/two cameras of the two camera movement equipment of management level 935 initialization are handled image and audio frequency, so that send remote equipment to and handle image and the audio frequency that receives from remote equipment.In certain embodiments, management level 935 produce synthetic (for example PIP) demonstration that is used for this equipment.In addition, management level 935 can change the operation of video conference module 925 according to the networking report that receives from network layer 940.
In certain embodiments, network layer 940 realizes being used for some or all network functions of video conference.For example, as described below, except other function, two camera movement equipment and the network between the remote equipment that the network layer 940 of some embodiment is set up video conference connect (not shown), and image is sent to remote equipment and receives image from remote equipment.In addition, the networking data that network layer 940 receives such as packet loss, unidirectional stand-by period and round trip delay time, and other various data are handled such data and are given management level 935 data report.
The video conference client 945 of some embodiment is to utilize the application of the video conference function of video conference module 925, such as video conference application, IP speech (VOIP) use (for example, Skype) or instant messaging use.In certain embodiments, video conference client 945 is independently to use, and in other embodiments, video conference client 945 is integrated in the Another application.
In certain embodiments, network interface 950 is the communication interfaces that allow video conference module 925 and video conference client 945 to send data by network (for example, cellular network, local area network (LAN), wireless network, network of network, internet etc.) and receive data.For example, if video conference module 925 to another equipment sending data on the internet (is for example wanted, image by the camera of two camera movement equipment), video conference module so 925 sends to described another equipment to image by network interface 950.
B. video conference setting
Figure 10 is the video conference request message sending and receiving sequence 1000 of the example of some embodiment of graphic extension conceptually.Figure 10 represents the video conference client 1010 on the equipment that runs on 1005, videoconference server 1015, and run on video conference request message sending and receiving sequence 1000 between the video conference client 1025 on the equipment 1020.In certain embodiments, video conference client 1010 is identical with the video conference client 945 shown in Fig. 9 with 1025.As shown in Figure 10, an equipment (that is, equipment 1005) request video conference, and another equipment (that is, equipment 1020) is replied this request.Two camera movement equipment of Miao Shuing can be carried out these two operations (that is, request of sending and response request) in this application.
The videoconference server 1015 of some embodiment route messages between the video conference client.Although some embodiment realize videoconference server 1015 on a computing equipment, but other embodiment realizes videoconference server 1015 on a plurality of computing equipments.In certain embodiments, videoconference server is the server that can openly visit, and it can handle the message with the numerous meetings of route simultaneously.Each video conference client 1010 and 1025 of some embodiment is through network interface, such as network interface 950 described above, communicate by letter with videoconference server 1015 by network (for example, cellular network, local area network (LAN), wireless network, network of network, internet etc.).
When the user of video conference client 1010 slave units 1005 received the request of video conference of (in operation 1) beginning and equipment 1020, the video conference request message sending and receiving sequence 1000 of some embodiment began.When the user of equipment 1005 selected to be presented at a certain user interface (UI) project of the user interface on the equipment 1005, the video conference client 1010 of some embodiment received the request of beginning video conference.Illustrate the example of this user interface below among Shuo Ming Figure 11 and Figure 14.
After video conference client 1010 received request, video conference client 1010 sent (in operation 2) video conference request to videoconference server 1015, and this request is designated as the addressee to equipment 1020 according to user's input.Videoconference server 1015 is transmitted to the video conference request video conference client 1025 of (in operation 3) equipment 1020.In certain embodiments, videoconference server 1015 is utilized push technology, and the video conference request is transmitted to video conference client 1025.That is, when receiving request from video conference client 1010, videoconference server 1015 beginnings transmit the video conference request to video conference client 1025, rather than wait for the request that client 1025 sends any message.
When the video conference client 1025 of some embodiment receives the video conference request, explicit user interface on equipment 1020, point out with user to equipment 1020, the user of equipment 1005 has sent the request of beginning video conference, and the user of prompt facility 1020 accepts or refuse this video conference request.Illustrate the example of this user interface below among Shuo Ming Figure 12.In certain embodiments, when video conference client 1025 receives (in operation 4) when accepting request from the user's of equipment 1005 video conference request, video conference client 1025 sends (in operation 5) video conference to videoconference server 1015 and accepts.During a certain user interface items of the user interface of graphic extension, the video conference client 1025 of some embodiment receives the request of accepting video request in the user of equipment 1020 selects as Figure 12.
Videoconference server 1015 received video conference acceptance from video conference client 1025 after, videoconference server 1015 accepted to be transmitted to (in operation 6) video conference client 1010 to video conference.Some embodiment of videoconference server 1015 utilize above-mentioned push technology, and video conference is accepted to be transmitted to video conference client 1010.
When receiving video conference when accepting, some embodiment set up (in operation 7) video conference between equipment 1005 and equipment 1020.Different embodiment differently sets up video conference.For example, the video conference of some embodiment is set up and is comprised: the connection between consulting device 1005 and the equipment 1020, definite to the bit rate of video coding with at equipment 1005 and equipment 1020 exchange videos.
In the above example, the user of equipment 1020 accepts the video conference request.In certain embodiments, equipment 1020 can be configured to (for example, by the preferential selection setting of equipment) and accept the video conference request of arrival automatically, and does not show UI.In addition, the user of equipment 1020 also can refuse (in operation 4) video conference request (for example, by being chosen in certain user interface items of the user interface that shows on the equipment 1020).Replace sending video conference and accept, video conference client 1025 sends the video conference refusal to videoconference server 1015, and videoconference server 1015 is transmitted to video conference client 1010 to the video conference refusal.Thereby do not set up video conference.
1. video conference is provided with user interface
In certain embodiments, according to ongoing conversation, initiate video conference.That is, in the user of mobile device and second user's communication, through the opposing party's permission, this user can become video conference to conversation.Concerning some embodiments of the present invention, the startup of this video conference that Figure 11 graphic extension is undertaken by two camera handheld mobile devices 1100.Figure 11 utilizes five operational phases 1110,1115,1120,1125 and 1130 of the user interface (" UI ") 1105 of equipment 1100, the startup of graphic extension video conference.
As shown in Figure 11, UI 1105 comprises name field 1135, choice menus 1140 and selectable UI project 1145.Name field 1135 is presented at user's name of being intended to ask to carry out with it the personage of video conference of the conversation other end.In this example, selectable UI project 1145 (it can be realized as selectable button) provides selectable end conversation (End Call) option to the user, to finish conversation.Choice menus 1140 shows the menu of selectable UI project, such as speakerphone project 1142, quiet project 1144, numeric keypad project 1146, telephone directory project 1148, maintenance project 1152, video conference project 1154 or the like.Different embodiment differently shows choice menus.Concerning the embodiment of Figure 11 graphic extension, choice menus 1140 comprises several onesize icons, and each icon is represented a different operation.Other embodiment provides rotatable menu, perhaps gives specific project with priority (for example, by making these projects become bigger).
State referring now in five stages 1110,1115,1120,1125 and 1130 of UI 1105 graphic extension in Figure 11 illustrates the operation of UI 1105.In the phase I 1110, between handheld mobile device user and Nancy Jones, set up conversation.Second stage 1115 be presented at the user select selectable video conference option one 154 (for example, by point 1150 singly refer to dub), to activate the UI 1105 after the video conference instrument.In this example, video conference option one 154 (it can be realized as selectable icon) allows the user to start video conference during conversing.In second stage, video conference option one 150 is highlighted, is activated with instruction video meeting instrument.Different embodiment can indicate such selection with different modes (for example, by frame or the text that highlights project).
Phase III 1120 is presented at according to the selection of selecting video conference option one 154, and equipment 1100 has started video conference and handled UI 1105 afterwards.Phase III is the transition maintenance stage of when video conference is set up in the equipment wait (for example, when equipment is waited for the equipment acceptance of the conversation other end or refused video conference).In the phase III 1120, when setting up the video conference connection, the user of equipment 1100 still can talk with the user (that is Nancy Jones) of another equipment.In addition, some embodiment allow the user by selecting of equipment 1100 to be presented at being used on the UI 1105 to cancel the optional UI project (not shown) of video conference request, cancellation video conference request in the phase III 1120.During this maintenance stage, different embodiment uses the difference among the UI 1105 to show and indicates this pending state.
As shown in Figure 11, in certain embodiments, utilize the full screen display of the video of taking by equipment 1100, and in the wait state of " Preview (preview) " symbol graphic extension phase III of the bottom of this video.Specifically, in Figure 11, the phase III 1120, demonstration was presented by the full frame of video of the camera of equipment by in the viewing area 1160 of UI 1105, the beginning that the graphic extension video conference is handled.In certain embodiments, positive camera is when the beginning video conference, the acquiescence camera of choice of equipment.Usually, when the beginning video conference, described front camera is facing to the user of equipment.Therefore, in the example of graphic extension, the phase III 1120 is illustrated as equipment 1100 user's of display device 1100 full screen video in Figure 11.Be located at during the phase III 1120, appear at the wait state of the further outstanding equipment of " Preview " indication under the video in the viewing area 1160.
In certain embodiments, can differently represent cambic the 3rd maintenance stage 1120.For example, some embodiment allow the user of equipment 1100 to select the camera at the back side as the camera that starts video conference.In order to allow this selection, some embodiment allow the user (for example, preferentially select to be provided with by menu) camera at the back side is appointed as the acquiescence camera that starts video conference, and/or allow the user from selecting the user after the video conference option one 154, show the camera of selecting the back side in the menu of the back side and positive camera.In these situations any, UI 1105 (for example, viewing area 1160) shows the video by the camera at the back side during the 3rd maintenance stage 1120.
In addition, other embodiment can be by the less version of display device 1100 video of taking the photograph, be kept at rest image on the equipment 1100 by demonstration, the message of the wait state by the equipment of highlighting is provided (for example, by showing " Conference Being Established " (setting up meeting), by do not show " Preview " indication, or the like, can point out the activation of video conference instrument.In addition, in the phase III 1120, if the UI of some embodiment 1105 provide this stage (for example, when the user waits for that the long-distance user answers his request), user's decision does not enter video conference, allow the user to cancel so and enter video conference, and return end (End) the button (not shown) of talking state.
Quadravalence section 1125 is illustrated in the long-distance user and has accepted the video conference request, and has set up after the video conference connection UI 1105 in an interim state.Under this transition state, show that (size of) viewing area 1160 reduces (that is, dwindling gradually) gradually in this example, by the video of the camera in front, shown in arrow 1175 for local user's video.Viewing area 1160 (that is, local user's video) dwindles, and makes that UI 1105 can be after viewing area 1160, and demonstration comprises the viewing area 1170 (for example, display window 1170) from the video of the camera of remote equipment.In other words, the dwindling of local user's video 1160 produces PIP and shows 1180, PIP show 1180 prospects with video of local user insert show 1160 and long-distance user's background master show 1170.In this example, background master shows that Ms that the 1170 front cameras that present remote equipment are taking its video (for example, Nancy Jones, the user of remote equipment), perhaps the back side camera of remote equipment is being taken Ms's's (for example, Nancy Jones is taking the Ms of its video) of its video video.Those of ordinary skill will appreciate that the cambic quadravalence section shown in Figure 11 is a kind of exemplary method that some embodiment use, and other embodiment is the cambic quadravalence section of flash demo differently.
Quadravalence section 1125 is the optional UI project 1132 of graphic extension in below the viewing area 1155 also.Optionally UI project 1132 (it can be realized as selectable button) shows under 1180 at PIP provides selectable closing session (End Conference) option one 132.The user can select closing session option one 132 to finish video conference (for example, by singly refer to dub).Different embodiment can allow the user with different mode closing sessions, such as by the switch on the conversion mobile device, and by sending voice commands, or the like.In addition, different embodiment can allow closing session option one 132 to fade away during video conference, thereby allows PIP to show that 1180 occupy whole viewing area 1185.When singly referring to dub the bottom of viewing area 1185, closing session option one 132 can reappear subsequently, makes the user can utilize closing session option one 132.In certain embodiments, the layout of viewing area 1155 is identical with the viewing area 1155 that further describes below.
Five-stage 1130 is illustrated in the UI 1105 after the animation that finishes the 4th transition state 1125.Specifically, five-stage 1130 is illustrated in during the video conference, and the PIP that is presented by UI1105 shows 1180.As mentioned above, PIP shows that 1180 comprise that two videos show: insert demonstration 1160 from the bigger background demonstration 1170 of remote camera with from the less prospect of local camera.
PIP shows that 1180 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Except this synthetic view, the equipment of some embodiment provides other synthetic view.For example, the bigger background that replaces having the long-distance user shows 1170, and bigger background shows that 1170 can be the local user, and the insertion of less prospect shows that 1160 is long-distance users.As further described below, some embodiment allow the user during video conference, switch between as the local camera of the camera that provides PIP to show 1180 insertion view and front view and/or remote camera.
In addition, some embodiment allow local video and long-distance video to appear among the UI 1105 two viewing areas (for example, left and right sides display window, perhaps display window up and down) side by side, perhaps in the viewing area that two diagonal angles are arranged.In certain embodiments, as further described below, by the preferential selection setting of equipment, perhaps by during video conference, the control that the user can select, the user can stipulate that PIP shows or the mode of acquiescence display mode.
When the user of the equipment 1100 of Figure 11 invited the long-distance user to carry out video conference, the long-distance user can accept or refuse this invitation.Figure 12 is illustrated in six different stages 1210,1215,1220,1225,1230 and 1235, the UI 1205 of long-distance user's equipment 1200, and described six phase table are shown in the device rendered of long-distance user and accept the sequence of operation that video conference is invited.The description of following UI 1205 equipment 1200 (promptly, receive the equipment of video conference request) the user be called invitee (invite recipient), the user of equipment 1100 (that is, sending the equipment of video conference request) is called inviter (invite requestor).In addition, in this example, the equipment 1200 of supposing the invitee is two camera apparatus, and is the same with inviter's equipment.But, in other example, one or two in these equipment all are single camera apparatus.
Phase I 1210 graphic extension is worked as the invitee from the inviter, the UI 1205 when John Smith receives video conference invitation.As shown in Figure 12, the UI 1205 of phase I comprises name field 1235, message hurdle 1240 and two optional UI projects 1245 and 1250.Name field 1235 shows the personage's who is just asking video conference name.In certain embodiments, name field 1235 shows the personage's who is just asking video conference telephone number, rather than personage's name.Message hurdle 1240 shows that the inviter gives invitee's invitation.In this example, the indication of " Video Conference Invitation (the video conference invitation) " in the message hurdle 1240 inviter is asking to carry out video conference with the invitee.Optionally UI project 1245 and 1250 (they can be realized as selectable button) provides the selectable invitee of confession to be used to " refusal request (Deny Requst) " and " (Accept Request) accepts request " option one 245 and 1250 of refusing or accepting the invitation.Different embodiment can differently show these options and/or show other option.
When seeing " the Video Conference Invitation " symbol that is presented in the message hurdle 1240, by selecting " refusal request " option one 245 or " accepting request " option one 250 among the UI respectively, the invitee can refuse or accept request.Second stage 1215 is illustrated in the example shown in Figure 12, and the user selects " accepting request " option one 250.In this example, the light finger by the user dubs hits " accepting request " option one 250, realizes described selection, and by the highlighting of option one 250, points out described selection.Provide other technology to select " acceptance " or " refusal request " option one 245 and 1250 (for example, doublely click gently, or the like) in certain embodiments to point out described selection (for example, highlighting the frame or the text of UI project).
Phase III 1220 is presented at the invitee and adheres to video conference UI 1205 afterwards.In this stage, UI 1205 enters preview mode, and preview mode shows that in viewing area 1244 the full frame of video from the front camera of remote equipment presents.Front camera in this case is facing to the user (that is the Nancy Jones in this example) of remote equipment.Therefore, the image that shows her by described preview mode.This preview mode makes the invitee can guarantee that her video is correctly shown, and before video conference begins (for example, before beginning actual transmissions video), she pleases oneself to its appearance.In certain embodiments, can under viewing area 1244, show the symbol such as " Preview " symbol, be in preview mode to point out the invitee.
Some embodiment allow camera that the invitee selects the back side acquiescence camera of video conference to start with, perhaps select the camera at the front or the back side when the beginning video conference, and are as further described below.In addition, other embodiment shows that differently invitee's preview shows the less image of the corner that places viewing area 1244 (for example).In addition other embodiment does not comprise this preview mode, but after the invitee accepts request, begins video conference immediately.
In the phase III, UI 1205 shows two optional UI projects 1275 and 1246.One of them overlaps on the viewing area 1244, and another is below viewing area 1244.Optionally UI project 1275 is that the user can select with beginning video conference " acceptance " button 1275.Optionally UI project 1246 is if the invitee does not add video conference in this stage decision, so her selectable " end " button 1246.
Quadravalence section 1225 is presented at the invitee and selects " acceptance " button 1275 UI1205 afterwards.In this example, " acceptance " button 1275 is highlighted, can begin video conference at any time to point out the invitee.Available in other embodiments different mode is pointed out such selection.
Five-stage 1230 is illustrated in the invitee and accepts after the video conference request, UI 1205 in an interim state.In this transition stage, the size of the viewing area 1244 of demonstration invitee's video (in this example, described video is by the camera in front) reduces (that is, dwindling gradually) gradually, shown in arrow 1260.Invitee's video dwindles, and makes that UI 1205 can be after viewing area 1244, and demonstration comprises the viewing area 1265 (for example, display window 1265) from the video of inviter's camera.In other words, the dwindling of invitee's video produces PIP and shows 1280, and PIP shows that 1280 prospects with video of invitee insert viewing area 1244 and inviter's background master and show 1265.
In this example, background master shows that the 1265 front cameras that present local device are taking the man of its video (that is, John Smith, the user's of local device 1100) video.In another example, this video can be man's's (for example, being taken the man of its video by John Smith) the video of the back side its video of camera of local device.The embodiment cambic five-stage of flash demo differently differently.
The UI of five-stage 1230 also shows and is included in during the video conference, make another user audio mute optional UI project 1285 (for example, mute button 1285), the optional UI project 1287 of end video conference (for example, closing session button 1287), viewing area 1155 (for example, tool bar or menu bar) with the optional UI project 1289 of the switching camera that further specifies below (for example, switching camera button 1289).Thereby the invitee can select any optional UI project 1285-1289 (for example, by singly refer to dub), to carry out the operation of wishing during video conference.Different embodiment allows the invitee according to different modes, for example switches the switch on the mobile device, by providing voice commands etc., carries out operation arbitrarily.
Although Figure 12 represents the example layout of viewing area 1155, but, some embodiment provide the different layouts of viewing area 1155, and such as the layout of the viewing area 1155 of Figure 11, this layout includes only the optional closing session UI option one 132 that is used to finish video conference.Other layout of viewing area 1155 can comprise any multiple different optional UI project that is used to carry out difference in functionality.In addition, five-stage 1230 expressions are presented at the viewing area 1155 of the bottom of UI 1205.The different embodiment of viewing area 1155 can be displayed on the diverse location in the UI 1205, and/or are defined as different shapes.
Figure 12 is expressed as static viewing area (that is, viewing area 1155 always is shown) to viewing area 1155.But, in certain embodiments, viewing area 1155 is dynamic display area.In some such embodiment, viewing area 1155 is not shown usually.On the contrary, have only when receiving trigger event (for example, such as dubbing viewing area 1280 once, the user of voice commands and so on selects), just show described viewing area 1155.(for example receiving user's selection, select optional quiet UI project 985), perhaps the time of ormal weight (for example, 3 seconds) is afterwards, viewing area 1155 disappears, and the time of described ormal weight can be provided with appointment by the preferential selection of user by mobile device or video conference application.In some such embodiment, viewing area 1155 is shown automatically after video conference begins and is disappeared according to above mentioned same way as.
The 6th stage 1235 was illustrated in the UI1205 after the animation that finishes the 5th transition stage.Specifically, the 6th stage was illustrated in during the video conference, and the PIP that is presented by UI 1205 shows 1280.As mentioned above, PIP shows that 1280 comprise that two videos show: insert demonstration 1244 from the bigger background demonstration 1265 of local camera with from the less prospect of remote camera.PIP shows that 1280 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Except this synthetic view, the equipment of some embodiment provides other synthetic view.For example, the background that replaces having bigger invitee shows that it can be inviter's video that bigger background shows, less prospect inserts to show it can is invitee's video.As further described below, some embodiment allow the user to control insertion view and the front view of PIP in showing, switchably to show local camera and remote camera.In addition, some embodiment allow local video and long-distance video to appear among the UI 1205 two viewing area (for example, left and right sides display window, perhaps display window up and down) side by side, perhaps in the viewing area that two diagonal angles are arranged.As further described below, by the preferential selection setting of equipment, perhaps by during video conference, the control that the user can select, mode or acquiescence display mode that the user can specify PIP to show.
Although Figure 12 represented to present and accept the sequence of operation that video conference is invited according to six different operational phases, but, some embodiment can realize described operation with the less stage.For example, some such embodiment can omit and present phase III 1220 and quadravalence section 1225, thereby after the user selects " accepting request " option one 250, enter five-stage 1230 from second stage 1215.Realize described operation (promptly with the less stage, present and accept video conference and invite) other embodiment can omit phase I 1210 and second stage 1215, thereby when the invitee when the inviter receives the invitation of video conference, present the phase III 1220 to the user.
Figure 13 graphic extension by first and the phase III be combined into a stage, and second and the quadravalence section be combined into a stage, carry out the example of the operation of graphic extension among Figure 12 with less stage.Especially, Figure 13 is illustrated in five different stages 1390,1392,1394,1230 and 1235, the UI 1205 of long-distance user's equipment 1200.Phase I 1390 is similar to the stage 1110, except name field 1295 display name " Jobn Smith ", to point out outside personage's the name of the conversation other end.That is, between the user of long-range mobile device and the user of local device (that is the John Smith in this example), set up conversation.Second stage 1392 and phase III 1394 are similar to phase I 1210 and the second stage 1215 of Figure 12, also show except second stage 1392 and phase III 1394 outside user's (that is the Nancy Jones in this example) the preview of long-range mobile device.Quadravalence section 1230 is identical with the 6th stage 1235 with the five-stage 1230 of Figure 12 with five-stage 1235.
Except during conversing, activate outside the video conference instrument by selectable option, some embodiment allow the user of two camera apparatus directly to initiate video conference, and needn't at first make a phone call.Another kind of such alternative approach of video conference is initiated in Figure 14 graphic extension.Figure 14 is illustrated in the UI 1405 of seven different phases 1410,1415,1420,1425,1430,1435 and 1440, and described seven different phases show the alternative operation sequence that starts video conference.
In the phase I 1410, the user browses the Affiliates List on the mobile device, seeks the personage that he wishes to carry out with it video conference, and the mode of searching the contact person with him for conversation is similar.In second stage 1415, the personage 1455 (for example, by on this people's name 1455 singly refer to dub 1460) that the user selects him to be ready to carry out with it video conference.This selection triggers information and the various user-selectable options that UI 1405 shows the contact person.In this example, the name 1455 of Jason is highlighted, is the people that the user is intended to carry out with it video conference to point out this.Different embodiment can point out this selection with different modes.Although the user of second stage 1415 permission equipment 1400 passes through Affiliates List, select the user to be intended to carry out with it the people of video conference, but some embodiment allow the user to pass through " Recents (recently) " call history and select this people, and the user that described " Recents " call history is listed equipment 1400 carries out the personage's of video conference or conversation particular number or name recently with it.
In the phase III 1420, after the name 1455 of having selected the personage, UI 1405 shows selected personage's information 1462 and various optional UI project 1468,1472 and 1470.In this example, one of each optional UI project 1472 (can be realized as optional icon or button) provides the video conference instrument.Video conference option one 472 allows the user to invite the personage according to contact person's 1466 identifications to add video conference.Different embodiment is (for example, with different arrangements) display message 1462 and optional UI project 1468,1472 and 1470 differently.
Quadravalence section 1425 expression users select " video conference " option ones 472 (for example, by singly refer to dub).In this example, " video conference " option one 472 is highlighted, has been activated to point out video conference instrument 1472.In different embodiment, can differently point out this selection (for example, text or frame) by highlighting selected icon.
Five, the 6th and the 7th stage 1430,1435 and 1440 was similar to the 3rd, the 4th and five-stage 1120,1125 and 1130 of graphic extension among Figure 11, can be by understanding for the 5th, the 6th and the 7th stage 1430,1435 and 1440 with reference to the discussion in these stages.Briefly, five-stage 1430 graphic extensions wait for that the long-distance user responds the cambic maintenance stage that video conference is invited.The 6th stage 1435 was illustrated in the long-distance user and accepts after the video conference request, the size of (demonstration local user's video) viewing area 1480 reduces gradually, make that UI 1405 can be after viewing area 1480, demonstration comprises the viewing area 1492 from the video of long-distance user's camera.In the 7th stage 1440, during video conference, UI 1405 presents PIP and shows 1447.In certain embodiments, the layout of the viewing area 1155 in the 6th stage 1435 and the 7th stage 1440 is similar to the layout of the viewing area 1155 of Figure 12 described above.
Several modes of video conference are set up in Figure 10,11,12,13 and 14 expressions.In certain embodiments, during conversing, transmit voice data (for example, speech) by a communication port (by the communication network such as circuit exchanging communicating network or packet exchange communication network), and during video conference, transmit voice data by another communication port.Thereby, in such embodiments, before setting up video conference, (for example transmit voice data by a communication port, in a single day speech), and set up video conference, just transmitted audio frequency by different communication port (rather than the communication port that during conversing, uses).
For the seamless transitions of voice data from the video conference of conversing (for example, switch) is provided, some embodiment before setting up video conference, not finished call.For example, some embodiment are in finished call and before beginning to transmit audio/video data by the peer-to-peer communications session, set up reciprocity video conference and connect (for example, in finishing Figure 10 after the message sequence of graphic extension).On the other hand, other embodiment is in finished call and begin to present before the audio/video data of reception, set up reciprocity video conference and connect (for example in finishing Figure 10 after the message sequence of graphic extension), and begin to transmit audio/video data by this peer-to-peer communications session.
The reciprocity video conference of some embodiment connects the directly intercommunication mutually (rather than by for example central server communication) of mobile device that allows in the video conference.Some embodiment of equity video conference allow the mutual shared resource of mobile device in the video conference.For example, control communication port via video conference, by sending instruction to another mobile device from a mobile device, differently handle image (promptly to order described another mobile device, share its image processing resource), than exposure adjustment operation as will be described in further detail below, focal length adjustment operation and/or switch camera operation, the operation of described another mobile device of a described mobile device in can the Long-distance Control video conference.
2. dynamic bit rate setting
In general, mobile device in the video conference is by dissimilar communication networks, such as the communication port of different special uses and public wireless communication network (for example, the cellular network such as GSM, UMTS), mutual Data transmission (for example, Voice ﹠ Video image).Example below with reference to Figure 91 and this cordless communication network of 92 explanations.
Because the number at the mobile device of special time access communications network constantly changes, so the bandwidth that communication network can be used for carrying out video conference is different at different time.Even during video conference, available bandwidth also can change.In addition, flood communication network with high bit rate during video conference, perhaps attempt a large amount of signalings of calculating best video meeting bit rate are worthless.
Owing to these reasons, some embodiment adopt the new method of the initial optimal bit rate of designated meeting.Initial optimal bit rate for the identification video meeting, described method begins video conference with specific bit rate, if these embodiment do not detect the network condition that can worsen the video conference quality, so described method incrementally increases bit rate with specified time interval.
This embodiment of graphic extension example among Figure 15.The processing 1500 of the bit rate of the setting video conference of Figure 15 graphic extension some embodiments of the present invention conceptually.A part that is provided with as video conference (for example, a part of setting up as the video conference of graphic extension among Figure 10) is carried out and is handled 1500, with according to variety of network conditions, dynamically determines to transmit the bit rate of data (for example, Voice ﹠ Video image).In certain embodiments, handling 1500 is carried out by the management level 935 of top video conference module 925 with reference to figure 9 explanations.The more detailed form of this video conference module is described below with reference to Figure 16.
As shown in Figure 15, by bit rate being made as (1505) initial bit rate, begin to handle 1500.In certain embodiments, initial bit rate is the acquiescence reference speed rate of equipment.But, some embodiment allow the user to specify initial bit rate.1505, handle 1500 also by by initial bit rate, by one or more communication ports, transmit data (for example, the Voice ﹠ Video image) to remote equipment, start video conference.
Subsequently, handle the group network conditional parameter that the remote equipment of 1500 identifications (1510) from video conference receives.In certain embodiments, real time transport protocol (RTP) communication session of local device by setting up when beginning video conference receives a described group network conditional parameter from remote equipment.For example, some embodiment provide the network condition parameter by the extension feature of RTP.In addition, by pointing out the existence of extension header in the RTP packet header, and definition is about the extension header of additional information, and the RTP extension feature of some embodiment can be used to transmit the information (for example, a described group network conditional parameter) of any type.
In different embodiment, each equipment in the video conference transmits the different network condition/congestion parameter of many groups.Among Shuo Ming the embodiment, a described group network conditional parameter comprises unidirectional stand-by period and bandwidth estimation bit rate below.In other embodiments, a described group network conditional parameter comprise packet loss data and two-way time (RTT) delayed data.Thereby different embodiment can comprise a plurality of different network condition parameters arbitrarily in a described group network conditional parameter.
In certain embodiments, a group network conditional parameter that receives from the remote equipment of video conference is with by the initial bit rate of setting in operation 1505, during video conference, from local mobile device (promptly, carry out to handle 1500 mobile device) data (for example, Voice ﹠ Video) that send remote equipment to be basic.For example, in certain embodiments, by utilizing the timestamp of audio packet, calculate audio packet and connect by network, propagate into the used time of remote equipment from local mobile device, remote equipment can be determined the unidirectional stand-by period.Specifically, in certain embodiments, each audio packet is stamped timestamp.Do not having under the situation of packetization delay, the setting-up time that remote equipment should equate with the difference of timestamp receives audio packet at interval.But, when having unidirectional latency delays, remote equipment with than big time of the difference of timestamp every receiving audio packet.
In addition, in certain embodiments, remote equipment receives time of video packets by inspection, receives the time of adjacent video grouping and the size of described adjacent video grouping, determines the bandwidth estimation bit rate.That is, receive the time difference of two continuous videos groupings and the size of second video packets and be used to the available bandwidth that estimation network is connected.Some embodiment are many to the continuous videos grouping by checking, determine the bandwidth estimation bit rate.Top example utilizes the data (that is the video data that is used for determining the voice data of unidirectional stand-by period and is used for determining the bandwidth estimation bit rate) of particular type.But, in certain embodiments, also can use the data that connect other type that transmits by the network between local mobile device and the remote equipment.
After identification (1510) described group network condition, handle 1500 and determine subsequently whether (1515) unidirectional stand-by period has deteriorated into the threshold quantity that surpasses definition.In certain embodiments, described threshold quantity is defined as specific stand-by period amount, if the difference between current unidirectional stand-by period and previous unidirectional stand-by period surpasses the specific latency amount, determines that so the unidirectional stand-by period deteriorates into above threshold quantity.In other embodiments, threshold quantity is defined as the specific change rate of unidirectional stand-by period.Thereby (for example, current unidirectional stand-by period and previous unidirectional stand-by period) rate of change surpasses specific rate of change, determines that so the unidirectional stand-by period deteriorates into above threshold quantity when one group of unidirectional stand-by period.
When definite unidirectional stand-by period deteriorates into above threshold quantity, handle 1500 and finish.Otherwise, handle 1500 and determine whether (1520) current bit rate has reached the bandwidth estimation bit rate.In certain embodiments, the amount of bandwidth available (for example, 15kb/s (kbps)) of bandwidth estimation bit rate indication network connection.When handling 1500 definite current bit rate, handle 1500 and finish above the bandwidth estimation bit rate.When handling 1500 definite current bit rate, handle 1500 and enter operation 1525 above the bandwidth estimation bit rate.
1525, handle 1500 and determine whether current bit rate reaches the Maximum Bit Rate of definition.When processing 1500 determines that current bit rate surpass the Maximum Bit Rate of definition, handle 1500 end.Otherwise, handle 1500 and enter operation 1530, current bit rate is increased ormal weight.Different embodiment differently defines the amount that increases bit rate.The example that increases the ormal weight of current bit rate comprises that 32kbps, 64kpbs and multiple arbitrarily other increase the value of bit rate.
Whether subsequently, handle (1535) the official hour amount of determining passes by.The official hour amount can be 1 second, 2 seconds, 5 seconds or any other possible time quantum, because different embodiment differently defines described time quantum.The 1500 wait official hour amounts of handling are gone over, so that allow remote equipment can receive bit rate with new increase (in operation 1530), the data (for example, the Voice ﹠ Video image) that transmit from local mobile device, and determine the network condition parameter according to the bit rate of new increase.If the 1500 definite official hour amounts of handling are not also pass by, handle 1500 returns 1535 so, till the official hour amount in the past.Determine official hour amount past tense when handling 1500, handle 1500 returns 1510.Proceed as mentioned above to handle 1500 since 1510 operation, up to handle 1500 finish till.
When handling 1500 end (, after operation 1515,1520 or 1525), finish the setting of the bit rate of video conference, and determined optimal bit rate.Because the available bandwidth of video conference may change during video conference, therefore, some embodiment continue to adjust bit rate according to the group network conditional parameter (that is, unidirectional stand-by period and bandwidth estimation bit rate) from the remote equipment reception.Can during video conference, adjust bit rate by increasing bit rate.For example, finish if handle 1500 threshold quantities that deteriorate into above regulation because of the unidirectional stand-by period, and during video conference, the unidirectional stand-by period improves, some embodiment increase bit rate so.Similarly, if handle 1500 because of bit rate finishes above the bandwidth estimation bit rate, and during video conference, the bandwidth estimation bit rate increases, and some embodiment increase bit rate so.
On the contrary, can during video conference, adjust bit rate by reducing bit rate.For example, if during video conference, the unidirectional stand-by period continues to deteriorate into the threshold quantity that surpasses regulation, and some embodiment reduce bit rate so.In addition, if during video conference, bit rate continues to surpass bandwidth estimation bit rate (for example, the bandwidth estimation bit rate continues to reduce), and some embodiment reduce bit rate so.
In addition, handling 1500 uses unidirectional stand-by period and bandwidth estimation bit rate to determine whether to increase bit rate.But, those of ordinary skill will appreciate that in different embodiment, can use any multiple network conditional parameter to determine whether to increase bit rate.For example, determining whether to increase bit rate can be only based on RTT delayed data or packet loss data.
C. video conference architecture
As mentioned above, Figure 16 video conference of two camera movement equipment of some embodiment of graphic extension and software architecture of processing module 1600 conceptually.As shown in the figure, video conference and processing module 1600 comprise client application 1665, video conference module 1602, exchange of media module 1620, buffer 1625, photographic images processing unit (CIPU) driver 1630, encoder-driven device 1635 and decoder driver 1640.In certain embodiments, buffer 1625 is to preserve the image of video so that the frame buffer that shows on the display 1645 of two camera movement equipment.
In certain embodiments, client application 1665 is identical with the video conference client 945 of Fig. 9.As mentioned above, client application 1665 can be integrated in the Another application, perhaps is embodied as independently to use.Client application 1665 can be to utilize the application of the video conference function of video conference module 1602, such as video conference application, IP speech (VOIP) use (for example, Skype) or instant messaging use.
The client application 1665 of some embodiment sends instruction to video conference module 1602, instruction such as beginning meeting and closing session, receive instruction from video conference module 1602, instruction from the user of two camera movement equipment is routed to video conference module 1602 and produces and be presented on two camera movement equipment and allow the user and use mutual user interface.
D. video conference manager
As shown in Figure 16, video conference module 1602 comprises video conference manager 1604, image processing manager 1608, networking manager 1614 and buffer 1606,1610,1612,1616 and 1618.In certain embodiments, video conference module 1602 is identical with the video conference module 925 of graphic extension among Fig. 9, thus above carrying out about described some or all the identical function of video conference module 925.
In certain embodiments, video conference manager 1604 is responsible for when video conference begins, some or all of other modules of initialization video conference module 1602 (for example, image processing manager 1608 and networking manager 1614), the operation of control of video meeting module 1602 during video conference, with when video conference finishes, stop the operation of some or all of other modules of video conference module 1602.
The video conference manager 1604 of some embodiment is also handled image that the one or more equipment from video conference receive and by the image of one one or two camera of two camera movement equipment, so that be presented on two camera movement equipment.For example, the video conference manager 1604 of some embodiment is fetched the decoded picture that receives from another equipment that participates in video conference from buffer 1618, with the image of fetching from buffer 1606 by CIPU 1650 processing (that is the image of taking by two camera movement equipment).In certain embodiments, before image being presented on two camera movement equipment, video conference manager 1604 is gone back proportional zoom and composograph.That is, in certain embodiments, video conference manager 1604 produces PIP or other synthetic view, to be presented on the mobile device.The image that some embodiment proportional zooms are fetched from buffer 1606 and 1618, and the image that other embodiment proportional zoom is fetched from one of buffer 1606 and 1618.
Although Figure 16 is illustrated as the part of video conference module 1602 to video conference manager 1604, but some embodiment of video conference manager 1604 are realized as the assembly that separates with video conference module 1602.Thereby single video conference manager 1604 can be used to management and control several video conference modules 1602.For example, some embodiment will on local device, move independently video conference module with Multi-Party Conference in each side mutual, these video conference modules on the local device are by a video conference manager administration and control.
Before image is by encoder 1655 codings, the image that the image processing manager 1608 of some embodiment is handled by the camera of two camera movement equipment.For example, some embodiment of image processing manager 1608 are to the one or more operations in image execution exposure adjustment, focal length adjustment, perspective correction, dynamic range adjustment and the image zoom of CIPU 1650 processing.In certain embodiments, image processing manager 1608 controls send the frame rate of the coded image of another equipment in the video conference to.
Some embodiment of networking manager 1614 manage two camera movement equipment and participate in one or more connections between another equipment of video conference.For example, the networking manager 1614 of some embodiment is set up the connection between another equipment of two camera movement equipment and video conference, and when video conference finishes, is interrupted these connections when video conference begins.
During video conference, networking manager 1614 is another equipment that is sent to video conference by encoder 1655 image encoded, and the image path that another equipment from video conference is received by giving decoder 1660 so that decoding.In certain embodiments, networking manager 1614, rather than image processing manager 1608, control sends the frame rate of image of another equipment of video conference to.For example, some such embodiment of networking manager 1614 are by some coded frame in the coded frame of another equipment that abandons (that is, not transmitting) and should be transmitted to video conference, control frame speed.
As shown in the figure, the exchange of media module 1620 of some embodiment comprises camera source module 1622, video compressing module 1624 and video decompression module 1626.Exchange of media module 1620 is identical with the exchange of media module 310 shown in Fig. 3, and more details is provided simultaneously.Camera source module 1622 is by CIPU driver 1630, route messages and media content between video conference module 1602 and CIPU1650, video compressing module 1624 is by encoder-driven device 1635, route messages and media content between video conference module 1602 and encoder 1655, and video decompression module 1626 is by decoder driver 1640, route messages and media content between video conference module 1602 and decoder 1660.Some embodiment are embodied as the part of camera source module 1622 being included in TNR module 315 (not shown among Figure 16) in the exchange of media module 310, and other embodiment is embodied as TNR module 315 part of video compressing module 1624.
In certain embodiments, the CIPU driver 305 of graphic extension is identical with encoder-driven device 320 among CIPU driver 1630 and encoder-driven device 1635 and Fig. 3.The decoder driver 1640 of some embodiment is served as the communication interface between video decompression module 1626 and the decoder 1660.In such embodiments, decoder 1660 is decoded by another equipment image that receive and that pass through video decompression module 1626 routes of networking manager 1614 from video conference.After image was decoded, image was sent back to video conference module 1602 by decoder driver 1640 and video compressing module 1626.
Except carrying out during the video conference the Video processing, the video conference of two camera movement equipment of some embodiment and processing module 1600 also during video conference, are carried out the Audio Processing operation.The software architecture that Figure 17 graphic extension is such.As shown in the figure, video conference and processing module 1600 comprise video conference module 1602 (it comprises video conference manager 1604, image processing manager 1608 and networking manager 1614), exchange of media module 1620 and client application 1665.Other assembly and the module of video conference shown in Figure 16 and processing module 1600 have been omitted among Figure 17, with simplified illustration.Video conference and processing module 1600 also comprise frame buffer 1705 and 170, Audio Processing manager 1715 and audio driver 1720.In certain embodiments, Audio Processing manager 1715 is realized as independent software module, and in other embodiments, Audio Processing manager 1715 is realized as the part of exchange of media module 1620.
Audio Processing manager 1715 is handled by two camera movement equipment audio data captured, so that send another equipment in the video conference to.For example, Audio Processing manager 1715 receives by microphone 1725 audio data captured by audio driver 1720, to audio data coding, afterwards the voice data of coding is kept in the buffer 1705, so that send described another equipment to.Audio Processing manager 1715 is also handled the voice data of being caught and being received from described another equipment by another equipment in the video conference.For example, Audio Processing manager 1715 is fetched voice data from buffer 1710, and to the voice data decoding, decoded voice data is exported to loud speaker 1730 by audio driver 1720 subsequently.
In certain embodiments, video conference module 1602 is the part of bigger meeting module together with Audio Processing manager 1715 and related buffer thereof.When under the situation that does not exchange video content, when between several equipment, carrying out in many ways audio conferencing, 1600 of video conference and processing modules are utilized networking manager 1614 and Audio Processing manager 1715, so that easier by the audio exchange of Internet protocol (IP) layer.
With reference now to Figure 18,, the operation of the video conference manager 1604 of some embodiment is described.Figure 18 conceptually graphic extension by the video conference manager of some embodiment, the processing of carrying out such as the video conference manager 1604 of graphic extension among Figure 16 1800.This can be equal to by the management level 935 of Fig. 9 carries out.In certain embodiments, when the user of two camera movement equipment (for example accepts, by being presented at the user interface on two camera movement equipment) the video conference request, perhaps when the user of another equipment accepted request that the user of two camera movement equipment sends, video conference manager 1604 was carried out and is handled 1800.
By receiving the instruction of (1805) beginning video conference, begin to handle 1800.In certain embodiments, instruction receives from client application 1665, or receives from the user by the user interface that is presented on two camera movement equipment, and is transmitted to video conference manager 1604 by client application 1665.For example, in certain embodiments, when the user of two camera movement equipment accepts the video conference request, instruction be receive by user interface and transmit by client application.On the other hand, when the user of another equipment accepted from request that local device sends, some embodiment received instruction from client application, and do not have user interface interaction (user interface interaction that but, can have previous transmission initial request).
Subsequently, handle the first mutual module of 1800 initialization (1810) and video conference manager 1604.Some embodiment comprise CIPU 1650, image processing manager 1608, Audio Processing manager 1715 and networking manager 1614 with the mutual module of video conference manager 1604.
In certain embodiments, initialization CIPU 1650 comprises that instruction CIPU 1650 begins to handle the image by one one or two camera of two camera movement equipment.Some embodiment begin to fetch image from buffer 1610 by instruction image processing manager 1608, and the image of handling and encoding and fetch, and initialisation image is handled manager 1608.For initialization Audio Processing manager 1715, some embodiment instruct Audio Processing manager 1715 to begin coding by microphone 1725 audio data captured, be kept at voice data in the buffer 1710 (voice data that receives from another equipment) with decoding, so that export to loud speaker 1730.The initialization of the networking manager 1614 of some embodiment comprises that instruction networking manager 1614 is set up with the network of another equipment in the video conference and is connected.
Handling 1800 determines subsequently whether also surplus (1815) initialized any module of wanting is arranged.When the initialized module of wanting is arranged, handle 1800 returns 1810, when surplus with another module of initialization.When all modules that need all have been initialised, handle 1800 and produce (1820) for the composograph that is presented on two camera movement equipment (that is local display).These composographs can comprise that those composographs of being shown among the Figure 65 that the following describes (promptly, PIP or other are synthetic to be shown), and can comprise from the image of the camera of two camera movement equipment of this locality with from the various combinations of the image of the camera of another equipment of participation video conference.
Next, handle 1800 and determine whether (1825) have made change to video conference.Some embodiment by the user be presented at the mutual of user interface on two camera movement equipment, reception is to the change of video conference, and other embodiment is by networking manager 1614, from the change (that is, Long-distance Control) of another equipment reception to video conference.In certain embodiments, also can be from client application 1665, other module in the perhaps video conference module 1602 receives the change that video conference is provided with.The video conference setting also can change owing to the variation of network condition.
When changing, handle 1800 and determine that whether (1830) are changes to network settings to the change of video conference.In certain embodiments, change of described change or network settings or image taking are provided with change.When the change to video conference is change to network settings, handle and revise (1840) network settings, enter operation 1845 subsequently.The network settings of some embodiment change and comprise the bit rate of change to image encoding, perhaps to the frame rate of another equipment transmitted image.
When the change to video conference is not change to network settings, handles 1800 and determine that described changes are changes that image taking is provided with, thereby enter operation 1835.Handle 1800 and carry out the change that (1835) are provided with image taking subsequently.In certain embodiments, change to the image taking setting can comprise the switching camera (promptly, any camera video on the switch dual camera movement equipment), focal length is adjusted, exposure is adjusted, show or do not show from the image of one one or two camera of two camera movement equipment and amplify or dwindle the image that is presented on two camera movement equipment, and other is provided with change.
In operation 1845, handle 1800 and determine whether to finish video conference.When processing 1800 is determined not finish video conference, handle 1800 returns 1820.When processing 1800 determines that video conferences will finish, handle 1800 and finish.When handle 1800 when client application 1665 receives the instruction that finishes video conference (, receive owing to user interface by local two camera movement equipment, the perhaps instruction that receives from another equipment that participates in video conference), some embodiment of processing 1800 determine to finish video conference.
In certain embodiments, when video conference finished, video conference manager 1604 was carried out and is not shown in the various operations of handling in 1800.Some embodiment instruct CIPU 1650 to stop to produce image, and instruction networking manager 1614 interrupts being connected with the network of another equipment in the video conference and instructing image processing manager 1608 to stop processing and coded image.
E. the time noise reduces
Some embodiment comprise the processing video image, reduce module with the special time noise that reduces the noise in the video.The module that reduces the time noise of some embodiment compares the successive image in the video sequence, eliminates harmful noise with identification with from video.
Figure 19 is the software architecture of this time noise reduction (TNR) module 190 of some embodiment of graphic extension conceptually.Some embodiment are embodied as the part (for example, being embodied as the part of exchange of media module as shown in Figure 3) of application to TNR module 1900, and other embodiment is embodied as TNR module 1900 by other and uses the independent utility of using.Other embodiment in addition is embodied as TNR module 1900 part of the operating system of moving on two camera movement equipment.In certain embodiments, TNR module 1900 is realized that by one group of API described one group of API provides some or all functions of TNR module 1900 to other application.
As shown in Figure 19, TNR module 1900 comprises TNR manager 1905, difference block 1910, the average module 1915 of pixel and motion history module 1920.Although Figure 19 is expressed as independently module to three modules 1910,1915 and 1920, but some embodiment realize the function of these modules that the following describes in individual module.The TNR module 1900 of some embodiment receives input picture, benchmark image and motion history as input.In certain embodiments, input picture is the image of handling at present, and benchmark image is the image formerly that input picture compares with it in the video sequence.TNR module 1900 output output image (input picture that a kind of noise of form reduces) and output movement history.
TNR manager 1905 directs data of some embodiment flowing in TNR module 1900.As shown in the figure, TNR manager 1905 receives input picture, benchmark image and motion history.TNR manager 1905 is also exported output image and output movement history.TNR manager 1905 sends to difference block 1910 to input picture and benchmark image, receives difference image from difference block 1910.
In certain embodiments, difference block 1910 is handled the data that receive from TNR manager 1905, and the data after handling are sent to TNR manager 1905.As shown in the figure, difference block 1910 receives input picture and benchmark image from TNR manager 1905.The difference block 1910 of some embodiment produces difference image by deduct the pixel value of an image from the pixel value of another image.Difference image is sent to TNR manager 1905.Difference between two images of difference image indication of some embodiment is so that the part that has changed of input picture and the part that remains unchanged of input picture are compared in identification with previous image.
TNR manager 1905 also sends to the average module 1915 of pixel to input picture and benchmark image.As shown in the figure, some embodiment also send to the average module 1915 of pixel to motion history.But, other embodiment can only send input picture and benchmark image, and does not send motion history.In any one embodiment, the image that TNR manager 1905 all receives after handling from the average module 1915 of pixel.
The average module 1915 of the pixel of some embodiment utilizes motion history to determine whether about the ad-hoc location in the image, obtains the mean value of pixel from input picture and benchmark image.In certain embodiments, motion history comprises the probable value of each pixel in the input picture.The representative of specific probable value is with respect to the respective pixel in the benchmark image, change the probability of (that is dynamic pixel) of the respective pixel in the input picture.For example, if the probable value of the specific pixel in the input picture is 20, indicate so with respect to the respective pixel in the benchmark image, the probability that the specific pixel in the input picture has changed is 20%.As another example, if the probable value of the specific pixel in the input picture is 0, to indicate so with respect to the respective pixel in the benchmark image, the specific pixel in the input picture does not change (that is static pixels).
Different embodiment differently preserves the probable value of input picture.Some embodiment can be kept at the probable value of each pixel of input picture in the data array.Other embodiment can be kept at (for example, in the group pattern) in the dimension matrix identical with the resolution of the image of video to probable value.For example, if the resolution of the image of video is 320 * 240, matrix also is 320 * 240 so.
When except input picture and benchmark image, also when TNR manager 1905 received motion history, the average module 1915 of pixel read the probable value of each pixel in the input picture to the average module 1915 of pixel.If the threshold value that the probable value of the specific pixel in the input picture is lower than regulation (for example, 5%, 20%), average module 1915 bases of pixel are in this specific pixel so, probably can there be motion, thereby the difference between the image of this pixel may be asked the mean value of the respective pixel value in this particular pixel values and the benchmark image owing to the supposition of noise.
If the probable value of the specific pixel in the input picture is not less than the threshold value of regulation, so the average module 1915 of pixel do not revise input picture specific pixel (that is, the pixel value of this pixel keep with input picture in the same).This is because motion is positioned at this specific pixel probably, thereby the difference between the image is not the result of noise probably.In certain embodiments, when motion history was not sent to the average module 1915 of pixel, the average module 1915 of pixel was asked each pixel in the input picture and the mean value of the respective pixel in the benchmark image.By 1915 outputs of the average module of pixel and the image after being sent to the processing of TNR manager 1905 comprise not by the input image pixels value of average any pixel with by the average pixel value of the average any pixel of the average module of pixel 1915.
In certain embodiments, motion history module 1920 is handled the data that receive from TNR manager 1905, and gives TNR manager 1905 the result data loopback.The motion history module 1920 of some embodiment receives input picture and motion history from TNR manager 1905.Some embodiment are this data input Bayes estimator, so that produce the new motion history (that is one group of probable value) that can use in average about the pixel of next input picture.Other embodiment uses other estimator to produce new motion history.
With reference now to Figure 20,, the operation of TNR module 1900 is described.Figure 20 is the processing 2000 of the time noise of the image of the reduction video of some embodiment of graphic extension conceptually.Receive (2005) input picture, benchmark image and motion history from TNR manager 1905 and begin to handle 2000.Input picture is the image of handling for noise reduction at present.In certain embodiments, benchmark image is the previous image the image sequence of the video that receives from CIPU.But, in other embodiments, benchmark image is the output image (that is the output of TNR module 1900) by the processing generation of previous input picture.Motion history is the output movement history by the processing generation of previous input picture.
When input picture was first image of video, the TNR module 1900 of some embodiment was not handled first image (that is, not to first image applications TNR).In other words, TNR manager 1905 receives first image, and just exports first image.In other embodiments, when input picture was first image of video, first image was used as input picture and benchmark image, TNR module 1900 these images of processing as described below.In addition, when input picture was first image of video, motion history was empty (that is, null value is zero entirely, or the like), and TNR manager 1905 is the empty motion history of output just, as output movement history.
TNR manager 1905 determines subsequently whether (2010) input picture is static.Determine that in order to make this some embodiment send to difference block 1910 to input picture and benchmark image, receive difference image from difference block 1910.When the difference between two images was lower than the threshold value (for example, 5% difference, 10% difference etc.) of regulation, some embodiment were categorized as still image to input picture.
When input picture is still image, TNR manager 1905 sends to the average module 1915 of pixel to input picture and benchmark image, with the pixel of calculating (2015) input picture mean value with the pixel of benchmark image, so as from still image any noise of minimizing.Processing subsequent enters the operation 2040 that the following describes.
When input picture was not still image, the TNR manager sent to the average module 1915 of pixel to input picture, benchmark image and motion history so that handle.The pixel that the average module 1915 of pixel is selected in (2020) input picture.By utilizing motion history, the average module 1915 of pixel determines whether the probability of motion of (2025) pixel is lower than certain threshold level, as mentioned above.
If the probability of selected pixel is lower than certain threshold level, the average module 1915 of pixel is calculated the described pixel of (2030) input picture and the mean value of the respective pixel in the benchmark image so.Otherwise this pixel is not averaged, and output image will be identical with the input picture in described specific pixel.The average module 1915 of pixel determines subsequently whether surplus (2035) in input picture any unselected pixel.If have also not processed any pixel, handle return 2020 so to select next pixel.The average module 1915 executable operations 2020-2030 of pixel are till having assessed all pixels.
Processing subsequent is upgraded (2040) motion history.As shown in Figure 19 and as mentioned above, motion history module 1920 is upgraded motion history according to input picture.New motion history is together with being exported by the TNR manager from image after the processing of the average module of pixel.
F. Tu Xiangchuliguanliqi ﹠amp; Encoder
Except the time noise reduction and image processing operations carried out by CIPU and/or CIPU driver, some embodiment carry out various image processing operations at the image processing layer 930 of video conference module 925.These image processing operations can comprise the adjustment of exposure adjustment, focal length adjustment, perspective correction, dynamic range and image zoom or the like.
Figure 21 graphic extension conceptually carries out the processing 2100 of such image processing operations.In certain embodiments, some or all operations of processing 2100 are carried out by the image processing manager 1608 of Figure 16 and the combination of encoder-driven device 1635.In some such embodiment, the processing (for example, convergent-divergent, dynamic range adjustment, perspective correction etc.) that image processing manager 1608 is carried out based on pixel.Some embodiment carry out processing 2100 to the image that will be transmitted to another equipment that participates in video conference during video conference.
With reference now to Figure 16,, illustrates and handle 2100.By fetch (2105) image from buffer 1606, begin this processing.In certain embodiments, the image of fetching is the image (that is an image in the image sequence) of video.This video can be by the camera of carrying out the equipment of processing 2100 thereon.
Subsequently, handle 2100 pairs of images of fetching and carry out (2110) exposure adjustment.Some embodiment are by being presented at the user interface on two camera movement equipment, and adjustment exposes.Operation is adjusted in the example exposure of this embodiment of Figure 22 graphic extension.
The three phases 2210,2215 and 2220 of the UI 2205 of Figure 22 reference device 2200 illustrates exposure adjustment operation.Phase I 2210 graphic extension comprises the UI 2205 of viewing area 2225 and viewing area 1155.As shown in the figure, viewing area 2225 shows the man's that the sun and face and health are all black dull image 2230.Black dull face and body shows that this man does not obtain appropriate exposure.Image 2230 can be the video image by the camera of equipment 2200.As shown in the figure, viewing area 1155 comprises the optional UI project 2250 that is used to finish video conference.In certain embodiments, the layout of viewing area 1155 is identical with the layout of the viewing area 1155 of Figure 12 described above.
The a certain zone of the user by selecting viewing area 2225 of second stage 2215 graphic extension equipment 2200 is initiated exposure and is adjusted operation.In this example, select by the realization Anywhere that finger 2235 is placed in the viewing area 2225.In certain embodiments, the user selects exposure adjustment from the menu that possible image setting is adjusted.
Phase III 2220 is presented at finishes after the exposure adjustment operation described man's image 2240.As shown in the figure, image 2240 is similar to image 2230, but the man in the image 2240 is by appropriate exposure.In certain embodiments, the image of appropriate exposure is the image of taking after the image of incorrect exposure.The exposure by the successive image of the camera of equipment 2200 is adjusted in the exposure adjustment operation of initiating in second stage 2215.
Return Figure 21, handle 2100 and subsequently image is carried out (2115) focal length adjustment.Some embodiment carry out the focal length adjustment by being presented at the user interface on two camera movement equipment.Figure 23 is the example of this focal length adjustment operation of graphic extension conceptually.
Three different phases 2310,2315 and 2320 of the UI 2305 of Figure 23 reference device 2300, graphic extension focal length adjustment operation.Phase I 2310 graphic extension comprises the UI 2305 of viewing area 2325 and viewing area 1155.Viewing area 2325 presents the blurred picture 2330 by the man of the camera of equipment 2300.This fuzzy this man's of explanation image 2330 misalignment focuses.That is, when this man's of camera image 2330, the lens of camera are not focused in this man.In addition, image 2330 can be the video image by the camera of equipment 2300.As shown in the figure, viewing area 1155 comprises the optional UI project 2350 that is used to finish video conference.In certain embodiments, the layout of the viewing area 1155 of the viewing area 1155 of the layout of viewing area 1155 and Figure 12 described above is identical.
Focal length adjustment operation is initiated in a zone of the user by selecting viewing area 2325 of second stage 2315 graphic extension equipment 2300.In this example, select by the realization Anywhere that finger 2335 is placed in the viewing area 2225.In certain embodiments, the user selects the focal length adjustment from the menu that possible image setting is adjusted.
Phase III 2320 is presented at finishes after the focal length adjustment operation described man's image 2340.As shown in the figure, image 2340 is identical with image 2330, but the man in the image 2340 seems more clear.The lens of this explanation camera are gathered in this man rightly.In certain embodiments, the appropriate image that focuses on is the image of taking after the image of incorrect focal length.The focal length by the successive image of the camera of equipment 2300 is adjusted in the focal length adjustment operation of initiating in second stage 2315.
Return Figure 21, handle 2100 pairs of images and carry out (2120) image zoom.Some embodiment carry out image zoom to image, are used for bit number (that is, reduce bit rate) to image encoding with minimizing.In certain embodiments, handle 2100 and carry out image zoom, as following with reference to as described in the figure 26.
Handle 2100 and subsequently image is carried out (2125) perspective correction.In certain embodiments, handle 2100 and carry out perspective correction, following described in Figure 24.Such perspective correction relates to the data of utilization by the orientation of the two camera movement equipment of identification and one or more accelerometers that move and/or gyro sensor acquisition.Described data are used to revise image subsequently, to proofread and correct incorrect perspective.
After image is carried out perspective correction, handle the dynamic range of 2100 adjustment (2130) images.In certain embodiments, the dynamic range of image is that each pixel in the image can have the scope of probable value.For example, the image with dynamic range of 0-255 can be adjusted to the scope of 0-128, perhaps any other number range.The dynamic range of adjusting image can reduce the quantity (that is, reducing bit rate) that will be used for the bit of image encoding, thereby makes image smoothing.
The dynamic range of adjusting image also can be used for various other purposes.A purpose is to reduce picture noise (for example, image is taken by noisy camera sensor).In order to reduce noise, can adjust the dynamic range of image, make black level be redefined, to comprise somber (that is the black of pulverizing (crush black)).In this way, the noise of image is reduced.Another purpose of dynamic range adjustment is to adjust one or more colors or color gamut, so that strengthen image.For example, some embodiment can suppose that the image of positive camera is the image of people's face.Therefore, can adjust the dynamic range of this image, to strengthen redness and pink, make people's cheek seem ruddy/more ruddy.Dynamic range adjustment operation also can be used for other purpose.
At last, handle 2100 and determine that (2135) are used for the one or more rate controller parameters to image encoding.In certain embodiments, such rate controller parameter can comprise quantization parameter and frame type (for example, predictive frame, bidirectional frame, intracoded frame).This processing subsequent finishes.
Although handling each operation of 2100 is illustrated according to specific order and carries out, but, those of ordinary skill will appreciate that the many operations (exposure adjustments, focal length adjustment, perspective correction etc.) in these operations can carry out according to random order, and has nothing to do each other.That is, the processing of some embodiment can be carried out the focal length adjustment before exposure is adjusted, perhaps can similarly revise the processing of graphic extension among Figure 21.
1. perspective correction
As mentioned above, some embodiment carried out perspective correction to image before demonstration or transmitted image.In some cases, on two camera movement equipment one one or multi-section camera can complete face to its subject, and in uncorrected image, object is known from experience the distortion that seems.Can use perspective correction to handle image, make image will reflect verily that subject in the image is in people's appearance.
Figure 24 summarizes the image processing manager of ground graphic extension by some embodiment, handles 2400 such as the perspective correction that the image processing manager of graphic extension among Figure 16 is carried out.The processing 2400 of some embodiment is carried out by the image processing layer 930 shown in Fig. 9 (it can comprise image processing manager 1608).Some embodiment carry out in the operation 2125 of handling 2100 and handle 2400, so that before showing or transmitting the video image of taking recently, and the perspective of proofreading and correct described image.
Handle 2400 from autoacceleration flowmeter sensor reception (2405) data, in certain embodiments, described accelerometer sensor is the part of two camera movement equipment.The accelerometer sensor measurement of some embodiment is along the rate of change (that is the acceleration of equipment) of the speed of one or more equipment.This is handled also and receives (2410) data from gyro sensor, and in certain embodiments, described gyro sensor also can be the part of two camera movement equipment.The gyroscope of some embodiment and accelerometer sensor can be used to discern the orientation of two camera movement equipment either alone or in combination.
Subsequently, handle 2400, determine the perspective correction amount that (2415) will carry out according to data from accelerometer and gyro sensor acquisition.Usually, when described orientation when axle is farther, in order to produce best image, need more perspective correction.Some embodiment calculate the distortion parameter (warp parameter) of representing the perspective correction amount according to the orientation of equipment.
After determining the perspective correction amount that to carry out, handle 2400 and receive (2420) image by the camera of two camera movement equipment.Can be to carrying out described processing by each image in the video sequence of this camera.Some embodiment can carry out independent calculating to each image from two cameras on two camera movement equipment respectively.
Described processing subsequent is revised (2425) image according to the perspective correction amount of determining.Except other expression of distortion parameter or perspective correction amount, some embodiment also use baseline image or out of Memory (for example, user's input should about its point of proofreading and correct).After revising image, handle 2400 and finish.
Figure 25 example image of some embodiment of graphic extension conceptually handles operation.Figure 25 graphic extension is by first image processing operations 2505 of first image processing module, 2520 execution of not using perspective correction and second image processing operations of being carried out by second image processing module 2565 that uses perspective correction 2550.
As shown in the figure, first image 2510 to the block 2515 that produced by the angled aerial prospective of watching block downwards carries out first image processing operations 2505.Compare with the bottom of block 2515, the top of block is nearer from described perspective.Thereby block 2515 seems towards the camera tilt of taking first image 2510.Figure 25 has also represented first image 2525 after the processing after being handled by first image processing module 2520.As shown in the figure, identical reprocessing appears in the block 2515 in first image 2525 after the processing, because first image processing module 2520 does not carry out any perspective correction.
Second image 2555 to block 2560 carries out second image processing operations 2550.Block 2560 is identical with block 2515 in first image 2510.Figure 25 also is illustrated in second image 2575 after the processing after the processing that perspective correction device 2570 by second image processing module 2565 carries out second image 2555.Perspective correction device 2570 can use handles 2400, so that proofread and correct the perspective of second image 2555.According to the data that become downward angle tilt from accelerometer with the gyrostatic camera of pointing out to take second image 2555 (may also according to other data), perspective correction device 2575 can be proofreaied and correct second image, make it seems that block seem to watch as the crow flies in second image 2575 after processing.
2. convergent-divergent and bit stream are handled
Image zoom and bit stream are handled the row that belong in the above the function of carrying out with reference to the image processing layer 930 by some embodiment of Figure 21 explanation.Image zoom (2130 carrying out in operation) relates to that ratio is in certain embodiments amplified or ratio downscaled images (that is, revising the number of the pixel that is used to represent this image).In certain embodiments, bit stream is handled and is related in the data insertion bit stream of the size of image behind the indication convergent-divergent.In certain embodiments, this convergent-divergent and bit stream are handled and are carried out by encoder-driven device (for example, driver 1635).
Figure 26 is the software architecture of this encoder-driven device 2600 of some embodiment of graphic extension conceptually, and shows example convergent-divergent and the bit stream processing operation of being undertaken by 2600 pairs of example image of encoder-driven device 2605.In certain embodiments, image 2605 is the images by the video of another equipment camera of two camera movement equipment, that be used for sending to video conference.Referring to Figure 16, in certain embodiments, video image propagates into buffer 1606 by CIPU driver 1630 and camera source module 1622 from CIPU 1650, and image processing manager 1608 is fetched described video image from buffer 1606.Afterwards, image is sent to encoder-driven device 1635 by buffer 1610 and video compressing module 1624 through image processing (for example, focal length adjustment, exposure adjustment, perspective correction) in image processing manager 1608.
As shown in the figure, encoder-driven device 2600 comprises processing layer 2610 and rate controller 2645.Illustrate the example of the rate controller of some embodiment below among Shuo Ming Figure 30.Processing layer 2610 comprises image sealer 2615 and bit stream manager 2625.In certain embodiments, these modules are carried out various operations to image before image is encoded and afterwards.Although in this example, image sealer is expressed as the part of the processing layer 2610 of encoder-driven device 2600, but some embodiment are embodied as image sealer the part (that is, carrying out image zoom before image and size data are sent to the encoder-driven device) of image processing manager 1608 rather than encoder-driven device 2600.
As shown in the figure, before image through-rate controller 2645 is sent to encoder 2650, image sealer 2615 zoomed images.Image 2605 is sent out by scaler 2615 and is dwindled into image 2630 by ratio.Except the ratio downscaled images, some embodiment can also the ratio enlarged image.
As shown in Figure 26, the image that some embodiment ratios are dwindled input (for example, image 2605), the image that ratio is dwindled (for example subsequently, 2630) (for example overlap the spatial redundancy image, image 2635) on, the size of described spatial redundancy image (just as usually saying) identical with input picture (that is, the line number of the pixel of image 2605 is identical with the line number and the columns of the pixel of spatial redundancy image 2635 with columns).The upper left corner that the image 2630 that some embodiment dwindle ratio overlaps the spatial redundancy image (as shown in the figure, to produce composograph 2640), and the doubling of the image that other embodiment dwindles ratio to the different piece of spatial redundancy image (for example, central authorities, the upper left corner, directly over, under, lower right or the like).
In certain embodiments, the spatial redundancy image is all to be a kind of color (for example, black, blueness, redness, white etc.) basically, perhaps has the image of repeat patterns (for example, grid, striped etc.).For example, the spatial redundancy image 2635 shown in Figure 26 has the cruciform pattern of repetition.Because the cause of repeatability, the spatial redundancy part of composograph 2640 can easily be compressed into a spot of data by encoder.In addition, if a series of image is all dwindled by ratio, and each image in sequence, the spatial redundancy image of use is all identical, and time compression can be used to further be reduced to the amount that presents the needed data of coded image so.
Some embodiment of image sealer 2615 also produce the size of indicating scaled images, and (size data 2620 of for example, the size of the image 2630 of proportional zoom) size data 2620, and handle generation sends to bit stream manager 2625.The size data 2620 of some embodiment is according to the columns (that is, height and width) of the line number and the pixel of the pixel of scaled images 2630, the size of indication scaled images 2630.In certain embodiments, size data 2620 is also indicated the position of scaled images 2630 in composograph 2640.
After image was scaled, composograph 2640 through-rate controllers 2645 were sent to encoder 2650.In certain embodiments, following further described in detail, rate controller 2645 controls are by the bit rate (that is size of data) of the image of encoder 2650 outputs.Encoder 2650 compression and the coded images of some embodiment.Encoder 2650 can use H.264 coding or another kind of coding method.
The bit stream manager 2625 of some embodiment receives the bit stream of one or more coded images from encoder 2650, and size data is inserted in the bit stream.For example, in certain embodiments, bit stream manager 2625 receives size datas 2620 from image sealer 2615, and size data 2620 is inserted from the bit stream 2655 of the coding composograph 2640 of encoder 2650 receptions.In this case, the output of bit stream manager 2625 is the amended bit streams 2660 that comprise size data 2620.The diverse location that different embodiment inserts size data 2620 in the bit stream 2655.For example, bit stream 2660 shows that size data 2620 is inserted in the beginning of bit stream 2660.But, other embodiment size data 2620 be inserted in bit stream 2655 ending, be inserted in the centre of bit stream 2655 or be inserted in any other position in the bit stream 2655.
In certain embodiments, bit stream 2655 is the bit streams that comprise a series of one or more coded images of composograph 2640.In some such embodiment, the image in this series all is scaled to identical size, and the size of size data 2620 these scaled images of indication.Be transmitted to after the equipment of the video conference other end at image, receiving equipment can extract size information from bit stream, and the image that utilizes described size information correctly to decode and receive.
Figure 27 graphic extension conceptually handles 2700 by the encoder-driven device of two camera movement equipment such as the image zoom that driver 2600 is carried out.Handle 2700 from receiving (2705) image (for example, image 2605) beginning by the camera of two camera movement equipment.When two camera apparatus during with two camera images, some embodiment handle 2700 to carrying out from the image of two cameras.
Subsequently, handle the image that 2700 convergent-divergents (2710) receive.As mentioned above, different embodiment zoomed image 2605 differently.For example, the image 2605 among Figure 26 is dwindled by ratio and is superimposed on spatial redundancy image 2635, thereby produces composograph 2640.
Handle 2700 and subsequently scaled images (for example, comprising the composograph 2640 of scaled images 2630) is sent to (2715) encoder 2650, so that coding.Handle some embodiment of 2700 rate controller of the bit rate of scaled images 2630 (being included in the composograph 2640) by determining the encoder encodes image is sent to encoder 2650.The encoder 2650 of some embodiment compresses and coded images (for example, utilizing discrete cosine transform, quantification, entropy coding or the like), and the bit stream that has coded image is returned to encoder-driven device 2600.
Subsequently, handling 2700 indicates the data (for example, size data 2620) of the size of scaled images to send to (2720) bit stream manager.As shown in Figure 26, in certain embodiments, (that is, a module in the encoder-driven device 2600 sends to another module in the encoder-driven device 2600 to size data) carried out in this operation in encoder-driven device 2600.
After scaled images is by encoder 2650 codings, handle 2700 from encoder reception (2725) bit stream.As shown in the figure, some embodiment receive described bit stream at the bit stream manager that has received size data in addition.The bit stream that receives comprises the composograph of coding, also can comprise the one or more other image in the video sequence.
Handle 2700 and subsequently the data (for example, size data 2620) of the size of indication scaled images are inserted in the bit stream, finish subsequently.As shown in Figure 26, in certain embodiments, this operation is also carried out by the bit stream manager.As mentioned above, different embodiment are the different piece of size data insertion bit stream.In the example of graphic extension, size data 2620 is inserted in the place that begins of bit stream 2655, as shown in resulting bit stream 2660.Now, this bit stream can be transmitted to another equipment that participates in video conference, and at described another equipment, this bit stream can be decoded and be watched.
In certain embodiments, decoder driver (for example, driver 1640) is carried out the reverse functions of encoder-driven device.That is, the bitstream extraction size data of decoder driver from receiving sends bit stream to decoder, and utilizes the image of size data convergent-divergent decoding.Figure 28 is the software architecture of this decoder driver 2800 of some embodiment of graphic extension conceptually, and expression is handled and zoom operations by the example bit stream that the bit stream 2825 of 2800 pairs of examples of decoder driver carries out.
In certain embodiments, bit stream 2825 is by the bit stream of the coded image of the video of the camera of an equipment in the video conference (for example to comprise, and be transmitted to the equipment that decoder driver 2800 is run on bit stream from the encoder-driven device such as driver 2600).Referring to Figure 16, in certain embodiments, this bit stream is received by networking manager 1614, and is sent to buffer 1616, and described bit stream is fetched by video decompression module 1626 from buffer 1616, and is sent to decoder driver 1640.
As shown in the figure, decoder driver 2800 comprises processing layer 2805.Processing layer 2805 comprises image sealer 2810 and bit stream manager 2820.In certain embodiments, these modules 2810 and 2820 before image is decoded and are afterwards carried out various operations to the image that receives.Although in this example, image sealer 2810 is expressed as the part of the processing layer 2805 of decoder driver 2800, but, some embodiment are embodied as image processing manager 1608 to image sealer, rather than the part of decoder driver (promptly, after decoder driver 2800 sends image, carry out image zoom).
As shown in the figure, the bit stream manager of some embodiment 2820 receives one or more coded images (that is) bit stream, the image in the video sequence, and extract size data from this bit stream sends to decoder 2835 to bit stream afterwards and decodes.For example, as shown in Figure 28, bit stream manager 2820 receives the bit stream 2825 of a coded image, extracts size datas 2815 from bit stream 2825, and a bit stream 2830 (not having size data 2815) that obtains is sent to decoder 2835 decodes.As shown in the figure, in certain embodiments, bit stream manager 2820 sends to image sealer 2810 to the size data 2815 that extracts.
The size data 2815 of some embodiment is identical with the size data 2620 that is inserted in the bit stream by encoder-driven device 2600.As above described in the explanation of Figure 26, the size data 2815 of some embodiment is indicated the size of subimage 2845 according to the columns of the line number and the pixel of the pixel of subimage 2845.Size data 2815 also can be indicated the position of subimage 2845 in bigger spatial redundancy image 2840.In this example, bit stream 2825 shows the size data that begins to locate 2815 that is inserted in bit stream 2825.But, as mentioned above, different embodiment inserts size data 2815 diverse location of bit stream 2825.
The image sealer 2810 of some embodiment is utilized the size data that receives from bit stream manager 2820, extracts subimage from image.For example, Figure 28 graphic extension image sealer 2810 receives the image 2840 that comprises subimage 2845 from decoder 2835.As shown in the figure, the image sealer 2810 of some embodiment is extracted subimage 2845 from image 2840.The image that extracts can be displayed on two camera movement equipment subsequently.
Figure 29 is the decoder driver by the equipment that participates in video conference of some embodiment of graphic extension conceptually, and the image of carrying out such as driver 2800 extracts processing 2900.By receiving the bit stream (for example, bit stream 2825) of (2905) coded image, begin this processing.Described bit stream can send another equipment that carries out video conference from the equipment that is run on decoder driver, perhaps can be stored in the memory of described equipment.When described equipment received image from a plurality of sources, some embodiment handled 2900 to the image from each source.
Subsequently, handle 2900 from bitstream extraction (2910) size data.As mentioned above, can find described size data by the diverse location in bit stream.Some embodiment know and seek described size data wherein, and other embodiment seeks specific markers, and described mark points out that bit stream that described size data is arranged in reception where.In certain embodiments, size data is pointed out the size (for example, pixel count in each row and the pixel count in each row) and the position of subimage in coded image of subimage.
Handle 2900 and subsequently the size data that extracts is sent to (2915) image sealer.As shown in Figure 28, in certain embodiments, (that is, a module in the decoder driver sends to another module in the decoder driver to size data) carried out in this operation in decoder driver.
Handle 2900 and also bit stream is sent to (2920) decoder, so that decoding.In certain embodiments, decoder decompresses and decoding bit stream (for example, utilizing inverse discrete cosine transform, inverse quantization etc.), and the image of reconstruct is returned to decoder driver.
After the decoded device decoding of bit stream, handle 2900 from decoder reception (2925) decoded picture.As shown in the figure, some embodiment are at the image sealer reception decoded picture that also receives size data from the bit stream manager.The size data that the processing subsequent utilization receives extracts (2930) subimage from decoded picture.As shown in the figure, as shown in the size data 2815, extract subimage 2845 from the upper left corner of decoded picture 2840.Now, the subimage of extraction can be displayed on the display device (for example, the screen of two camera movement equipment).
3. rate controller
In certain embodiments, two of equipment cameras have different many groups characteristics.For example, in certain embodiments, the front camera is the low resolution camera of optimizing for the shooting of motion video image, and back side camera is the high-resolution camera of optimizing for the shooting of rest image.Because a variety of causes such as cost, function and/or the geometry of equipment, other embodiment can use the various combination of the camera of different qualities.
Camera with different qualities can be introduced different pseudomorphisms.For example, compare with the low resolution camera, the high-resolution camera can show more noise.With compare by the image of low resolution camera, can show higher levels of space or time complexity by the image of high-resolution camera.In addition, the different cameras with different optical character can be introduced different γ values in the image of taking.The different sensitization mechanism that is used for photographic images by different cameras also can introduce different pseudomorphisms.
In the distinctive pseudomorphism of these cameras some are hidden the pseudomorphism that is produced by other source.For example, in the image of being taken by the high-resolution camera with high-level noise, not too as seen the pseudomorphism of the accessory substance of handling as video coding becomes.When coding noise (such as quantizing distortion) was hidden in after the distinctive pseudomorphism of camera, video coding was handled and can be used bigger quantization step to obtain less bit rate.On the other hand, when camera is introduced less pseudomorphism (such as with regard to the low resolution camera), video coding is handled can utilize more tiny quantization step, so that avoid the unacceptable vision distortion level that causes by quantizing.Thereby, handle with the video coding of ignoring the distinctive characteristic of these cameras and to compare, be optimized to the video coding that utilizes or compensate the distinctive characteristic of these cameras and handle that can to obtain better rate-distortion compromise.
Compromise for the distinctive characteristic of these cameras being used to realize rate-distortion, some embodiment realizations are handled with respect to two kinds of video codings that each camera of two cameras is optimized respectively.Figure 30 graphic extension has the example of the system that handles about two video codings of two cameras 3060 and 3070.As shown in Figure 30, system 3000 comprises encoder-driven device 3010, rate controller 3020 and 3040 and video encoder 3030.Encoder 3030 is becoming bit stream 3080 and 3090 from video camera 3060 with 3070 encoding video pictures of taking.
In certain embodiments, video encoder driver 3010 is software modules of moving on one or more processing units.It provides other assembly of video encoder 303 and system, such as the interface between video camera, image processing module, network management module and the storage buffer.The video image that encoder-driven device 3010 control is taken flowing from camera and image processing module to video encoder 3030, it also provides coded bit stream 3080 and 3090 pipelines to storage buffer and network management module.
As shown in Figure 30, encoder-driven device 3010 comprises two different instances 3020 and 3040 of rate controller.These a plurality of examples can be two different rate controllers that are used for two different cameras, or about two different cameras, a rate controller that disposes according to two kinds of different modes.Specifically, in certain embodiments, these two rate controllers 3020 and 3040 are represented two independent rate controllers.On the other hand, in other embodiments, these two rate controllers 3020 and 3040 are two kinds of different configurations of single rate controller.
Figure 30 also shows the encoder-driven device 3010 that comprises status buffer 3015, and status buffer 3015 is kept at the encoding state information of the rate controlled operation of using during the video conference.Specifically, in certain embodiments, these two different rate controllers, perhaps two of identical rate controller kinds of differences are configured in during the video conference, share the identical encoding state information that is kept in the status buffer 3015.Unified rate controller operation in the capture video meeting frequently of this shared permission double vision of state information.This sharing also allows in the meeting of single video capture video, switch best video coding during the camera operation (that is, allowing to manipulate the encoding state information of operating maintenance) by about the rate controlled of the coding of video that last camera is taken the photograph about the rate controlled of the coding of video that current camera is taken the photograph.Figure 30 is expressed as the part of encoder-driven device 3010 to status buffer 3015, but other embodiment can realize status buffer 3015 outside encoder-driven device 3010.
In status buffer 3015, different embodiment preserves the data of different types of representing encoding state information (for example, dissimilar coding parameter).The current goal bit rate that an example of this encoding state information is video conference.A kind of mode of recognition objective bit rate has been described in the III.B joint above.Other example of this encoding state information comprises bit rate and other encoding state information of the frame of buffer fullness, largest buffer degree of filling, one or more nearest codings.
Rate controller can use target bit rate (perhaps being kept at another encoding state parameter in the status buffer) to calculate one or more parameters of using subsequently in its rate controlled operation.For example, as further described below, the rate controller of some embodiment uses the current target bit rate to come the quantization parameter QP of computing macro block or frame.For example, some embodiment utilize the current target bit rate to calculate quantification and adjust parameter, and these embodiment are from quantizing to adjust the quantization parameter QP that parameter obtains macro block and/or frame.Therefore, during the camera handover operation in video conference, sharing target bit rate between (two rate controllers, perhaps two of a rate controller kinds of different configurations) two kinds of rate controlled operations allows encoding state data to the operation of rate controlled formerly of the video coding of last camera are benefited from the rate controlled operation of the video coding of current camera.
Figure 30 graphic extension comprises the encoder-driven device 3010 of two different rate controller examples 3020 and 3040.But, in other embodiments, these rate controller examples 3020 and 3040 are built in the video encoder 3030.Video encoder 3030 is becoming digital bit stream 3080 and 3090 by camera 3060 with 3070 encoding video pictures of taking.In certain embodiments, video encoder produces the bit stream meet conventional video encoding standard (for example, H.264 MPEG-4).In some such embodiment, video encoder is carried out the encoding operation that comprises estimation, discrete cosine transform (" DCT "), quantification and entropy coding.Video encoder also carries out the decode operation with the functions reversed of encoding operation.
In certain embodiments, encoder 3030 comprises the quantizer module 3032 that is used to quantize.Quantizer module is by quantization parameter 3022 or 3042 controls from rate controller 3020 or 3040.In certain embodiments, each quantization parameter is set by the rate corresponding controller, and is the function of one or more attributes of the camera relevant with this rate controller, and is as further described below.Rate controller can reduce the bit number that is used to encode by setting bigger quantization step, perhaps increases employed bit number by setting littler quantization step.By the control quantization step, rate controller also determines can introduce how many distortions in the video image of coding.Thereby rate controller can be realized trading off between bit rate and the picture quality.When realizing that rate-distortion is compromise, rate controller Monitor bit (n.) rate is not so that can overflow storage buffer, underflow storage buffer or surpass the transmission channel capacity.The also necessary control bit rate of rate controller is so that the unacceptable image quality distortion that best possible picture quality is provided and avoids causing by quantizing.In certain embodiments, each rate controller is kept at the monitor data according to one group of status data value representation in the status buffer 3015.In certain embodiments, to use the distinctive attribute of cameras to optimize rate-distortion compromise for rate controller 3020 and 3040.
In certain embodiments, each rate controller is by directly using correction factor to its quantization parameter, and it is compromise to optimize rate-distortion.In some such embodiment, correction factor is predetermined and is built in the equipment together with camera; Equipment does not need these correction factors of dynamic calculation.In other embodiments, system's utilization is dynamically determined the suitable correction factor specific to camera by the input picture of camera.In some such embodiment, system analyzes a series of inputted video image by camera in repeatedly encoding, to collect some statistics about camera.System utilizes these statisticss to obtain the correction factor of the quantization parameter optimized about this camera subsequently.
In certain embodiments, the distinctive correction factor of these cameras is applied to quantization parameter via the visual masking attribute of video image.The visual masking attribute of image or a part of image is the indication that can tolerate the pseudomorphism of how much encoding in described image or image section.Some embodiment calculate the visual masking attribute of the luminance energy that quantizes described image or image section, and other embodiment calculates the described image of quantification or the activation energy of image section or the visual masking attribute of complexity.With how to shelter attribute irrelevant for computation vision, and some embodiment utilize the visual masking attribute to calculate the correction of a frame of video or shelter quantization parameter.Some such embodiment are calculated as frame level visual masking attribute sheltering quantization parameter With benchmark visual masking attribute
Figure BSA00000402183700722
Function.In certain embodiments, use the video masking attribute
Figure BSA00000402183700723
With
Figure BSA00000402183700724
The quantization parameter of revising is expressed as:
Figure BSA00000402183700725
MQP wherein FrameBe frame shelter or revise quantization parameter, QP NomBe initial or nominal quantized value, and β FrameIt is the constant that is suitable for local statistics.In certain embodiments, benchmark visual masking attribute With nominal quantization parameter QP NomBe predetermined according to the initial or periodic evaluation of network condition.
In certain embodiments, the visual masking attribute in the equation (1)
Figure BSA00000402183700732
Be calculated as
Figure BSA00000402183700733
Wherein avgFrameLuma is the average brightness value of frame, and avgFrameSAD is the mean value and the number of the absolute difference of frame.Constant alpha, β, C, D and E are suitable for local statistics.In certain embodiments, these constants are suitable for the distinctive characteristic of camera.
Some embodiment also calculate certain part of video image, such as the quantization parameter of sheltering of macro block.In these cases, shelter quantization parameter and be calculated as macro block visual masking attribute
Figure BSA00000402183700734
Function:
Figure BSA00000402183700735
Wherein, in certain embodiments, β MBBe the constant that is suitable for local statistics, MQP FrameUtilize equation (1) and (2) to calculate.In certain embodiments, the visual masking attribute in the equation (3) Be calculated as
Figure BSA00000402183700737
Wherein avgMBLuma is the average brightness value of macro block, and avgMBSAD is the mean value and the number of the absolute difference of macro block.Constant alpha, β, A, B and C are suitable for local statistics.In certain embodiments, these constants are suitable for the distinctive characteristic of camera.
Be better than and utilize the distinctive constant of a plurality of cameras to calculate the quantization parameter of correction as mentioned above, some embodiment calculate quantization parameter by only utilizing the distinctive coefficient of single camera, realize the rate controlled specific to camera.For example, known visual masking attribute
Figure BSA00000402183700738
With
Figure BSA00000402183700739
And quantization parameter QP Frame, some embodiment utilize the distinctive coefficient μ of single camera to come the quantization parameter of computing macro block:
Figure BSA000004021837007310
For calculation equation (5), some embodiment are used separately as the visual masking attribute to the complexity measure of frame and macro block
Figure BSA000004021837007311
With
Figure BSA000004021837007312
Some embodiment are at QP MBCalculating in use the different distinctive coefficients of camera.For example, in certain embodiments, QP MBBe calculated as
Figure BSA00000402183700741
Wherein ρ is the coefficient of adjusting according to the distinctive characteristic of camera.
As mentioned above, status buffer 3015 is kept at during the video conference, and two different rate controller examples 3020 and the 3040 encoding state information that can share are so that obtain better coding result from their rate controlled operation.In certain embodiments, target bit rate R TIt is an example of this state shared information.This bit rate is the desirable bit rate to the series of frames coding.In general, this bit rate is with bps for unit representation, and is according to determining such as the top processing of describing in the III.B joint.
As mentioned above, the rate controller of some embodiment frame and/or the macroblock quantization parameter QP that utilize target bit rate to calculate it to export to video encoder 3030.For example, some embodiment utilize the current target bit rate to calculate quantification and adjust parameter, and these embodiment adjust the quantization parameter QP that parameter obtains macro block and/or frame from the quantification of calculating.In certain embodiments, quantize to adjust parameter and be and utilize by the bit rate of former frame, perhaps the runtime average of the bit rate of the former frame fraction representation that calculates divided by the current target bit rate.In other embodiments, described adjustment parameter is not an accurate Calculation in this manner, and (1) runtime average and (2) of being proportional to the bit rate of the bit rate of former frame or former frame are inversely proportional to the current target bit rate on the contrary in general.
Calculating after such quantification adjusts parameter, the rate controller of some embodiment utilizes its macro block and/or frame quantization parameter of being calculated of this parameter adjustment.A kind of mode of carrying out this adjustment is that macro block that calculates and/or frame quantization parameter be multiply by quantification adjustment parameter.The another kind of mode of carrying out this adjustment is according to quantizing to adjust calculation of parameter offset quantizing parameter value, macro block and/or the frame quantization parameter (for example deducting) that this offset parameter is applied to calculate subsequently.The rate controller of these embodiment is exported to video encoder 3030 to macro block and/or the frame quantization parameter adjusted subsequently.
In other embodiments, rate controller utilizes target bit rate to calculate other parameter of using in its rate controlled operation.For example, in certain embodiments, rate controller utilizes the visual masking intensity of this target bit rate correction macro block or frame.
G. manager networks
Figure 31 is the networking manager 3100 of some embodiment of graphic extension conceptually, such as the software architecture of the networking manager 1614 of graphic extension among Figure 16.As mentioned above, networking manager 3100 management two camera movement equipment that it ran on and the network between the remote equipment in the video conference connect (for example, connect set up, connect monitoring, connect and adjust, connect interruption etc.).During video conference, the networking manager 3100 of some embodiment is also handled the data that send remote equipment to and is handled the data that receive from remote equipment.
As shown in Figure 31, networking manager 3100 comprises session negotiation manager 3105, transmitter module 3115, general transmission buffer 3120, general transmission buffer-manager 3122, virtual transportation protocol (VTP) manager 3125, receiver module 3130 and medium transfer management device 3135.
Session negotiation manager 3105 comprises protocol manager 3110.Protocol manager 3110 guarantees that during video conference transmitter module 3115 uses correct communication protocol to transmit the rule of the communication protocol of data and the use of pressure obedience to remote equipment.Some embodiment of protocol manager 3110 support various communications protocols, such as real time transport protocol (RTP), transmission control protocol (TCP), User Datagram Protoco (UDP) (UDP) and HTTP (HTTP), or the like.
Connection between one or more remote equipments that session negotiation manager 3105 is responsible for setting up two camera movement equipment and participating in video conference, and after meeting, interrupt these connections.In certain embodiments, session negotiation manager 3105 also be responsible for to set up between the remote equipment in two camera movement equipment and the video conference (for example, transmission and receiver, video and/or audio stream) multimedia communication sessions (for example, utilizing Session Initiation Protocol).
Session negotiation manager 3105 also receives feedback data from medium transfer management device 3135, according to feedback data, by general transmission buffer-manager 3122, determines the operation (for example, be transmit or abandon packet/frame) of general transmission buffer 3120.In certain embodiments, this feedback can comprise unidirectional stand-by period and bandwidth estimation bit rate.In other embodiments, feedback comprises packet loss information and round trip delay time (for example, according to the grouping that sends to the remote equipment in the video conference and determine from the reception of the affirmation of remote equipment).According to information from medium transfer management device 3135, session negotiation manager 3105 can determine whether sending too much grouping, with instruction general transmission buffer-manager 3122, allow general transmission buffer 3120 transmit less grouping (that is adjustment bit rate as shown in Figure 15).
Transmitter module 3115 from video buffer (for example, the buffer 1612 of Figure 16) (for example fetches coded image, and, send the remote equipment in the video conference to form with bit rate), to the image subpackage so that by general transmission buffer 3120 and virtual transportation protocol manager 3125.Producing coded image and coded image is sent to the mode of transmitter module 3115 can be based on the instruction or the data that receive from medium transfer management device 3115 and/or session negotiation manager 3105.In certain embodiments, subpackage relates to the bit stream that receives is divided into a component group to image, each grouping all has specific size (promptly, by the size of session negotiation manager 3105) according to the specific protocol appointment, with the header that increases any needs (for example, address header, agreement are specified header or the like).
General transmission buffer-manager 3122 is controlled the operation of general transmission buffer 3120 according to the data and/or the instruction that receive from session negotiation manager 3105.For example, can instruct general transmission buffer-manager 3122 order general transmission buffers 3120 to transmit data, stop to transmit data, abandon data or the like.As mentioned above, in certain embodiments, when the remote equipment of conference participation seems lost packets,, can discern this situation according to the affirmation that receives from remote equipment.In order to reduce packet loss, can instruct general transmission buffer-manager 3122 to transmit grouping to remote equipment with lower speed.
General transmission buffer 3120 is preserved the data that receive from transmitter module 3115, and by VTP manager 3125 these data is sent to remote equipment.As mentioned above, according to the instruction that receives from general transmission buffer-manager 3122, general transmission buffer 3120 discardable groupings (for example, the image of video).
In certain embodiments, RTP is used to during video conference by UDP Data transmission grouping (for example, audio packet and video packets).Other embodiment utilizes RTP to divide into groups by the TCP Data transmission during video conference.In different embodiment, also can use other transport-layer protocols.
Some embodiment define the specific communications passage between two mobile devices with pair of end slogan (that is, source port number and destination port numbers).For example, a communication port between the mobile device can be with the pair of end slogan (for example, source port 50 and destination port one 00) definition, and another the different communication port between the mobile device can define different port numbers (for example, source port 75 and destination port one 50) with another.Some embodiment also utilize a pair of Internet protocol (IP) address to define communication port.In certain embodiments, different communication ports are used to transmit the data of different types grouping.For example, can in independent communication port, transmit video data packets, audio data packet and the grouping of control signaling data.Thereby, video communication channel transfer video data packets, and audio communication channel transmits audio data packet.
In certain embodiments, the control communication port is used for the information receiving between the local mobile device and remote equipment during the video conference.The example of this information receiving comprises and sends and receive request, notice and to the affirmation of described request and notice.Another example of information receiving comprises from an equipment to another equipment transmission remote control commands message.For example, by control communication port through local device, send the instruction of the operation of Long-distance Control remote equipment to remote equipment from local device, (for example can carry out the remote control operation that the following describes, commander equipment only sends the image from particular camera, perhaps only utilizes the particular camera photographic images).Different embodiment utilizes different agreements, realizes described control communication such as transmitting control protocol (RTCP), RTP expansion, SIP etc. in real time.For example, some embodiment utilize the RTP expansion to transmit one group of control messages in video conference between two mobile devices, and use and transmit another group control messages during SIP is grouped in video conference between described two mobile devices.
The VTP manager 3125 of some embodiment allows by single communication passage (for example, utilizing identical pair of end slogan), transmits and specifies the data of different types grouping that transmits by different communication port (for example, utilize different many to port numbers).Be used for a kind of technology of this purpose and relate to the recognition data packet type, the port numbers recognition data grouping of a pair of appointment by extracting packet is prescribed will be by the communication port of its transmission, with come the specified data grouping will be by described single communication channel transfer (that is, all packet be transmitted) by identical pair of end slogan by the pair of end slogan that the pair of end slogan of packet is modified as described single communication passage.
In order to understand a pair of initial port number of every kind of packet, some embodiment preserve the mapping with respect to a pair of initial port number of data packet type.Some such embodiment utilize the packet type field of agreement to distinguish subsequently and are multiplexed into a different grouping in the communication port.For example, some embodiment with VTP manager are multiplexed to audio frequency, video and control grouping in the rtp streaming, utilize the RTP packet type field to distinguish the audio frequency of another equipment in the video conference that is transmitted to, video and control grouping in a RTP passage.In some such embodiment, the VTP manager is also given described another equipment the control messages route in the SIP grouping.
Some embodiment discern and check packet signature (that is, the packet header form), utilize the different grouping (for example, distinguishing grouping that utilizes the RTP transmission and the grouping that utilizes SIP to transmit) of different agreement transmission with differentiation.In such embodiments, after the packet of determining different agreement, check the field of the packet (for example, utilizing voice data and the video data of RTP) of using same protocol as mentioned above, to discern different data types.In this manner, VTP manager 3125 transmits the predetermined different pieces of information grouping that transmits by different communication ports by the single communication passage.
Although a kind of mode by single communication combination of channels data of different types has been described above, but other embodiment utilizes other technology that different packet types is multiplexed in the communication stream.For example, a kind of technology of some embodiment relates to a pair of initial port number of being careful packet, and described a pair of initial port number is kept in the packet itself, so that extract after a while.Also exist the data of different types between two video conferencing participants is combined to a port to other technology in the passage.
When VTP manager 3125 by the virtual communication passage, when remote equipment received packet, VTP manager 3125 was checked the different grouping that the signature of packets utilizes different agreements to send with identification.This signature can be used to distinguish SIP grouping and RTP grouping.The VTP manager of some embodiment also utilizes the packet type field of some or all groupings, and multichannel is decomposed the various dissimilar grouping (for example, audio frequency, video and control grouping) that is multiplexed in the single virtual passage.After these dissimilar groupings of identification, the VTP manager associates each dissimilar grouping and its corresponding port check mark according to the port check mark of its preservation and the mapping of packet type.VTP 3125 revises the pair of end slogan of packet subsequently with the pair of end slogan of identification, and transmits packet so that unpack.Utilizing different technology different packet types to be multiplexed among other embodiment in the single passage, the VTP manager uses different technology to resolve grouping.
Such technology is multiplexed decomposes different groupings with multichannel by utilizing, VTP manager 3125 (is for example created single virtual communication port, single to port numbers), transmit video data, voice data and control signaling data by described single virtual communication port, and receive audio frequency, video and control grouping from remote equipment by described single virtual communication port.Thereby, viewpoint from network, data are to transmit by described single virtual communication port, and from the viewpoint of session negotiation manager 3105 and protocol manager 3110, video data, voice data and control signaling data are to transmit by different communication ports.
Be similar to the image that sends the remote equipment in the video conference to, the image that the remote equipment from video conference transmits receives by packet format.Receiver module 3130 receives grouping, and grouping is unpacked so that reconstructed image is kept at image in the video buffer (for example, the buffer 1616 of Figure 16) so that decoding afterwards.In certain embodiments, image is unpacked to relate to remove any header and only have the bit stream of view data (having size data) by grouping reconstruct.
Medium transfer management device 3135 (is for example handled the feedback data that receives from network, unidirectional stand-by period, bandwidth estimation bit rate, packet loss data, round trip delay time data, or the like), with the speed (that is bit rate) of dynamically and adaptively adjusting transfer of data.In some other embodiment, medium transfer management device 3135 is also fault-tolerant according to the feedback data control after handling, also can send to video conference manager 1604 to feedback data,, adjust and coding such as proportional zoom, size so that adjust other operation of video conference module 1602.Except when the remote equipment in the meeting can not be handled all whens grouping, the general transmission buffer is abandoned outside the grouping, video conference module and encoder can use than low bit rate image encoding, make will send less grouping concerning each image.
In certain embodiments, but how medium transfer management device 3135 other variable of watch-dog also disposes the operating power mode consumption and the thermal level of camera such as the meeting influence, as mentioned above.These data also can be used as the additional input (for example, if equipment just overheats, medium transfer management device 3135 can manage to make to handle and slow down so) that adds in the feedback data.
With reference now to Figure 16,, several exemplary operations of networking manager 3100 is described.At first explanation sends the image by the camera of two camera movement equipment in the video conference remote equipment, illustrates afterwards from remote equipment to receive image.Transmitter module 3115 is fetched and will be transmitted to the coded image of the remote equipment the video conference from buffer 1612.
The protocol manager 3110 definite appropriate agreements that will use (for example, transmitting the RTP of Voice ﹠ Video), session negotiation manager 3105 is given transmitter module 3115 this notice of settlement.Afterwards, 3115 pairs of images of transmitter module carry out subpackage, and the image of subpackage is sent to general transmission buffer 3120.General transmission buffer-manager 3122 receives instruction from session negotiation manager 3105, transmits or abandon image with order general transmission buffer 3120.VTP manager 3125 receives grouping from general transmission buffer 3120, and handles grouping, so that by the single communication passage grouping is sent to remote equipment.
When from remote equipment reception image, VTP manager 3125 receives the image of subpackage by virtualized single communication passage from remote equipment, and handle grouping in case the communication port that is used for receiving image by distribution (for example, the video communication passage), image is directed to receiver module 3130.
3130 pairs of groupings of receiver module unpack with reconstructed image, and image is sent to buffer 1616, so that by decoder 1660 decodings.Receiver module 3130 also is transmitted to medium transfer management device 3130 (for example, the affirmation of dividing into groups from the reception of the remote equipment in the video conference) to the control signaling message.
Several exemplary operations of networking manager 3100 have been described above.These are illustrative example, because various other embodiment will utilize different modules, perhaps under various functions differently are distributed in situation between the module, carry out these operations or different operations.In addition, module or other module of networking manager 3100 can be carried out other operation, such as the dynamic bit rate adjustment.
IV. adjust and control operation in the meeting
A. picture-in-picture is revised
1. lock onto corner (snap-to-corner)
Some embodiments of the present invention allow the user of two camera movement equipment to constitute the synthetic one or more viewing areas that show by moving around, and revise the synthetic demonstration that is presented on the equipment.Such example is the insertion viewing area that PIP shows that moves around.Figure 32 is illustrated in such example of carrying out during the video conference.In video conference, because a variety of causes has blocked the area-of-interest of background viewing area such as inserting the viewing area when prospect, the user may want mobile prospect to insert the viewing area.
Figure 32 is with reference to five different phases 3210,3215,3220,3225 and 3230 of UI 3205, and viewing area 3240 moving in the UI 3205 of equipment inserted in graphic extension.Phase I 3210 is illustrated in the UI 3205 during the video conference between the long-distance user of the local user of this equipment and remote equipment.UI 3205 among Figure 32 shows that the PIP that shows shows that identical PIP shows with after the beginning video conference in the five-stage of Figure 11.In this example, the video of being taken by local user's equipment is displayed in the insertion viewing area 3240, and the video of being taken by long-distance user's equipment is displayed in the background viewing area 3235.As shown in the figure, viewing area 1155 comprises the optional UI project 3245 that is used to finish video conference.In certain embodiments, the layout of viewing area 1155 is identical with the layout of the viewing area 1155 of Figure 12 described above.
Second stage 3215 graphic extension user by selecting are inserted viewing area 3240, start to lock onto the corner operation.In this example, by finger 3255 is placed on any place of inserting in the viewing area 3240, realize selecting.As shown in the figure, the thick frame 3260 with inserting viewing area 3240 shows this selection.Different embodiment can point out such selection with different modes, such as by highlighting district 3240, swings or the like by making viewing area 3240.
Phase III 3220 is illustrated in the user and begins PIP is shown that 3250 insertion viewing area 3240 shows that from PIP a zone 3250 moves to this PIP and shows UI 3205 after another zone in 3250.In this example, insert the lower right corner that viewing area 3240 has begun to move to from the lower left corner of PIP demonstration 3250 PIP demonstration 3250, shown in arrow 3265.In this example, select to insert after the demonstration in second stage 3215, the user drags its finger 3255 by show 3250 the lower right corner towards PIP, moves and inserts viewing area 3240.Some embodiment are provided at other technology that PIP shows the insertion viewing area 3240 that moves around in 3250.
Quadravalence section 3225 is illustrated in the UI 3205 under the state of removing on the screen of user's slave unit 3200 after its finger 3255.In this state, inserting viewing area 3240 still moves towards point the lower right corner of moving the PIP demonstration 3250 that is identified according to the user in the phase III 3220.In other words, after insertion viewing area 3240 was moved in the lower right corner of PIP demonstration 3250, even removed finger 3255, UI 3205 still kept this and moves in finger 3255 beginnings.Move in order to keep described, before the user removed its finger 3255, the UI 3205 of some embodiment required user's drag operation greater than specific threshold quantity (for example, greater than specific distance, perhaps being longer than the specific time); Otherwise these embodiment can make insertion viewing area 3240 remain on its initial position, the lower left corner after moving insertion viewing area 3240 a little, perhaps move and insert viewing area 3240.
But, although even the user had stopped his drag operation before the insertion viewing area arrives its reposition, some embodiment still allow to insert the viewing area to be continued to move, but other embodiment requires the user to keep drag operation, till the insertion viewing area arrives its reposition.Some embodiment provide other technology of inserting the viewing area that moves.For example, some embodiment require the user before inserting that viewing area 3240 is actual and beginning to move, and where appointment will be directed to inserting viewing area 3240, or the like.Some embodiment are also by only with different angles inclination mobile device, can be slided in the viewing area and lock onto the corner.
Five-stage 3230 is illustrated in and inserts the reposition UI 3205 afterwards that viewing area 3240 arrives its lower right corner that is positioned at PIP demonstration 3250.In five-stage 3230, the elimination of thick frame 3260 indication locks onto the corner operation and finishes.
For make the above-mentioned the 3rd, the 4th and five- stage 3220,3225 and 3230 in graphic extension move easier, in case the UI 3205 of some embodiment adopts the user that the insertion viewing area is moved towards certain corner of PIP demonstration 3250, just allows to insert the locking discipline (snapping rule) that the viewing area quick lock in arrives this corner.For example, when dragging towards particular corner, the user inserts viewing area 3240 when surpassing threshold quantity, the direction of motion of viewing area 3240 is inserted in UI 3205 identifications of some embodiment, determine that described motion has surpassed threshold quantity, thereby automatically move subsequently and insert viewing area 3240, and further import the next grid point that inserts among the UI 3205 that viewing area 3240 can be locked into without the user.In certain embodiments, the unique grid point that provides for locking insertion viewing area 3240 is the grid point that is positioned at four corners of PIP demonstration 3250.Other embodiment provides other grid point of inserting (for example, PIP shows in 3250) among the UI 3205 that viewing area 3240 can be locked into (that is, insert the side of viewing area 3240 or the summit can be placed in it on or other grid point of aliging with it).
Additional embodiments can not adopt grid point, makes the insertion viewing area can be placed in the arbitrfary point in the PIP demonstration 3250.Again additional embodiments provide allow that the user opens or closes UI lock onto the grid point feature.In addition, except the video that slave unit is taken, different embodiment can allow the user to various projects, locks onto the corner operation such as execution such as icons.
Two other example 3330 and 3335 that locks onto the corner operation among Figure 33 graphic extension UI 3205.These are other locks onto the corner operation and shows and vertical or diagonal angle drag operation according to the user show the insertion viewing area 3240 of being moved by vertical or diagonal angle in 3250 at PIP.
Insert viewing area moving in PIP shows even Figure 32 and 33 illustrates, can utilize similar techniques mobile viewing area in the synthetic demonstration of the PIP of other type demonstration or other type but those of ordinary skill will appreciate that other embodiment.For example, as further described below, the PIP of some embodiment shows to have two or more prospect insertion demonstration, by utilization and top technology similar techniques with reference to Figure 32 and 33 explanations, can mobile these insertions show in the PIP demonstration.In addition, some embodiment utilize the similar techniques viewing area (for example, drag mobilely by the user, a viewing area is moved to the right side of screen from the left side of screen) that moves around in synthetic the demonstration.In addition, the mobile meeting of one or more viewing areas of synthetic demonstration causes the variation of the image processing operations of two camera movement equipment, such as response user's input, makes the viewing area in the synthetic again demonstration of video conference manager 1604.As further described below, some embodiment adopt when second viewing area by when the 3rd position moves to primary importance, locking and push technology that first viewing area is pushed open from primary importance.
2. rotation
When the user of the mobile device that is used for video conference rotated described mobile device in the session, some embodiment were rotated in the PIP that presents during the video conference and show.Figure 34 graphic extension is when making equipment 3400 when the upright position rotates to horizontal level, the rotation of the UI1105 of equipment 3400.When the long limit of screen when being vertical, equipment 3400 is held by perpendicular, and when the long limit that shields be level the time, equipment 3400 is held by horizontal.In the example of graphic extension, UI 1105 is from the perpendicular horizontal transverse views of optimizing of holding of holding vertical view rotation becoming equipment 3400 of optimizing for equipment in Figure 34.This spinfunction makes is held or during horizontal holding, the user can watch the UI 1105 with the stand up position demonstration by perpendicular when mobile device 3400.
Figure 34 is according to six different operational phases 3410,3415,3420,3425,3430 and 3435, the rotation of graphic extension UI 1105.Phase I 3410 is illustrated in the UI1105 during the video conference between the long-distance user of the local user of this equipment and remote equipment.UI 1105 among Figure 34 shows that with after setting up video conference the PIP that shows shows that identical PIP shows 1180 in the five-stage of Figure 11.In this example, the video that local user's equipment is taken is presented at and inserts in the viewing area 1160, and the video that long-distance user's equipment is taken is presented in the background viewing area 1170.In showing viewing area 1155 under 1180, PIP comprises that the user can select to finish the optional UI project 3485 (for example, " closing session (End Conference) " button 3485) of video conference (for example, by singly refer to dub).
Second stage 3415 is illustrated in the user and begins banking equipment 3400 UI 1105 afterwards.In this example, the user begins equipment 3400 is held from erecting to hold to tilt to by horizontal, shown in arrow 3460.The outward appearance of UI 1105 does not change.In other cases, the user may change into want equipment 3400 from horizontal hold to tilt to by perpendicular hold, in these cases, UI1105 switches to the view of vertical optimization from the view of level optimization.
Phase III 3420 is illustrated in equipment 3400 from being tilted to by the UI under the state after horizontal the holding 1105 by perpendicular holding.In this state, the outward appearance of UI 1105 does not still change.In certain embodiments, making equipment 3400 tilt to surpass threshold quantity and keeping surpassing threshold quantity after a period of time, trigger rotary manipulation.In Figure 34, in the example of graphic extension, suppose that threshold quantity and rotary speed can not cause UI 1105 rotations, till the short period interval after equipment is placed on horizontal level.Different embodiment has different threshold quantities and the stand-by period that is used to trigger rotary manipulation.For example, some embodiment can have the threshold value of so low triggering rotary manipulation, make UI 1105 to be seemed the not orientation of tube apparatus 3400 always be shown with stand up position as it.In other embodiments, the user of equipment 3400 can stipulate when to trigger rotary manipulation (for example, preferentially selecting to be provided with by menu).In addition, some embodiment can not postpone described rotation after equipment is tilted above threshold quantity.In addition, different embodiment can allow to trigger rotary manipulation with different modes, such as by switch switch on the mobile device, by send voice commands, according to the selection of being undertaken by menu, or the like.
Quadravalence section 3425 is illustrated in the UI 1105 after the beginning rotary manipulation.Some embodiment flash demo rotation viewing areas are to provide the feedback about rotary manipulation to the user.The example of a kind of animation that Figure 34 graphic extension is such.Specifically, Figure 34 begins rotation together in its quadravalence section 3425 performance viewing areas 1180 and 1155. Viewing area 1180 and 1155 centers on axle 3465 (that is z the axles) rotation through the center of UI 1105. Viewing area 1180 and 1155 is rotated identical quantity, but along with the opposite direction rotation of the rotation of equipment 3400 inclination of equipment 3400 (for example, by).In this example, because equipment 3400 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), so rotary manipulation can make viewing area 1180 and 1155 half-twist in the counterclockwise direction.When viewing area 1180 and 1155 rotations, viewing area 1180 and 1155 dwindles pari passu, to be fit to UI 1105, makes viewing area 1180 and 1155 still can appear at fully on the UI 1105.Some embodiment can provide the message (for example, by showing words " Rotating (rotation) ") of the state of indicating equipment 3400.
Five-stage 3430 is illustrated in viewing area 1180 and 1155 and is rotated counterclockwise UI 1105 after 90 ° from vertical view to transverse views.In this stage, viewing area 1180 and 1155 has been rotated, but also is not deployed into the whole width of UI 1105.The ending that arrow 3475 is pointed out at five-stage, viewing area 1180 and 1155 will begin side direction and launch, to be fit to the whole width of UI 1105.Different embodiment can not comprise this stage, because can side by side carry out described expansion with the rotation in the quadravalence section 3425.
The 6th stage 3435 was illustrated in viewing area 1180 and 1155 and has been unfolded, thereby occupied the whole demonstration UI 1105 afterwards of UI 1105.As mentioned above, other embodiment can differently realize this rotation.Concerning some embodiment, only the screen of equipment rotation is surpassed threshold quantity and just can trigger described rotary manipulation, and the orientation of tube apparatus 3400 not.
In addition, other embodiment can be provided for indicating the different animations of rotary manipulation.The rotary manipulation of carrying out in Figure 34 relates to viewing area 1180 and the 1155 center rotations around UI 1105.On the other hand, can make the viewing area center on they separately the central shaft of viewing area rotate separately.A kind of such method has been shown among Figure 35.Figure 35 represents that the PIP of flash demo UI 1105 shows the alternative approach of the rotation of 1180 viewing area 1170 and 1160.Among Figure 35 the PIP of graphic extension show 1180 with Figure 11 in the PIP of graphic extension show that 1180 is identical.
Figure 35 is according to six different operational phases 3410,3415,3420,3525,3530 and 3535, and graphic extension PIP shows 1180 rotation.The operation of the preceding three phases of UI 1105 is with identical as the operation of the preceding three phases of explanation among the UI 1105 in Figure 34.In the phase III of Figure 34 and 35, equipment 3500 all becomes horizontal holding from perpendicular holding, and the rotation of UI 1105 does not also begin.
The alternative approach of the described rotation of quadravalence section 3525 graphic extension flash demos.In the quadravalence section, rotary manipulation begins.Specifically, quadravalence section 3525 shows the beginning of the rotation of viewing areas 1170 and 1160. Viewing area 1170 and 1160 rotates around axle 3567 and 3565 (that is the z axles) at the center of passing each viewing area respectively.Viewing area 1170 and 1160 is rotated identical amount, but along with the opposite direction rotation of the rotation of equipment 3500 inclination of equipment 3500 (for example, by).With top in the quadravalence section 3425 of Figure 34 graphic extension similar because equipment 3500 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), so rotary manipulation can make viewing area 1170 and 1160 half-twist in the counterclockwise direction.When viewing area 1170 and 1160 whens rotation, viewing area 1170 and 1160 scaled to be fit to UI 1105 makes viewing area 1170 and 1160 still can appear at fully on the UI 1105.
Five-stage 3530 is illustrated in viewing area 1170 and 1160 and all is rotated counterclockwise UI 1105 after 90 ° from vertical view to transverse views.In this stage, viewing area 1170 and 1160 has been rotated, but also is not deployed into the whole width of UI 1105.In addition, viewing area 1160 also is not moved to its final position.Insert viewing area 1160 PIP show in 1180 the final position by as show that at PIP the position in 1180 determines (for example, inserting the lower left corner of viewing area 1160 in PIP demonstration 1180) in the insertion viewing area 1160 as shown in the phase I 3410.In this stage, insert viewing area 1160 still in the upper left corner of UI 1105.
The ending that arrow 3580 is pointed out at five-stage 3530, viewing area 1170 and 1160 will begin side direction and launch, and be fit to up to main viewing area 1170 till the whole width of UI 1105 of the horizontal equipment of holding.In addition, arrow 3575 points out to insert the lower left corner that viewing area 1160 will slide into PIP demonstration 1180.
Different embodiment can differently realize this point.In certain embodiments, inserting moving of viewing area 1160 can take place simultaneously with the expansion of main viewing area 1170, perhaps can occur in sequence.In addition, some embodiment can or launch the back convergent-divergent before launch main viewing area 1170, in the expansion and insert viewing area 1160, show 1180 to produce new PIP.In this example, in 1160 and 1170 whens rotation in the viewing area, viewing area 1155 disappears.But, in certain embodiments, viewing area 1155 can during rotation be retained on the UI 1105, and rotates together with viewing area 1160 and 1170.
The 6th stage 3535 was illustrated in inserts viewing area 1160 and arrives its repositions, and viewing area 1160 and 1170 appropriately launched, with the UI 1105 after the whole width that is fit to UI 1105.In this example, insert the lower left corner that viewing area 1160 is positioned at PIP demonstration 1180 now, overlap on the main viewing area 1170.PIP demonstration 1180 has now to have with the PIP of phase I 3410 and shows 1180 identical demonstrations arrangements.The viewing area 1155 indication rotary manipulations that occur under PIP demonstration 1180 in the 6th stage are finished.As mentioned above, only the screen of equipment rotation is surpassed threshold quantity and just can trigger described rotary manipulation, and the orientation of tube apparatus 3500 not.
In the example with reference to Figure 34 and 35 explanations, the orientation of viewing area 1170 also changes (that is, from vertically becoming laterally) in the above.That is, after the rotation viewing area 1170, show 1180 by horizontal development PIP in the phase III 3420, make it be fit to whole UI 1105, the orientation of viewing area 1170 is from vertically becoming laterally.In certain embodiments, when equipment 3500 was rotated, the video of being taken by remote equipment rotated, but showed that the orientation of the viewing area of the video of being taken by remote equipment remains unchanged.Illustrate such example among Figure 36.Figure 36 is similar to Figure 35, and except the video that is presented in the viewing area 1170 rotates, but viewing area 1170 keeps showing along the longitudinal direction.
Also illustrate wherein the example that viewing area 1155 remains on the rotary manipulation of same position (rather than rotate like that as shown in Figure 35 and horizontal development, show 1180 to fill up PIP) among Figure 36.In addition, Figure 36 comprises and top layout in the identical viewing area 1155 of the layout of viewing area 1155 illustrated in fig. 12.As shown in the figure, in stage 3640,3645,3650,3655,3685 and 3690, when equipment 3500 rotations, viewing area 1155 is retained in same position.
Some embodiment provide a kind of rotary manipulation, the directed change (rather than keeping identical orientation as shown in Figure 35 like that) that wherein shows the viewing area of the video of being taken by local device is rotated the operation orientation of local device afterwards to be reflected in to local device.Figure 36 is with reference to six different stages 3640,3645,3650,3655,3685 and 3690, the example of this rotary manipulation of graphic extension UI1105.In Figure 36, the phase I 3640 shows along the longitudinal direction inserts viewing area 1160, inserts the video that viewing area 1160 shows by the camera of equipment 3500.Second is identical with second and the phase III 3415 and 3420 of Figure 35 with 3650 with the phase III 3645, because their indication equipments 3500 are in the inclination in each stage of rotary manipulation.At this moment, the camera of equipment 3500 is along the horizontal direction photographic images.In order to point out this transformation, some embodiment provide as the 4th and five-stage 3655 and 3685 as shown in animation, and other embodiment does not provide any animation.
In quadravalence section 3655, be presented at the image that inserts in the viewing area 1160 and be rotated, itself be not rotated but insert viewing area 1160 because second and phase III 3445 and 3650 in the inclination of equipment 3500 rotate to horizontal direction inserting viewing area 1160.In five-stage 3685, the image rotating that inserts in the viewing area 1160 is inserted viewing area 1160 by horizontal development to fill, 1160 beginnings of insertion viewing area are moved towards the territory, lateral areas, lower-left of PIP demonstration 1180, an insertion viewing area 1160 is placed and insert viewing area 1160 at the identical relative position of the PIP of phase I 3640 demonstration.
In certain embodiments, show that the orientation of the viewing area of the video of being taken by remote equipment also changes, remote equipment is rotated the operation orientation of remote equipment afterwards to be reflected in.Four different phases of the UI 1105 of Figure 37 graphic extension equipment 3500, wherein (1) shows the change in orientation of the viewing area (viewing area 1160 in this example) of the video of being taken by local device, to be reflected in local device is rotated the operation orientation of local device afterwards, (2) demonstration is rotated the operation afterwards orientation of local device to be reflected in to remote equipment by the change in orientation of the viewing area (viewing area 1170 in this example) of the video of remote equipment shooting.
In the phase I 3705, UI 1105 is identical with UI 1105 among Figure 36.Specifically, phase I 3705 shows the viewing area 1160 and 1170 of portrait orientations, because equipment 3500 shows by portrait orientation, and remote equipment is the (not shown) of portrait orientation.From the phase I 3705 to second stage 3710, by equipment 3500 from stand up position to the lateral attitude half-twist, local device is rotated operation.Second stage 3710 is presented at the rotary manipulation UI 1105 afterwards of the equipment of finishing 3500.In second stage, the video that is presented in viewing area 1170 and 1160 has rotated to stand up position.But, have only the viewing area 1160 of local capture video to rotate to transversal orientation, because only local device (that is, equipment 3500) has been carried out rotary manipulation from portrait orientation.Viewing area 1170 keeps portrait orientation.
From second stage 3710 to the phase III 3715,, remote equipment is rotated operation by remote equipment is rotated to the lateral attitude (not shown) from stand up position.Phase III 3715 is presented at the UI 1105 after the rotary manipulation of finishing remote equipment.In the phase III, the viewing area 1170 that is presented at video in the viewing area 1170 and long-range capture video rotates to transversal orientation from portrait orientation, because only remote equipment has been carried out rotary manipulation.Thereby the phase III of UI 1105 all shows the viewing area 1170 and 1160 of local and remote capture video by transversal orientation.
From the phase III 3715 to quadravalence section 3720, by equipment 3500 from the lateral attitude to the stand up position half-twist, local device is rotated operation.Quadravalence section 3720 is presented at finishes this rotary manipulation UI 1105 afterwards.In quadravalence section 3720, the video that is presented in viewing area 1160 and 1170 has rotated to stand up position.But, have only the viewing area 1160 of local capture video to rotate to portrait orientation, because only local device (that is, equipment 3500) has been carried out rotary manipulation from transversal orientation.Viewing area 1170 keeps transversal orientation.
From quadravalence section 3720 to the phase I 3705, by remote equipment from the lateral attitude to stand up position half-twist (not shown), remote equipment is rotated operation.In this case, the phase I 3705 is presented at and finishes this rotary manipulation viewing area 1170 afterwards.So the UI 1105 in this stage shows the viewing area 1160 and 1170 of portrait orientation.Although Figure 37 illustrates a series of different rotary operation, but other embodiment can carry out any many rotary manipulations according to any many different sequences.
The rotary manipulation that Figure 34,35,36 and 37 explanations are carried out local and remote device during video conference.When local mobile device being rotated operation, some embodiment are notified to remote equipment to rotary manipulation, so that allow remote equipment can carry out any modification (for example rotation shows the viewing area of the video of local device) to the video of local device.Similarly, when remote equipment being rotated operation, remote equipment is given local device this operational notification, makes local device carry out any modification to the video of remote equipment.Some embodiment are provided at during the video conference, transmit the control communication port of the notice of rotary manipulation between local device and remote equipment.
Although Figure 34,35,36 and 37 graphic extensions can realize the different modes of the animation that rotates,, those of ordinary skill can show the animation of rotation with different modes but will appreciate that other embodiment.In addition, the animation of rotary manipulation can cause the variation of the image processing operations of local mobile device, such as making video conference manager 1604 be presented at image in the viewing area with synthetic again viewing area of different angles and proportional zoom in UI 1105.
3. window size adjustment
The size of the insertion viewing area that the PIP that some embodiment allow the user of mobile device to be adjusted to present during the video conference shows.Different embodiment provides convergent-divergent to insert the different technologies of viewing area.Figure 38 graphic extension convergent-divergent inserts a kind of method of viewing area.In the method, the user by selecting of mobile device is inserted a corner of viewing area, enlarges subsequently or dwindles and insert the viewing area, adjusts the size of inserting the viewing area.
In Figure 38, the UI 3800 of mobile device 3825 with the long-distance user's of another mobile device video conference during, present PIP and show 3865.PIP shows that 3865 comprise that two videos show: background master viewing area 3830 and prospect are inserted viewing area 3835.Background master viewing area 3830 occupies most PIP and shows 3865, and that prospect is inserted viewing area 3835 is less and overlap on the background master viewing area 3830.In this example, background master viewing area 3830 presents the personage's who holds guitar video, and this personage is assumed that the personage by positive its video of camera of remote equipment, perhaps by the personage of the back side its video of camera of remote equipment.Prospect is inserted the video that viewing area 3835 presents the personage who wears a hat, and in this example, this personage is assumed that the personage by positive its video of camera of local device, perhaps by the personage of the back side its video of camera of local device.Show below 3865 it is to comprise that () viewing area 1155 for example, button 3860, optional UI project 3860 allows these projects of user by selecting to finish video conference for the optional UI project 3860 that is marked as " End Conference (closing session) " at PIP.
PIP shows that 3865 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace having and be used for from the showing than overall background of the video of remote equipment, bigger background shows it can is video from local device, and less prospect inserts to show it can is video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear among the UI3800 two viewing area (for example, left and right sides display window, perhaps display window up and down) side by side, perhaps in the viewing area that two diagonal angles are arranged.In certain embodiments, the mode of PIP demonstration or acquiescence display mode can be stipulated by the user.In other embodiments, PIP shows that also can comprise the prospect that a bigger background shows and two less inserts demonstration.
Figure 38 is according to four operational phases of UI 3800, graphic extension zoom operations.In the phase I 3805, prospect is inserted viewing area 3835 significantly less than background master viewing area 3830.In addition in this example, prospect is inserted the lower right corner that viewing area 3835 is positioned at PIP demonstration 3865.In other example, prospect is inserted viewing area 3835 can have different sizes, perhaps is arranged in PIP and shows 3865 zones of different.
In second stage 3810, initiate zoom operations.In this example, want a corner (for example) of the insertion viewing area 3835 of convergent-divergent, initiate this operation by finger 3840 is placed on the upper left corner of inserting viewing area 3835 by selecting the user.Second stage 3810 usefulness of UI 3800 are inserted the thick frame 3845 of viewing area 3835 and are pointed out this selection.In this stage, the user can enlarge or dwindle insertion viewing area 3835 (for example, by showing at PIP on 3865 away from inserting viewing area 3835 or dragging its finger 3840 towards inserting viewing area 3835).
Phase III 3815 is illustrated in shown in arrow 3850, by moving its finger 3840 (in this example promptly away from inserting viewing area 3835, by move his finger towards the diagonal angle, the upper left corner of UI 3800), the user begins to enlarge insertion viewing area 3835 UI 3800 afterwards.Shown in arrow 3855, moving at height and Width of finger 3840 enlarged insertion viewing area 3835 pari passu in addition.In other example, the user can utilize identical technology (that is, by dragging finger towards inserting viewing area 3835), dwindles and inserts viewing area 3835.
Quadravalence section 3820 is presented at and finishes the convergent-divergent UI 3800 afterwards that inserts viewing area 3835.In this example, reached desirable size in case insert viewing area 3835, the user is just by stopping to drag of its finger 3840 so, and shows that from PIP 3865 remove his finger, finishes the convergent-divergent of insertion viewing area 3835.As the result of this process, the insertion viewing area 3835 behind the convergent-divergent is greater than its initial size in the phase I 3805.Insertion viewing area zoom operations has been finished in the elimination indication of thick frame 3845 now.
Some embodiment provide and allow the user during video conference, show that at PIP convergent-divergent inserts other technology of viewing area 3835 in 3865.A kind of other such technology of Figure 39 graphic extension.An a kind of edge (that is, at one of side that inserts viewing area 3835) by select inserting viewing area 3835 of Figure 39 graphic extension enlarges subsequently or dwindles and insert viewing area 3835, comes convergent-divergent to insert the technology of viewing area 3835.
Figure 39 is according to four operational phases of the UI 3800 of Figure 38, this zoom operations of graphic extension.The phase I 3805 of Figure 39 is identical with the phase I 3805 among Figure 38.Specifically, in this stage, the PIP that UI 3800 graphic extensions of equipment 3925 have the less prospect insertion viewing area 3835 in the bigger background master viewing area 3830 and the lower right corner that is positioned at PIP demonstration 3865 shows 3865.Convergent-divergent inserts two kinds of different technologies of viewing area 3835 among the identical UI 3800 although Figure 38 and 39 is illustrated in, and will not provide this two kinds of technology but those of ordinary skill will appreciate that some embodiment in identical UI.
The beginning of second stage 3910 graphic extension zoom operations.In this example, this user of user by selecting wants a side (for example, finger 3840 being placed on the top margin or side that inserts viewing area 3835) of the insertion viewing area 3835 of convergent-divergent, initiates zoom operations.In this example, the user is placed on his finger 3840 on the top margin that inserts viewing area 3835, so that realize described selection.Second stage 3910 usefulness are inserted this selection of thick frame 3845 indications of viewing area 3835.
Phase III 3915 is illustrated in shown in arrow 3950, and the user inserts viewing area 3835 (that is, vertically moving towards the top of PIP demonstration 3865) by its finger 3840 is moved apart, and begins to enlarge insertion viewing area 3835 UI 3800 afterwards.In addition, shown in arrow 3955, moving at height and Width of finger 3840 all enlarged insertion viewing area 3835 in proportion.In other example, the user dwindles viewing area 3835 by utilizing identical technology (that is, by dragging finger 3840 towards inserting viewing area 3835).
Quadravalence section 3920 is presented at finishes the convergent-divergent UI3800 afterwards that inserts viewing area 3835.In this example, in case insert after viewing area 3835 reaches desirable size, the user is just by stopping to drag of its finger 3840 so, and the display screen of slave unit removes his finger 3840, finishes the convergent-divergent of insertion viewing area 3835.As the result of this process, the insertion viewing area 3835 behind the convergent-divergent is greater than its initial size in the phase I 3805.Insertion viewing area zoom operations has been finished in the elimination indication of thick frame 3845 now.
Response drag operation, some embodiment are adjusted the size of inserting viewing area 3835, as shown in Figure 38 and 39 pari passu at height and Width.Other embodiment can allow the user to adjust height and/or the width that inserts viewing area 3835, and does not influence another attribute.The example of a kind of such convergent-divergent process of Figure 40 graphic extension.
Specifically, Figure 40 graphic extension mobile device UI 3800 similar UI 3,800 4025 and Figure 38, except when when select inserting the level in the lump at edge of viewing area 3835 or this edge of vertical moving, the UI 3800 of Figure 40 allows to insert outside viewing area 3835 along continuous straight runs and/or the vertical direction expansion.PIP among Figure 40 graphic extension UI 3800 and Figure 38 shows that 3865 similar PIP show 3865, except present insertion viewing area 3835 the upper right corner of PIP demonstration 3865.PIP shows that 3865 comprise that two videos show: background master viewing area 3830 and prospect are inserted viewing area 3835.In this example, background master viewing area 3830 presents by the front camera of remote equipment or the video of back side camera.Prospect is inserted viewing area 3835 and is presented the video of being taken by the front camera or the background camera of local device.
Be similar to Figure 38, Figure 40 is according to four operational phases of UI 3800, graphic extension zoom operations.Phase I 4005 is similar with the phase I 3805 of Figure 38, except present insertion viewing area 3835 the upper right corner.Other three phases 4010,4015 and 4020 is similar to three phases 3910,3915 and 3920, except the selection on the base of inserting viewing area 3835 with move and only cause inserting viewing area 3835 and vertically enlarges, and do not influence outside the width of insertion viewing area 3835.
Figure 38,39 and 40 provides by corner of select inserting viewing area 3835 or side, allows user's convergent-divergent PIP to show the example embodiment of 3865 insertion viewing area 3835.Some embodiment provide convergent-divergent to insert other technology of window 3835.For example, some embodiment of Figure 41 graphic extension allow by selecting to insert the inside of viewing area 3835, and convergent-divergent inserts viewing area 3835.In this method, the user is placed on the screen by pointing 4155 and 4156 to two, and drags described two fingers away from each other or closely, adjusts the size of inserting viewing area 3835.
In Figure 41, the UI 3800 of mobile device 4140 with the long-distance user's of another mobile device video conference during, provide PIP to show 3865.In order to simplify the explanation of UI 3800, the PIP with Figure 38 among Figure 41 graphic extension UI 3800 shows that 3865 similar PIP show 3865.
Figure 41 is according to seven operational phases of UI 3800, this zoom operations of graphic extension.The expansion of viewing areas 3835 is inserted in preceding four- stage 3805,4110,4115 and 4120 expressions, and then three phases represents to insert dwindling of viewing area 3835.Phase I 3805 among Figure 41 is identical with the phase I 3805 among Figure 38.Specifically, in this stage, UI 3800 graphic extensions have bigger background master viewing area 3830 and less prospect and insert the PIP of viewing area 3835 and show 3865.In this example, background master viewing area 3830 presents by the front camera of remote equipment or the video of back side camera.Prospect is inserted viewing area 3835 and is presented by the front camera of local device or the video of back side camera.
Second stage 4110 is illustrated in initiates zoom operations UI 3800 afterwards.In this example, this user of user by selecting wants the insertion viewing area 3835 (for example, inserting in the viewing area 3835 by two fingers 4155 and 4156 are placed on) of convergent-divergent, initiates zoom operations.Second stage 4110 usefulness of UI 3800 are inserted the thick frame 4190 of viewing area 3835 and are pointed out this selection.
Phase III 4115 is illustrated in shown in arrow 4160, the user is by remotely moving its finger 4155 and 4156 (promptly mutually, upper left corner moveable finger 4155 towards PIP demonstration 3865, lower right corner moveable finger 4156 towards PIP demonstration 3865), begin to enlarge insertion viewing area 3835 UI 3800 afterwards.Shown in arrow 4165, moving at height and Width of finger 4155 and 4156 enlarged insertion viewing area 3835 pari passu.
Quadravalence section 4120 is presented at finishes the convergent-divergent UI3800 afterwards that inserts viewing area 3835.In this example, the user is by stopping its finger 4155 and 4156 drags, and the display screen of slave unit removes its finger 4155 and 4156, finishes the convergent-divergent that inserts viewing area 3835.As the result of this process, the insertion viewing area 3835 behind the convergent-divergent is greater than its initial size in the phase I 3805.The elimination indication of thick frame 4190 has been finished now and has been inserted the viewing area zoom operations.
At five-stage 4125, by being placed on, insert on the viewing area 3835 on two fingers 4155 and 4156, and the user reselects and inserts viewing area 3835.The 6th stage 4130 was illustrated in shown in arrow 4170, and the user begins to dwindle insertion viewing area 3835 UI 3800 afterwards by closely moving its finger 4155 and 4156 mutually.Shown in arrow 4175, insertion viewing area 3835 has been dwindled in moving at height and Width of finger 4155 and 4156 pari passu.
The 7th stage 4135 was similar to the quadravalence section 4120 of Figure 41, except by described operation, outside the size of inserting viewing area 3835 has been dwindled.The elimination indication of thick frame 4190 has been finished now and has been inserted the viewing area zoom operations.
Several example user interface of the insertion viewing area that allows user's convergent-divergent PIP demonstration are for example understood in the above-mentioned explanation of Figure 38-41.In certain embodiments, the convergent-divergent that inserts the viewing area can cause the variation of the image processing operations of two camera movement equipment, such as the input that makes video conference manager 1604 response users, and the proportional zoom of the insertion viewing area during change PIP shows and synthetic.In addition, in certain embodiments, the layout of the layout of the viewing area 1155 among Figure 38-41 and the viewing area 1155 of Figure 12 described above is identical.
4. identification area-of-interest
Some embodiment allow the user during video conference, area-of-interest (ROI) in the video that identification shows, so that (for example revise image processing, image processing manager 1608 among Figure 16), coding (for example, encoder 1655 among Figure 16), the behavior during video conference of mobile device and camera thereof, perhaps their combination.Different embodiment provides different technology to come this area-of-interest in the identification video.Figure 42 graphic extension is used for the area-of-interest of identification video, so that the user interface of some embodiment of the picture quality of raising video.
In Figure 42, the UI 4200 of mobile device 4225 with the long-distance user's of another mobile device video conference during, present PIP and show 4265.PIP among Figure 42 show basically with Figure 41 in PIP show similar.Specifically, the PIP among Figure 42 shows and to comprise that two videos show: background master show 4230 and prospect insert and show 4235.In this example, background master shows that 4230 present the personage's who sets and wear a hat video, described tree and personage are assumed that tree and the personage by positive its video of camera of remote equipment, perhaps by the tree and the personage of the back side its video of camera of remote equipment.Prospect insert to show that 4235 present man's video, and in this example, described man is assumed that the man by positive its video of camera of local device, perhaps by the personage of the back side its video of camera of local device.PIP show below be to comprise that () viewing area 1155 for example, button 4260, optional UI project 4260 allows these projects of user by selecting to finish video conference for the optional UI project 4260 that is marked as " End Conference (closing session) ".
This PIP shows a kind of mode of the synthetic view that just presents the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace having and be used for from the showing than overall background of the video of remote equipment, bigger background shows it can is video from local device, and less prospect inserts to show it can is video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear among the UI two viewing area (for example, left and right sides display window, perhaps display window up and down) side by side, perhaps in the viewing area that two diagonal angles are arranged.In other embodiments, PIP shows that also can comprise the prospect that a bigger background shows and two less inserts demonstration.In certain embodiments, the mode of PIP demonstration or acquiescence display mode can be stipulated by the user.
Figure 42 is according to four operational phases of UI 4200, and graphic extension ROI identifies operation.As shown in the phase I 4205, be presented on background and show that the video in 4230 has very low quality (that is, video image is fuzzy).In this example, the user of mobile device 4225 is intended to background is shown that the zone that occurs personage's face 4270 in 4230 is identified as area-of-interest.
In second stage 4210, initiate the operation of identification area-of-interest.In this example, by (for example selecting to be presented on zone that background shows that user among the video in 4230 wants to be identified as area-of-interest, by on the screen of equipment, near the position the character facial 4270 of the demonstration in background demonstration 4230 dubs finger 4250), initiate this operation.
As shown in the phase III 4215, the zone that user's zone is selected to make UI 4200 to draw to select around the user around frame 4275 (for example, dashed rectangle 4275).Quadravalence section 4220 is presented at the UI 4200 after the sign that finishes area-of-interest.As the result of this process, and to compare in the phase I 4205, the quality of the video in the area-of-interest significantly improves.The ROI selection operation has been finished in elimination indication around frame 4275 now.In certain embodiments, the ROI identification process also causes the identical variation of variation that local device 4225 is caused with it to the same video that is presented on the remote equipment.For example, in this example, the interior image quality of area-of-interest that is presented at the same video on the remote equipment also significantly improves.
In certain embodiments, the user can or around frame 4275 (for example dwindle in phase IIIs 4215 amplification, by finger 4250 is placed in the display frame, and towards the upper right corner moveable finger 4250 of screen, to amplify around frame 4275, perhaps towards the lower left corner moveable finger 4250 of screen, to dwindle) around frame 4275.Some embodiment also allow the user phase IIIs 4215 rotating ring around frame 4275 (for example, by finger 4250 is placed in the display frame, and in display frame level or vertical moving finger 4250).In some other embodiment, the selection in described zone can not make UI 4200 draw around frame 4275 in the phase III 4215.
Other embodiment provides the different technologies that allows the area-of-interest in the User Recognition video.A kind of other such technology of Figure 43 graphic extension.In Figure 43, the shape of user by drawing the Sensurround region-of-interest, identification area-of-interest.In this example, described shape is a rectangle, but also can be other shape (for example, any other polygon, circle, ellipse etc.).Some embodiment provide the alternative technique of Figure 43 in the equipment UI of the sort of technology that also is provided at graphic extension among Figure 42.But, other embodiment does not provide these two kinds of technology in identical UI.
Figure 43 is according to five operational phases of UI 4200, and this ROI sign of graphic extension is operated.Phase I 4205 among Figure 43 is identical with the phase I 4205 among Figure 42.Specifically, in the phase I 4205, UI 4200 graphic extensions have bigger background master and show 4230 and be positioned at less prospect that PIP shows 4265 the lower left corner and insert and show that 4235 PIP shows 4265.
In second stage 4310, initiate the operation of identification area-of-interest.In this example, select definition (for example to be presented on primary importance that the background master shows the area-of-interest in the video that presents in 4230 by continuing for some time, by on the screen of equipment, continue for some time and finger 4350 is placed on background shows near the shown character facial 4270 in 4230 the position), initiate described operation.In the phase III 4315, UI 4200 utilizes and closes on the round dot 4355 of the selected primary importance on the background viewing area 4230, and described primary importance 4370 has been selected in indication.
Quadravalence section 4320 is illustrated in the second place 4375 UI 4200 afterwards that the user has selected the definition area-of-interest.In this example, by shown in arrow 4360, after round dot 4355 occurs, begin in the screen of equipment, to drag finger 4350 from primary importance, and stopping at position between the tree of the cap of the demonstration in background viewing area 430 and demonstration, the user selects the second place 4375.As shown in the quadravalence section, this rectangular shaped rim 4365 that causes UI4200 to draw area-of-interest that drags, described rectangular shaped rim 4365 has first and second positions 4370 and 4375 on its relative summit.
Five-stage 4325 is illustrated in the sign of area-of-interest and has finished UI4200 afterwards.In this example, the user has been by in case identified the area-of-interest of expectation, just stop to point 4350 drag and the display screen of slave unit is removed finger 4350, finish the sign of area-of-interest.Five-stage 4325 graphic extensions are by should dilatory (drawing) process, and compare in the phase I 4205, and the quality of the video in the area-of-interest significantly improves.In certain embodiments, the same to the variation that local device 4225 is caused with it, this dilatory process also causes identical variation to the display frame on the remote equipment.For example, in this example, the interior picture quality of area-of-interest that is presented at the same video on the remote equipment can significantly improve.
The area-of-interest in the identification video is for example understood in top Figure 42 and 43 description, so that improve the different modes of the picture quality of institute's identified region.In certain embodiments, the image quality of improving the area-of-interest of identification can cause the variation of the encoding operation of two camera movement equipment, such as when to video coding, to the region allocation more bits of being discerned.
Some embodiment allow the area-of-interest in the User Recognition video, so that mobile device or its camera are made different changes.For example, the area-of-interest in Figure 44 graphic extension identification video is to enlarge or to dwindle the example of area-of-interest in display frame.In this method, the user is by selecting a certain zone in the display frame as the center of area-of-interest as, enlarges subsequently or dwindles described area-of-interest, the area-of-interest in the identification video.
In Figure 44, the UI 4400 of mobile device 4425 with the long-distance user's of another mobile device video conference during, present PIP and show 4265.PIP among Figure 44 shows that 4265 show that with the PIP of Figure 42 4265 is similar basically, but the prospect of Figure 44 is inserted and shown that 4235 are positioned at PIP and show 4265 the lower left corner.
Figure 44 is according to four operational phases of UI 4400, graphic extension ROI selection operation.As shown in the phase I 4405, background shows that 4430 present to have in background and show the man in 4430 left side and show the video of the tree 4440 on 4430 right side in background.In addition, tree 4440 is less relatively, only occupies the right side of background viewing area 4430.In this example, the user of mobile device 4425 is intended to the zone that occurs tree 4440 on the background viewing area 4430 is identified as area-of-interest.
In second stage 4410, initiate the operation of identification area-of-interest.In this example, (for example show zone 4440 among the video in 4430, that the user wishes to be identified as area-of-interest by selecting to be presented on background, by two fingers 4445 and 4446 being placed on the background viewing area 4430, showing the place of tree 4440), initiate described operation.In second stage 4410, by dragging its finger 4445 and 4446 away from each other, the user can make area-of-interest 4440 enlarge, and occupies more most background viewing area 4430.By closely dragging its finger 4445 and 4446 mutually, the user can also make area-of-interest 4440 dwindle, thereby occupies the background viewing area 4430 of smaller portions.
Phase III 4415 is illustrated in shown in arrow 4450, the user is by moving its finger 4445 and 4446 away from each other (promptly, finger 4445 moves towards the upper left corner of background viewing area 4430, finger 4446 moves towards the lower right corner of background viewing area 4430), begin to make area-of-interest 4440 to enlarge to occupy the UI4400 after the more most background viewing area 4430.In certain embodiments, the finger demonstration of moving also remote equipment causes the same variation of variation that local device is caused with it.For example, in this example, the area-of-interest of same video will enlarge, thereby occupy the more most background viewing area 4430 of remote equipment.In certain embodiments, the expansion of the area-of-interest in local demonstration and/or the long-range demonstration causes one or two mobile device or its camera to revise their one or more other operations, and is as further described below.
Quadravalence section 4420 is presented at the sign of area-of-interest and has finished UI 4400 afterwards.In this example, in case area-of-interest reaches the ratio of expectation in background viewing area 4430 after, finger 4445 and 4446 drag of user by stopping the user so, and remove finger 4445 and 4446 on the display screen of slave unit, finish the sign of area-of-interest.As the result of this process, area-of-interest has occupied most background viewing area 4430.Finish the sign operation of area-of-interest now.
Above some examples illustrate the how area-of-interest in the identification video of user, to improve the picture quality (for example, by increasing bit rate) in the selected area-of-interest in the video to the encoding region of interest of video.In certain embodiments, the area-of-interest in the identification video causes the image processing operations of mobile device, such as the variation of exposure, proportional zoom, focusing etc.For example, the area-of-interest in the identification video can cause the differently image of proportional zoom and synthetic video (for example, the area-of-interest of zoom is wanted in identification) of video conference manager 1604.
In other embodiments, the area-of-interest in the identification video causes the variation (for example, frame rate, zoom, exposure, proportional zoom, focusing etc.) of operation of the camera of mobile device.In other other embodiment, the area-of-interest in the identification video causes the variation of the encoding operation of mobile device, such as to region allocation more bits, proportional zoom of identification or the like.In addition, although the operation of example ROI described above sign only can cause a kind of modification in the above-mentioned modification to mobile device or its camera, but in some other embodiment, ROI sign operation meeting causes more than a kind of modification the operation of mobile device or its camera.In addition, in certain embodiments, the layout of the layout of the viewing area 1155 of Figure 42-44 and the viewing area 1155 of Figure 12 described above is identical.
B. switch camera
Some embodiment are provided at during the video conference, switch the method for camera (that is, changing the camera of photographic images).Different embodiment provides the distinct methods of realizing switching camera operation.The camera that some embodiment provide this equipment of switching of being carried out by two camera movement equipment (promptly, the local switching) method, and other embodiment provides the method for the camera (that is long-range switching) of described another equipment of another pair camera movement equipment switching in the instruction video conference for two camera movement equipment.In addition other embodiment provides the method for carrying out local switching and long-range switching simultaneously.The IV.B.1 joint will illustrate and carry out local processing of switching camera operation on two camera movement equipment.The IV.B.2 joint will illustrate the processing of carrying out long-range switching camera operation on two camera movement equipment.
1. the local camera that switches
Some embodiment of Figure 45 graphic extension carry out at local two camera movement equipment, with the video conference of the long-range mobile device that comprises at least one camera during, between two cameras of local device, carry out switching processing 4500.In certain embodiments, handle 4500 by 1604 execution of the video conference manager shown in Figure 16.For the purpose of illustrating, following discussion will call camera 1 to a camera of local two camera movement equipment, and another camera of local two camera movement equipment is called camera 2.
By beginning (4505) video conference between local two camera movement equipment and long-range mobile device, begin to handle 4500.Subsequently, (for example, camera 1) video image sends to (4510) long-range mobile device, so that show on long-range mobile device the current selected camera from local two camera movement equipment in processing 4500.4510, handle 4500 video images that also receive from long-range mobile device according to described video image and it, produce and the demonstration composograph.
Handle 4500 and determine subsequently whether (4515) receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer local two camera movement equipment the user request (for example, user interface by local two camera movement equipment), user's that perhaps should long-range mobile device request (for example, user interface by long-range mobile device), finish video conference.When processing 4500 receives the request that finishes video conference, handle 4500 and finish.
When processing 4500 does not receive the request that finishes video conference, handle 4500 and determine subsequently whether the user of (4520) local pair camera movement equipment has instructed this local device to switch the camera that is used for video conference.When handling 4500 definite (4520) described local devices, handle 4500 returns 4510 by instruction switching camera.But, when handling 4500 definite (4520) described local devices, handle 4500 and enter operation 4525 by instruction switching camera.
4525, processing 4500 sends to long-range mobile device and points out that local two camera movement equipment will switch the notice of camera.In certain embodiments, handle 4500, send described notice by VTP manager 3125 multiplexed video conference control channels with voice-grade channel and video channel by as mentioned above.
After sending its notice, handle 4500 and carry out (4530) switching camera operation.In certain embodiments, carry out (4530) switching camera operation and comprise that instruction CIPU stops to begin with camera 2 capture video images with camera 1 capture video image.These instructions can only instruct CIPU to switch to catch from the image of the pel array related with camera 2, and begin to handle these images.On the other hand, in certain embodiments, can follow one group of initiation parameter for the instruction of CIPU, described one group of initiation parameter instruction CIPU:(1) according to one group of specific setting operation camera 2, (2) catch the video that produces by camera 2 with specific frame rate, and/or (3) handle video image from camera 2 according to one group of specific setting (for example, resolution etc.).
In certain embodiments, switch camera instruction (4530) and also comprise the instruction that untapped camera is switched to aforesaid the 4th kind of operating power pattern.In this example, switching the camera instruction comprises to instruction camera 2, that switch to its 4th kind of operating power pattern.In addition, switch the camera instruction also comprise to camera 1, from its 4th kind of operating power mode switch to another kind of operating power pattern, such as first kind of operating power pattern, to save electric power, perhaps switch to the third operating power pattern, make that it can switch to the 4th kind of operating power pattern fast when being required photographic images, and the instruction of beginning photographic images.Switch the image that camera operation 4530 also relates to the synthetic image of being taken by the camera 2 of local two camera movement equipment (rather than image of being taken by camera 1) and receives from long-range mobile device, so that on two camera movement equipment, show.
After cameras are switched in 4530 instructions, handle 4500 and on local two camera movement equipment, carry out (4535) and switch the camera animation, to be presented at from the demonstration of the image of camera 1 with from the transformation between the demonstration of the image of camera 2.After the switching camera animation on local two camera movement equipment, handle 4500 circulations through operation 4510-4520, up to receive finish video conference request or new switching camera request till.
Some embodiment of Figure 46 graphic extension how to allow the UI1105 request by two camera movement equipment switch camera operation and these embodiment how flash demo switch an example of camera operation.Figure 46 is according to eight different operational phases 4610,4615,4620,4625,4630,4635,4640 and 4645 of the UI 1105 of this equipment, and camera operation is switched in graphic extension.The preceding four-stage graphic extension of UI 1105 receives the example of user's switching camera request.In some embodiments of the invention, the user of this equipment has other mechanism that produces this request.
The phase I 1130 of the UI 1105 of phase I 4610 and Figure 11 is identical, and it is illustrated in sets up video conference UI 1105 afterwards.In this stage, UI 1105 demonstrations comprise that the PIP that two videos show shows: insert demonstration from the bigger background demonstration of remote camera with from the less prospect of local camera.In this example, background master viewing area 1170 presents a Ms's video, in this example, described Ms is assumed that the Ms who is taken its video by remote equipment, and prospect is inserted the video that viewing area 1160 presents a man, in this example, described man is assumed that the man by positive its video of camera of local device.
Second stage 4615 shows subsequently by the selection of the PIP viewing area 1180 of UI 1105, begins to switch camera operation.As shown in the figure, show on 1180, realize described selection by user's finger 4670 being placed on PIP.Phase IIIs 4620 demonstration comprises and being used for during video conference, the UI 1105 of the optional UI project 4675 that request is switched between the camera of local device 4600 (for example, switching camera button 4675).The user that quadravalence section 4625 is illustrated in local device 4600 selects after (for example, by singly refer to dub) optional UI project 4675 and the UI 1105 after pointing out this selection by highlighting of optional UI project 4675.By selecting this optional UI project 4675, user instruction equipment 4600 is during video conference, and the front camera of slave unit 4600 switches to the back side camera of equipment 4600.In other example of the back side camera video of equipment 4600, the user switches to the front camera of equipment 4600 to the back side camera of selection meeting commander equipment 4600 slave units 4600 of optional UI project 4675 therein.After the quadravalence section, the video conference manager sends instruction to CIPU, and remote equipment begins to switch camera operation.
The back four-stage 4630,4635,4640 of UI 1105 and the example of the switching camera animation on the 4645 graphic extension local devices.This animation intention causes from the video of the front camera of local device and back side camera and is presented at impression on the tow sides of observing pane concomitantly, and at any time, described observation pane is merely able to allow the user see one of its tow sides.When the video conference intermediate request is switched camera, make this observation pane seemingly around the vertical axis rotation, make and show that the first forward direction user who observes pane the video of a camera that presents on the side of video of a camera is rotated away from the user, till another side of its observed pane replaced, described another side showed the video of another camera.The rotation animation of this observation pane of experiencing and phenomenon are in the viewing area that is used for a camera by (1), dwindle video image gradually from this camera, and described video image is used perspective correction operate, (2) are in described viewing area subsequently, enlarge video image gradually, and reduce to the perspective correction of described video image that operation realizes from another camera.
Therefore, five-stage 4630 graphic extensions are around the beginning of vertical axis 4682 " observing the rotation of pane ".In order to provide the rotation phenomenon of observing pane, UI 1105 has reduced the size of the video image of the front camera in the video viewing area 1160, and has used pivot operation, compares with the left side of video image to seem, the right side of video image is further from the user.
Pane half-twist is observed in the 6th stages 4635 graphic extensions, makes the user can only see the edge of pane, as is presented at shown in the fine rule 4686 of central authorities of viewing area 1160.The graphic extension of the 7th stages 4640 is observed pane and is continued rotation, makes the back side of observing pane 4688 engender now in face of the user, to show from the video of user's back side camera.Equally, in certain embodiments, this performance of rotation animation is the size of video image in video viewing area 4688 by reducing back side camera, and uses pivot operation, so that compare with the right side of video image, realize further from the user in the left side of video image.
The 8th stages 4645 graphic extension represents to switch the finishing of animation of camera operation.Specifically, this stage shows the video image by the automobile of the back side camera of equipment 4600 in viewing area 1160.
Example with reference to Figure 46 explanation calls the switching camera operation by switching the camera user interface above.Other embodiment differently calls the switching camera operation.For example, some embodiment for example on the UI 1105 of Figure 47, call the switching camera operation by making the UI that switches during the optional UI project of camera for good and all is presented at video conference.In Figure 47, switch camera button 1289 and be displayed in the viewing area 1155 together with mute button 1285 and closing session button 1287.The layout of viewing area 1155 is identical with the layout of top viewing area 1155 with reference to Figure 12 explanation.
Figure 47 is according to six stages: 4610,4790,4630,4635,4640 and 4645, and the switching camera operation of graphic extension UI 1105.The phase I 4610 of Figure 47 is similar to the phase I 4610 of Figure 46, except the layout demonstration mute button 1285 of viewing area 1155, and closing session button 1287 and switching camera button 1289, rather than outside the single closing session button.The user that second stage 4790 is illustrated in local device 4600 select (for example, by utilize finger 4670 singly refer to dub) switch the UI 1105 after the optional UI project 1289 of camera.In this example, by selecting optional UI project 1289, user instruction equipment 4600 is during video conference, and the front camera of slave unit 4600 switches to the back side camera of equipment 4600.The back four-stage of Figure 47 and the back four-stage of Figure 46 are similar, except the layout of viewing area 1155 with top in the phase I 4610 layout of explanation identical, so be not described further, to avoid bluring description of the invention with unnecessary details.
In certain embodiments, when long-range mobile device receives from the image of the different cameras of local two camera movement equipment (promptly, local two camera movement equipment switches camera) time, long-range mobile device also switches the camera animation, to be presented at from the demonstration of the image of a camera of two camera movement equipment with from the transformation between the demonstration of the image of another camera of local two camera movement equipment.Figure 48 is according to five operational phases 4810,4815,4820,4825 and 4830 of UI 4805, the example of one of this switching camera of graphic extension animation.Figure 48 represents the example switching camera animation on the long-range mobile device 4800.Each operational phase is identical with the example animation of Figure 46, and except being carries out the animation the image that is presented in the viewing area 4835, viewing area 4835 is the places that show on long-range mobile device 4800 from the image of local two camera movement equipment.Thereby, the image that is presented at the man in the viewing area 4835 is demonstrated seemingly on the vertical axis 4855 that is positioned at viewing area 4850 central authorities Rotate 180 ° by animation, with the transformation between the demonstration of the image of the demonstration of the image that is illustrated in the man in the viewing area 4835 and automobile 4870.The realization of the switching camera animation of some embodiment is identical with the realization of animation described above.
Top example illustrates the switching camera animation on the remote equipment with particular user interface layout.Other embodiment can realize this switching camera animation on the remote equipment with different user interface layout.For example, Figure 49 graphic extension has a this example of the remote equipment 4800 of different user interface layout 4805.Especially, the UI 4805 of Figure 49 has mute button 1285, the closing session button 1287 that is included in the viewing area 1155 and switches camera button 1289, and during video conference, described viewing area 1155 for good and all is presented at synthetic 4850 the side that shows.The layout of these three buttons has been described with reference to Figure 48 above.Except different user interface layout, five stages 4810,4815,4820,4825 of Figure 49 are identical with five stages 4810,4815,4820,4825 and 4830 of Figure 48 with 4830.
2. long-range switching camera
Figure 50 is illustrated in during the video conference, switches the processing 5000 of two cameras of remote double camera apparatus.Handle 5000 video conference manager execution by the equipment that comprises at least one camera.In the following discussion, the user instructs the equipment of long-range switching camera to be called as local device by it, and the equipment that switches its two cameras is called as remote equipment.In addition, in the following discussion, remote equipment is considered to switch between its front camera (camera 1 in other words) and its back side camera (camera 2 in other words).
Below with reference to Figure 51,52,53 and 54, the processing 5000 of Figure 50 is described.The UI 5105 of Figure 51 graphic extension local device 5100, during video conference, the user is switched between its two cameras by local device 5100 request remote equipments.Eight different operational phases 5110,5115,5120,5125,5130,5135,5140 and 5145 of Figure 51 graphic extension UI 5105.Figure 54 graphic extension receives the UI 5405 of the remote equipment 5400 that switches the camera request from local device 5100.Six different operational phases 5410,5415,5420,5425,5430 and 5435 of Figure 54 graphic extension UI 5405.
As shown in Figure 50, by between local device and remote equipment, starting (5005) video conference, begin to handle 5000.Handle 5000 subsequently (5010) receive a camera () image for example, from the front camera of each equipment generate the composograph of video conferences according to these images from each equipment.5010, handle 5000 and also the video image from local device is sent to remote equipment.
Subsequently, handle 5000 and determine whether (5015) receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer the user's of Local or Remote equipment request, finish video conference.When processing 5000 receives the request that finishes video conference, handle 5000 and finish.
When processing 5000 does not receive the request that finishes video conference, handle 5000 and determine that subsequently (5020) carries out the user (that is the user of local device) of the equipment of processing 5000 thereon and whether instruct this device request remote equipment to switch between it is used for the camera of video conference.Determine (5020) it handles 5000 returns 5010 when being initiated long-range switching camera by instruction when handling 5000.Determine (5020) it is handled 5000 and enter the operation 5025 that further specifies below when having been initiated long-range switching camera by instruction when handling 5000.
The example of the request of the camera of the preceding four-stage 5110,5115,5120 of the UI 5105 of Figure 51 and 5125 graphic extensions reception user's switching remote equipment.First and second stages 5100 are identical with first and second stages 4610 and 4615 of Figure 46 with 5115.Phase III 5120 is identical with the phase III 4620, except the phase III 5120 comprises that not only request local device 5100 switches the optional UI project 5175 of camera, and comprises that request remote equipment 5100 switches outside the optional UI project 5180 of camera.The user of quadravalence section 5125 graphic extension local devices 5100 selects to ask remote equipment to switch the UI project 5180 (for example, by optional UI project 5180 singly refer to dub 5170) of camera.By highlighting optional UI project 5180, point out described selection.Figure 51 represents to carry out an example of this operation, but other embodiment can differently ask remote equipment to switch the operation of camera.
Above with reference to the example of Figure 51 explanation by long-range switching camera user interface, call long-range switching camera operation.Other embodiment differently calls long-range switching camera operation.For example, some embodiment for example on the UI 5105 of Figure 52, call the switching camera operation by making the UI that switches during the optional UI project of camera for good and all is presented at video conference.In Figure 52, long-range switching camera button 5288 is displayed in the viewing area 1155 together with mute button 5282, closing session button 5284 and the local camera button 5286 of switching.
Figure 52 is according to six different stages 5110,5290,5130,5135,5140 and 5145, the long-range switching camera operation of the UI 5105 of graphic extension equipment 5100.The phase I 5110 of Figure 52 is similar to the phase I 5110 of Figure 51, except the layout demonstration mute button 5282 of viewing area 1155, the local switching camera button 5286, long-range switching camera button 5288 and the closing session button 5284.The user that second stage 5290 is illustrated in local device 5100 selects the optional UI project 5288 of (for example, by singly referring to dub 5170) long-range switching camera UI 1105 afterwards.The back four-stage of Figure 52 is similar to the back four-stage of Figure 51, except the layout of viewing area 1155 is identical with the top layout of describing in the phase I 5110, then be not described further, to avoid bluring description of the invention with unnecessary details.
Some embodiment provide with Figure 52 in the similar layout of layout of graphic extension, show in 5165 except the optional UI project of long-range switching camera is presented at PIP, rather than outside being presented in the viewing area 1155.The layout 5105 that Figure 53 graphic extension is such.Specifically, Figure 53 represents to have the PIP demonstration of the optional UI project 5180 of long-range switching camera and only has mute button 5282, local viewing area 1155 of switching camera button 5286 and closing session button 5284.
As mentioned above, when the user asks long-range switching camera, handle 5000 and enter operation 5025.In operation 5025, handle 5000 the request of switching camera sent to remote equipment.In certain embodiments, this request is by as mentioned above, is sent by the multiplexed video conference control channel of VTP manager with voice-grade channel and video channel.
After the request of switching camera is received, handles 5000 and determine whether (5030) remote equipment has responded the request of switching camera.In certain embodiments, remote equipment sends acceptance response (that is, sending affirmation) automatically by video conference control channel to local device.But, in other embodiments, the user of remote equipment must accept this request by the user interface of remote equipment.
Preceding two stages 5410 of the UI 5405 of Figure 54 and 5415 graphic extension long-distance users accept to switch the example of request of the camera of remote equipment 5400.Phase I 5410 shows that (1) is used to show the viewing area 5440 of described request being notified long-distance user's text, (2) be used to accept to switch remote equipment camera request optional UI project 5465 (for example, " permission (Allow) " button 5465), (3) be used to refuse to switch the optional UI project 5470 (for example, " refusal (Reject) " button 5470) of request of the camera of remote equipment.The user that second stage 5415 is illustrated in remote equipment subsequently (for example selects, by singly referring to dub 5480) be used to accept to switch the UI 5405 after the UI project 5465 of request of camera, by highlighting optional UI project 5465, indication user's described selection.
Determine (5030) it is handled 5000 and determine whether (5035) receive the request that finishes video conference when not receiving response from remote equipment when handling 5000.If handle 5000 so and finish.Otherwise, handle to receive (5040) image from the currently used camera of remote equipment and local device, produce the composograph of video conferences according to these images, the video image of local device is sent to remote equipment, and return 5030 subsequently.
Determine (5030) it is handled 5000 and determine the request whether (5045) remote equipments accept to switch camera when having received response from remote equipment when handling 5000.If not, handle 5000 returns 5010 so, continue to receive image from the described camera of another equipment.Otherwise, handle 5100 and receive (5050) image from another camera of remote equipment, on local device, carry out (5055) subsequently and switch the camera animation, with the transformation between the video of the remote camera of the video of the remote camera that shows previous utilization and the current utilization reception image of operation 5050 (that is).After operation 5055, handle and return operation 5010 described above.
About back four operational phases 5130,5135,5140 of 5105 graphic extensions of the UI among Figure 51 and an example of this long-range switching camera animation on the 5145 graphic extension local devices 5100.The example animated type of this example animation and graphic extension in the stage 4815,4820,4825 and 4830 of Figure 48 seemingly, except Figure 51 in viewing area 5150, the video of demonstration by the tree of the back side camera of remote equipment is outside the animation of replacement by the Ms's of the positive camera of remote equipment video.The back four-stage graphic extension of Figure 52 and Figure 53 and the identical animation of animation among Figure 51 are except Figure 52 and 53 viewing area 1155 comprise the optional UI project different with the viewing area 1155 among Figure 51.
In certain embodiments, when remote equipment switched camera, the UI of remote equipment also switched the camera animation, to show two transformations between the camera.When remote equipment 5400 switches cameras, be presented at the example of the switching camera animation on the remote equipment 5400 about back four operational phases 5420,5425,5430 of 5405 graphic extensions of the UI among Figure 54 and 5435 graphic extensions.The animated type of this animation and graphic extension in the stage 4630,4635,4640 and 4645 of Figure 46 seemingly, use video except the animation in the viewing area 5445, replace outside the video by the Ms of the positive camera of remote equipment 5400 by the tree of the back side camera of remote equipment 5400.
As mentioned above, Figure 46,47,48,49,51,52,53 and 54 is illustrated in the various examples of the switching camera animation of carrying out on the user interface.In certain embodiments, switch the variation that the camera animation causes the image processing operations of corresponding pair of camera movement equipment, such as proportional zoom, the synthetic and perspective distortion that can carry out by video conference manager 1604 and image processing manager 1608.
C. exposure is adjusted
During the video conference between two camera movement equipment and another mobile device, different embodiment provides the different technologies of adjustment by the exposure of the image of the camera of any one mobile device.Some embodiment provide the technology of adjustment by the exposure of the image of the camera of another equipment to the user of two camera movement equipment, and other embodiment provides the technology of adjustment by the exposure of the image of the camera of two camera movement equipment to the user.To describe some exemplary examples below in detail.
Figure 55 is illustrated in during the video conference, carries out the processing 5500 of long-range exposure adjustment operation on two camera movement equipment of some embodiment.In the following discussion, the user is called as local device by the equipment that it instructs remote equipment to adjust its exposure.In certain embodiments, handle 5500 video conference manager execution by local device.In addition, will handle 5500 with reference to Figure 56,57 and 58 explanations, Figure 56,57 and 58 users that illustrate local device ask remote equipment to expose and adjust the variety of way of operation.
As shown in Figure 55, by between local device and remote equipment, starting (5505) video conference, begin to handle 5500.Handle 5500 and receive (5510) video subsequently, on the display screen of local device, to show from remote equipment.Subsequently, handle 5500 and determine whether (5515) receive the request that finishes video conference.As mentioned above, some embodiment can receive the request that finishes video conference from the user of Local or Remote equipment.When processing 5500 receives the request that finishes video conference, handle 5500 and finish.
But, when processing 5500 does not receive the request that finishes video conference, handle 5500 and determine subsequently whether (5520) receive the request of the exposure of the camera of adjusting remote equipment.When handling 5500 when determining not receive the request of exposure of the camera of adjusting remote equipment, handle 5500 returns 5510, receive the other video of taking from remote equipment.Figure 56,57 provides three different examples that produce this request with 58 graphic extensions to the user.In Figure 56,57 and 58, phase I 5610,5710 and 5810 shows that all the PIP of local device 5600,5700 and 5800 shows 5625,5750 and 5835, PIP shows that 5625,5750 and 5835 show two videos: by the video of the camera of local device with by another video of the camera of remote equipment.In phase I 5610,5710 and 5810, background shows that the man in 5635,5760 and 5845 is more black dull, indicates this man not by appropriate exposure.
The video (for example, by click background show 5635) of second stage 561 graphic extensions of Figure 56 by selecting remote equipment, a kind of mode that the user of local device 5600 asks remote equipment to expose and adjust.In this manner, UI 5605 makes the user automatically, and the expectation of adjusting that exposes associates to area-of-interest to the selection of the area-of-interest that limited by square frame 5645 and user's instruction remote equipment, thereby the video conference manager contact remote equipment of instruction local device is adjusted operation to expose.The calculating that the area-of-interest of definition is used to expose and adjusts by remote equipment.
Be similar to the second stage 5615 of Figure 56, the second stage 5715 expression local users of Figure 57 are to the selection of the video of remote equipment, except the optional UI project 5770 that this selection instruction UI 5705 showed as shown in the phase III 5720.The user of quadravalence section 5725 graphic extension local devices selects optional UI project 5770, exposes as mentioned above with the instruction remote equipment and adjusts operation.
The second stage 5815 of Figure 58 is similar to the second stage of Figure 57, but replace the user that the selection instruction UI of the video of remote equipment is shown single optional UI project, user's selection instruction UI 5805 shows the menu of optional UI project 5855,5860,5865 and 580, as shown in the phase III 5820.Optional UI project comprises " automatic focus (Auto Focus) " project 5855, " automatic exposure (Auto Exposure) " project 5860, " switching camera (Switch Camera) " project 5865 and " cancellation (Cancel) " project 5870.In certain embodiments, " switching camera " optional UI project 5865 is used to ask local switching camera operation, and in other embodiments, " switching camera " optional UI project 5865 is used to ask long-range switching camera operation.Quadravalence section 5825 graphic extension users select " automatic exposure " project 5860, expose as mentioned above with the instruction remote equipment and adjust operation.
When processing 5500 determines that (5520) local users instruct local device request exposure to adjust operation, handle 5500 and send (5525) order to remote equipment, to adjust by present shooting and to the exposure of the video of the camera of local device transmission video by video conference control channel.After operation 5525, handle 5500 and return operation 5510 described above.
In certain embodiments, before remote equipment exposes the adjustment operation, require the user of remote equipment that permission is provided, and in other embodiments, after receiving request from local device, remote equipment expose and adjust operation automatically.In addition, in certain embodiments, some video conference functions are realized by video conference manager 1604.In some such embodiment, video conference manager 1604 is carried out exposure and is adjusted operation by the exposure setting that instruction CIPU 1650 adjusts the transducer of the remote equipment camera that is using.
Figure 56,57 and 58 final stage 5620,5730 and 5830 show the video of remote equipment more brightly, and this indicates this man by appropriate exposure.Although Figure 56,57 and 58 provides the example of the exposure adjustment request that receives the exposure of proofreading and correct remote equipment, but some embodiment provide the mode of the exposure of the camera of asking local device to adjust local device to the user of local device.Can produce such request similarly with the mode that the request remote equipment of graphic extension in Figure 56,57 and 58 is adjusted the exposure of its camera.
Figure 56 described above-58 expression is used to expose and adjusts several user interfaces of operation.In certain embodiments, the change that operation can cause the image processing operations of two camera movement equipment is adjusted in exposure, such as calling the exposure adjustment processing 5900 that further describes below.The change of operation of camera that operation can also cause two camera movement equipment of capture video is adjusted in exposure, such as the exposure setting of change camera.
1. exposure adjustment method
Figure 59 conceptually graphic extension by the image processing manager of some embodiment, the exposure adjustment processing of carrying out such as the image processing manager of graphic extension among Figure 16 5900.In certain embodiments, handling 5900 is parts of top exposure adjustment operation with reference to Figure 55,56,57 and 58 explanations.In some such embodiment, image processing manager 1608 is carried out and is handled 5900, and by sending the exposure setting that camera is adjusted in instruction to video conference manager 1604, described video conference manager 1604 instruction CIPU 1650 adjust camera sensor 405a or 405b, as mentioned above.
In certain embodiments, handle 5900 by 930 execution of the image processing layer shown in Fig. 9, and in other embodiments, handle 5900 by 465 execution of the statistics engine shown in Fig. 4.Some embodiment are to handling 5900 by the image of the camera of (Local or Remote) equipment in the video conference, and other embodiment carries out with the form of the part (for example, operation 2110) of the processing 2100 of graphic extension among Figure 21 and handles 5900.Some embodiment expose and adjust operation, with exposure by the camera of two camera movement equipment be not too bright, neither too dark image.In other words, carry out to handle 5900, with according to the maximized as far as possible mode photographic images of the quantity that makes details.
By receiving (5905) image, begin to handle 5900 by the camera of two camera movement equipment.In certain embodiments, when the image that receives is first two field picture by the camera of the equipment in the video conference, first two field picture was not handled for 5900 (that is, before first two field picture, not having any image of determining exposure value according to it).Handle 5900 and read the pixel value that (5910) receive the regulation zone in the image subsequently.Different embodiment differently defines described zone.Some such embodiment define difform zone, and such as square, rectangle, triangle, circle etc., and other this embodiment is at the diverse location of image, such as the center, directly over, under etc. the described zone of definition.
Subsequently, the mean value of the pixel value in the regulation zone of processing 5900 calculating (5915) images.Handle 5900 and determine whether the calculating mean value of (5920) pixel value equals specific setting.Different embodiment defines different particular values.For example, some embodiment are defined as described particular value the median pixel value of the dynamic range of image.In certain embodiments, the scope of definition numerical value, rather than single value.In such embodiments, handle 5900 and determine that the calculating mean value of (5920) pixel values is whether within the prescribed limit of numerical value.
When the calculating mean value of pixel value is not equal to specific setting, handle 5900 according to calculating mean value adjustment (5925) exposure value.When the calculating mean value of pixel value equals specific setting, handle 5900 and finish.In certain embodiments, exposure value is represented the time quantum that camera sensor is exposed.In certain embodiments, adjusted exposure value is used to expose and will be received the next frame image of the camera of image by shooting.After adjusting exposure value, handle 5900 and finish according to the mean value that calculates.
In certain embodiments, repeat and handle 5900, till the calculating mean value of pixel value equals specific setting (perhaps in the prescribed limit at numerical value).Some embodiment constantly handle 5900 during video conference, and other embodiment is during video conference, with official hour at interval (for example, 5 seconds, 10 seconds, 30 seconds etc.) handle 5900.In addition, during video conference, the processing 5900 of some embodiment was being handled before 5900, dynamically redefined specific pixel value.
Figure 60 is the example of the exposure adjustment operation of some embodiment of graphic extension conceptually.Example 6000,6010 and 6015 all is presented at the left side to the image 6020 by the camera of two camera movement equipment.Specifically, image 6020 demonstrations are back to the black dull personage of the sun.The exposure of black dull personage's indicating image be not high enough to expose personage's face or health.Each example 6000,6010 and 6015 right side are illustrated respectively in the image of taking after the image 6,020 6025,6030 and 6035.In certain embodiments, the image on image 6020 and right side is the image by the video of the camera of two camera movement equipment.In other embodiments, the image on image 620 and right side is that camera by two camera movement equipment is at the different rest images of taking constantly.
The operation that first example 6000 graphic extensions are adjusted without any exposure.Thereby it is identical with image 6020 that image 6025 seems.Owing to do not carry out any exposure adjustment, therefore and the figure kind in the image 6020 seemingly, the personage in the image 6025 is still black dull.
In second example 6010, image 6020 exposed adjust operation.In certain embodiments, exposure adjustment is operated by handling 5900 and is utilized regulation zone 6040 to carry out.Adjust operation according to exposure, the exposure of camera is adjusted, and camera utilizes adjusted exposure photographic images 6030.As shown in Figure 60, the personage in the image 6030 is not as black dull in the image 6025.But, personage's face and health are still unintelligible in the image 6030.
The 3rd the exposure adjustment operation that example 6015 expressions are carried out image 6020.Be similar to second example 6010, the exposure adjustment operation of the example 6015 of some embodiment utilizes regulation zone 6045 to carry out by handling 5900.Adjust operation according to exposure, the exposure of camera is adjusted, and camera utilizes adjusted exposure photographic images 6035.As shown in Figure 60, the personage in the image 6035 is by correct exposure, because personage's face and health are all as seen.
In certain embodiments, the selection in regulation zone can be undertaken by the user of two camera movement equipment.The feedback loop that equipment itself can be adjusted by the top exposure mentioned in CIPU 400 is adjusted it automatically and is used to expose and adjusts the regulation zone of operation.Statistics engine 465 among Fig. 4 can be collected data, with the image of determining whether exposure is suitable for taking, and adjusts camera sensor (for example, by being connected with the direct of sensor assembly 415) in view of the above.
D. focal length adjustment
Figure 61 is illustrated in during the video conference, adjusts the processing 6100 of the focal length of two camera movement equipment.In the following discussion, the user is called as local device by the equipment of the camera focal length of its instruction remote equipment adjustment remote equipment.In certain embodiments, the processing 6100 of Figure 61 is carried out by the video conference manager 1604 of local device.In addition, handle 6100 below with reference to Figure 62 and 63 explanations, Figure 62 and 63 users for local device provide the request remote equipment to carry out two kinds of by way of example of focal length adjustment operation.
As shown in Figure 61,, begin to handle 6100 by between local device and remote equipment, starting (6105) video conference.Handle 6100 and receive (6110) video subsequently, so that on the display screen of local device, show from remote equipment.Subsequently, 6115, handle 6100 and determine whether to receive the request that finishes video conference.As mentioned above, in certain embodiments, can answer the user's of Local or Remote equipment request, finish video conference.When processing 6100 receives the request that finishes video conference, handle 6100 and finish.
Otherwise whether it receives the request of the focal length of the remote camera of adjusting remote equipment to handle 6100 definite (6120).When handling 6100 when determining that it does not receive the request of focal length of the remote camera of adjusting remote equipment, handle 6100 returns 6110, receive other video from remote equipment.Three kinds of different modes of this request of generation that Figure 62,63 different embodiment with 64 graphic extensions provide to the user.In Figure 62,63 and 64, phase I 6210,6310 and 6472 shows that all the PIP of local device 6200,6300 and 6471 shows 6225,6335 and 6482, PIP shows that 6225,6335 and 6482 all show two videos, by the video of local device shooting and another video of taking by remote equipment.Viewing area 1155 among Figure 62 and 63 and 1155 shows the closing session button.But, in Figure 64, the layout of viewing area 1155 is identical with the layout of the viewing area of Figure 12 described above 1155.In addition, can be chosen in the switching camera button 6488 shown in the viewing area 1155,, perhaps call long-range switching camera operation in other embodiments to call the local camera operation of switching in certain embodiments.As shown in phase I 6210,6310 and 6472, the video that is presented at the remote equipment in the background demonstration 6235,6345 and 6480 blurs.
The video (for example, by single-point on the video of remote equipment 6240) of second stage 6215 graphic extensions of Figure 62 by selecting remote equipment simply, the method that the user of local device adjusts to remote equipment request focal length.According to the method, UI 6205 automatically the user to the selection of the area-of-interest that limited by square frame 6245 and user's instruction remote equipment to described area-of-interest executable operations (for example, focal length adjustment operation) expectation associates, thereby the video conference manager 1604 contact remote equipment of instruction local device 6200, operate to carry out adjustment (such as, focal length adjustment operation).The area-of-interest of definition is used for the calculating that focal length is adjusted by remote equipment.
The second stage 6315 of Figure 63 represents that similarly the local user is to the selection of long-distance video (for example, by user's dubbing the video of remote equipment).But, be different from the example of graphic extension among Figure 62, this selection instruction UI 6305 among Figure 63 shows as the menu in the optional UI project 6355,6360,6365 and 6370 (they can be realized as optional button) as shown in the phase III 6320.These optional UI projects comprise " automatic focus (Auto Focus) " project 6360, " automatic exposure (Auto Exposure) " project 6365, " switching camera (Switch Camera) " project 6370 and " cancellation (Cancel) " project 6355.In certain embodiments, " switching camera " optional UI project 6370 is used to ask local switching camera operation, and in other embodiments, " switching camera " optional UI project 6370 is used to ask long-range switching camera operation.Quadravalence section 6325 graphic extension local user is subsequently selected automatic focus project 6360.
The second stage 6474 of Figure 64 represents similarly that once more the local user is to the selection of long-distance video (for example, by user's dubbing the video of remote equipment).But, be different from the example of graphic extension in Figure 63, this selection instruction UI 6478 request focal length adjustment operations (that is, in second stage 6474) among Figure 64.After finishing focal length adjustment operation, UI 6478 shows the menu (that is, in the phase III 6476) of optional UI project 6484 and 6486, and optional UI project 6484 and 6486 can be realized as optional button.These optional UI projects comprise " automatic exposure " project 6486 and " cancellation " project 6484.
When processing 6100 determines that (6120) local users instruct local device request focal length adjustment operation, handle 6100 by video conference control channel, send (6140) order to remote equipment, catch and transmit the focal length of the camera of its video at present to adjust remote equipment.After 6140, handle and return operation 6110 described above.
In certain embodiments, before remote equipment carried out this operation, the user of remote equipment must provide permission, and in other embodiments, after the request that receives local device, remote equipment automatically performs this operation.In addition, in certain embodiments, the focal length setting of the camera that uses of remote equipment is adjusted in focal length adjustment operation during video conference.In some such embodiment, as mentioned above, some video conference functions are realized by video conference module 1602.In these embodiments, video conference manager 1604 instruction CIPU 1650 adjust the transducer of the remote equipment camera that is using.
The video of Figure 62,63 and 64 final stage 6220,6330 and the appropriate remote equipment that focuses on of 6476 expressions.Although Figure 62,63 and 64 provides the example of the focal length adjustment request that receives the focal length of proofreading and correct remote equipment, but, some embodiment allow the user of local device to ask local device to adjust the focal length of the camera of local device.Can be similar to the method for adjusting the focal length of its camera at the request remote equipment shown in Figure 62,63 and 64, produce such request.
Figure 62,63 and 64 graphic extensions allow the user to carry out three kinds of example user interface of focal length adjustment operation.In certain embodiments, focal length adjustment operation causes taking the change of operation of camera of two camera movement equipment of the video that is presented among the UI, such as the focal length of change camera.
As above described in Figure 56 and 62, the area-of-interest of regulation is respectively applied for the exposure adjustment of video and the calculating that focal length is adjusted by long-range mobile device.But, in some other embodiment, the user can be used to instruct remote equipment to carry out one or more operations to the selection of area-of-interest.For example, in certain embodiments, can expose and adjust and the focal length adjustment, thereby the instruction remote equipment carries out two operations according to the area-of-interest of definition.
E. frame rate control
During video conference, some embodiment may wish to adjust or keep another equipment in video conference to transmit speed (that is frame rate) by the video image of the camera of two camera movement equipment.For example, suppose bandwidth fixed, some such embodiment reduce the frame rate of video, and with the image quality of raising video image, and other such embodiment increases the frame rate of video, with smoothed video (that is, reducing shake).
Different embodiment is provided at during the video conference, the different technologies of the frame rate of control of video image.An example that illustrates is previously adjusted the VBI of the sensor assembly 415 of camera, with the speed of control and treatment by the image of camera.As another example, some embodiment of the management level 935 of the video conference module 925 shown in Fig. 9 come control frame speed by abandoning image.Similarly, some embodiment of image processing layer 930 come control frame speed by abandoning image.Some embodiment provide other technology in addition to come control frame speed, such as the frame that abandons in the general transmission buffer 3120.
V. two cameras
A. assembled view
1. picture-in-picture: the demonstration of two remote camera
Some embodiment allow two camera movement equipment any one in arranging according to several demonstrations, show from during video conference video of taking from this mobile device and the video of taking from another pair camera movement equipment.Figure 65 graphic extension shows the example of arranging from the difference of the video that one or more pairs of camera movement equipment are taken.In Figure 65, the another one user of the user of two camera movement equipment 6500 (device A) and another pair camera movement equipment 6505 (equipment B) is carrying out video conference mutually.
Four kinds of examples that Figure 65 arranges the demonstration of device A are illustrated in the left side.Four kinds of device A show that arranging is first view 6510, second view 6515, three-view diagram 6520 and the 4th view 6525.In addition, Figure 65 also is illustrated in the right side to four kinds of examples that the demonstration of equipment B is arranged.Four kinds of equipment B show that arranging is first view 6565, second view 6570, three-view diagram 6575 and the 4th view 6580.In this example, device A only shows two videos of the camera of slave unit A, and equipment B shows two videos of the camera of slave unit A, and shows one or two video of the camera of slave unit B.
In first view 6510, the UI 6585 of device A provides synthetic and shows 6512.Synthetic demonstration 6512 comprises two viewing areas: the viewing area 6535 of the video of the positive camera of the viewing area 6530 of the video of the back side camera of demonstration slave unit A and demonstration slave unit A.In this example, viewing area 6530 is positioned at the first half of synthetic demonstration 6512, and viewing area 6535 is positioned at synthetic 6512 the Lower Half that shows.In first view 6510, these two viewing area sizes are identical.The viewing area 6530 on top is just showing the video on mountain range, and the supposition of described mountain range is just by the mountain range of the back side camera of device A.Viewing area 6535 is just showing the tree and the man that attaches the names of pre-determined candidates, and described tree and the man that attaches the names of pre-determined candidates supposition is just by the tree and the man that attaches the names of pre-determined candidates of the positive camera of device A
UI 6585 in second view 6515 provides synthetic and shows 6517, synthetic show that 6517 comprise two viewing areas identical with first view 6510, except viewing area 6535 (video of the positive camera of demonstration slave unit A) is being positioned at synthetic 6517 the first half that shows, viewing area 6530 (video that shows the back side camera of slave unit A) is positioned at outside the Lower Half of synthetic demonstration 6517.
In three-view diagram 6520, UI 6585 provides PIP to show 6595.PIP shows that 6595 comprise two viewing areas: the viewing area 6535 of the video of the positive camera viewing area, that show slave unit A and the viewing area 6530 of inserting the video of the back side camera viewing area, that show slave unit A as prospect as a setting.In this view, background viewing area 6535 occupies most PIP and shows 6595, and it is less and overlap on the background viewing area 6535 of a part to insert viewing area 6530.
The PIP that UI 6585 in the 4th view 6525 also presents the viewing area 6530 that comprises as shown in three-view diagram 6520 and 6535 shows 6598.Be different from PIP and show 6595, PIP demonstration 6598 comprises the main as a setting viewing area 6530 that shows (taking from the back side of device A camera) and inserts the viewing area 6535 (taking from the front of device A camera) of demonstration as prospect.In addition, PIP demonstration 6598 presents (that is, the width of PIP demonstration 598 is greater than height) by transverse views.
Above two viewing areas 6530 and 6535 vertical stacked two views and two PIP views of video of two cameras of four kinds of possible synthetic views-wherein show, first equipment of UI of example graphic extension device A.Concerning the UI of device A, other view also is possible.For example, two viewing areas can level or the diagonal angle stacked, perhaps can use different PIP views.
The UI that represents equipment B about each view of equipment B graphic extension can adopt different views.These views comprise two cameras of slave unit A, and one one or the video of multi-section camera of equipment B.In first view 6565 of equipment B, the UI6590 of equipment B provides PIP to show 6568.PIP show 6568 comprise be presented at device A on the identical synthetic viewing area 6569 of synthetic demonstration 6512, and the insertion viewing area 6550 that shows the video of one of camera by equipment B (for example, front camera) shooting.Synthetic viewing area 6569 comprise the back side camera that shows slave unit A video viewing area 6531 and show the viewing area 6536 of video of the positive camera of slave unit B.The synthetic demonstration 6569 of the video of display device A occupies most PIP and shows 6568, and it is less and overlap synthetic the demonstration on 6569 to insert viewing area 6550.Viewing area 6550 shows smiling faces' video, and described smiling face's supposition is just by the smiling face of positive its video of camera of equipment B.
The UI 6590 of the equipment B in second view 6570 provides PIP to show 6572.PIP shows that 6572 comprise viewing area 6550 (video that shows the positive camera of slave unit B) and have the viewing area 6531 of video of the camera that shows slave unit A and 6536 synthetic demonstration 6573.Synthetic show that 6573 is identical with the synthetic demonstration 6517 in second view 6515 of device A, and occupy most PIP demonstration 6572.Show in 6568 equally with PIP in first view 6565, viewing area 6550 is less and overlap in the synthetic demonstration 6573.Specifically, in these two views, viewing area 6550 overlaps on the part of viewing area 6531 of video of the back side camera that shows slave unit A.
In three-view diagram 6575, UI 6590 provides the PIP in the three-view diagram 6520 with device A to show 6595 similar PIP demonstrations 6577.PIP shows that 6577 also comprise other viewing area 6550, and as the second insertion viewing area, second inserts viewing area 6550 overlaps on the background viewing area 6536.Two bottoms of inserting viewing area 6531 and 6550 Horizontal Tiles in background master viewing area 6536.
UI 6590 in the 4th view 6580 provides synthetic and shows 6582.Synthetic show 6582 comprise that three demonstration: PIP show 6583, viewing area 6550 and viewing areas 6540 (for example, be used to show by the back side camera of equipment B video).PIP show 6583 with the 4th view 6525 of device A in PIP show that 6598 is identical, and occupy most synthetic viewing area 6582. Viewing area 6540 and 6550 is less, and Horizontal Tile is under PIP viewing area 6583.
Although Figure 65 illustrates four kinds of possibility views of equipment B, but other many views also are possible.The synthetic demonstration of the background of the video of device A can Horizontal Tile, rather than vertical tile, inserting the viewing area can overlap on the camera viewing area, front of device A, rather than overlap on the camera viewing area, the back side, bigger viewing area can display device B the video of camera, rather than the video of the camera of device A, insert the viewing area and can be positioned at different positions, or the like.
The every group of arrow 6560 that comes from each view of device A shows and do not exist being illustrated in the demonstration on the device A and being illustrated in the requirement of the relevance between the demonstration on the equipment B.For example, even the arrangement viewing area video that device A is being pressed view 6510 (for example, the arrangement of selecting according to the user of device A), equipment B also can be by four kinds of arrangements of graphic extension, perhaps press among Figure 65 unshowned multiple other and arrange one of any display video (for example, the arrangement of selecting according to the user of equipment B).In other words, the demonstration of the demonstration arrangement of device A and equipment B is arranged irrelevant.Some embodiment do not send the viewing area to another equipment from an equipment, and just transmit video (for example, with the form of coding), and described video is presented in its corresponding viewing area by equipment.
2. special-purpose PIP
Some embodiment allow the user of two camera movement equipment during video conference, in PIP shows the prospect of video are superimposed upon on another video.In certain embodiments, the prospect of video is mixed in another video in such a manner, makes them show as demonstration by the single video of single camera.Figure 66 is illustrated in the PIP demonstration, inserts the example of this stack of prospect on background video of video.
Figure 66 is according to seven operational phases 6620,6625,6630,6635,6640,6660 and 6665 of UI 6670, and this video superimpose of graphic extension is operated.Phase I 6620 is illustrated in during the video conference with remote equipment, has the UI 6670 that PIP shows two camera movement equipment 6600 of 6682.As shown in the phase I 6620, PIP shows that 6682 comprise that two videos show: background master show 6610 and prospect insert and show 6605.Background master shows that 6610 occupy most UI 6670, and prospect insert to show that 6605 is less and overlap the background master and show on 6610.
In this example, background viewing area 6610 shows the video on mountain range, and the supposition of described mountain range is the mountain range that one of camera by remote equipment is taken.Prospect is inserted the video that viewing area 6605 shows the personage that attaches the names of pre-determined candidates, and described personage's supposition is the personage that one of camera by local device is taken its video.PIP show below 6682 be labeled as " End Conference (closing session) " optional UI project 6685 (for example, button 6685), described optional UI project 6685 allows this project of user by selecting (for example, by clicking or double-click this button), finishes video conference.
Calling of the optional menu 6675 of second stage 6625 graphic extensions.In certain embodiments, by selecting (for example touching) PIP viewing area 6682, can call the menu of optional UI project 6675.Replace this call operation, perhaps combine with this call operation, some embodiment also allow the user to pass through other operation, such as one or more other physics inputs of operating or utilizing equipment by different touch screens, call the menu of optional UI project 6675.
Phase III 6630 shows UI 6670, and UI 6670 has one group of optional UI project that is used to select the video superimpose operation calling.In this example, show the pop-up menu 6675 that demonstration has several optional UI projects on 6682 at PIP.The menu 6675 of optional UI project comprises " Flip PIP (upset PIP) ", and optional UI project 6640 (for example, button 6640), " Specialized PIP (special-purpose PIP) " optional UI project 6645 (for example, button 6645), " Cancel (cancellation) " optional UI project 6690 (for example, button 6690).In this example, select " Flip PIP " button 6640 can cause UI 6670 exchange backgrounds demonstrations 6610 and insert showing 6605 (will go through in the next section), select " Specialized PIP " button 6645 can cause the operation of UI 6670 beginning video superimpose, and select " Cancel " button 6690 to show 6682 and remove pop-up menu 6675 from PIP.Other embodiment comprises different or more project in PIP pop-up menu 6675.
Quadravalence section 6635 is illustrated in the user and has selected " Specialized PIP " button 6645 (for example dubbing button 6645 by the finger 6695 with him) UI 6670 afterwards.Highlight button 6645 by showing on 6670, indicate this selection at UI.Some embodiment use different indicated number (for example, highlighting the frame of selected item or the text in the selected item).
Five-stage 6640 is illustrated in the UI 6670 after the operation of beginning video superimpose.In this stage, UI 6670 allows the user to select him to want from which video to extract to want to use which video as the background in the overlay video as prospect and he.UI 6670 provides variety of option by being presented at the pop-up menu with several optional UI projects 6680 in the PIP demonstration 6682.The pop-up menu 6680 of optional UI project comprises " Select Inset (select to insert and show) ", and optional UI project 6655 (for example, button 6655), " Select Main (selecting main the demonstration) " optional UI project 6650 (for example, button 6650), " Cancel " optional UI project 6692 (for example, button 6692).
Select " Select Inset " button 6655 can cause UI 6670 that the prospects (that is the man who, attaches the names of pre-determined candidates) from the insertion video 6605 of the camera of local device are superimposed upon on background master's video 6610 from the camera of remote equipment.On the other hand, select " Select Main " button 6650 can cause 6670 prospects (that is mountain range) of UI to be superimposed upon from the insertion video 6605 on the camera of local device from background master's video 6610 of the camera of remote equipment.In certain embodiments, this causes the switching of two video push (feed), makes that the video in inserting viewing area 6605 will occupy most UI 6670 at present, and the video in main viewing area 6610 will be applied on the ahorse main video at present.Select " Cancel " button 6692 can abandon the video superimpose operation, and 6682 remove pop-up menus 6680 from the PIP viewing area.
The 6th stage 6600 was illustrated in the user and selects " Select Inset " button 6655 (for example, dubbing button 6655 by the finger 6695 with him) UI 6670 afterwards.Highlight button 6655 by showing on 6670, indicate this selection at UI.Some embodiment utilize different indicated number (for example, highlighting the frame of selected item or the text in the selected item).
The 7th stage 6665 was illustrated in finishes video superimpose operation UI 6670 afterwards.As shown in UI 6670, from viewing area 6605, extract the prospect (that is the man who, attaches the names of pre-determined candidates) of inserting viewing area 6605.From screen, eliminate to insert show 6605 window frame and background (that is, other except that described prospect all) in addition.At last, in such a manner prospect (that is, the man who attaches the names of pre-determined candidates) is mixed in the background video 6610, makes it show as single video.Can use various technology to remove the background of inserting video.The pixel that some embodiment identification is not moved with respect to other pixel, seek invariable pattern or color, use the baseline image of comparing with the image that comprises prospect and deduct difference or use different technology.
Although the example graphic extension of Figure 66 is on being superimposed on background viewing area 6610 time, the prospect of inserting viewing area 6605 is retained in the same position among the UI, but this example can how realizing of stack just.Some embodiment move to ad-hoc location among the UI 6670 (for example, one of center, corner, or the like) to the prospect video.Be similar in the feature shown in IV.A.1 and the IV.A.3 joint, some embodiment allow the user of local device to drag the prospect video of stack back and forth in UI, perhaps change the size of the prospect video of stack.
Can use different technology determine video image which or which partly be " prospect " of the operation of aforesaid video superimpose.This method of some embodiment determines which or which part (if any) of video image is dynamic.Dynamic part is considered to " prospect ", because normally static (that is, not the having motion) of the background of video image.In such embodiments, at specific a period of time inner analysis video image.If in described a period of time, the difference between the value of specific pixel is not more than the threshold value (for example, 5%, 10%, 15%) of regulation, thinks that so described specific pixel is a static pixels.After having analyzed each pixel in the video image, the dynamic pixel of video image (that is, not being static pixel) is considered to " prospect " of video image.
Figure 67 graphic extension can be by an example of this technology of video conference manager 1604 or image processing manager 1608 prospects that carry out, that determine video image.Specifically, Figure 67 graphic extension shows a series of six image 6705-6730 of the video of the personage attach the names of pre-determined candidates and tree.In this example, suppose that described personage is not fully still on one's feet, may speak.As mentioned above, analyzing each pixel in the video image, is dynamically or static to determine pixel.For example, determine that difference between the value of the pixel 6735 among the image 6705-6730 is whether greater than the threshold value of regulation.Here, because therefore the pixel 6735 part ground of representative except that the personage think that pixel 6735 is static.After all pixels in having analyzed image 6705-6730, determine that the personage in the image is dynamic, and the remainder of image is static.Thereby described personage is with by top " prospect " extracted with reference to the operation of Figure 66 explanation.
3. the video during the exchange picture-in-picture shows
Some embodiment allow the user of two camera movement equipment during video conference, two viewing areas during exchange PIP shows (that is, in PIP shows, insert the viewing area and become the background viewing area, the background viewing area becomes inserts the viewing area).Figure 68 is illustrated in during the video conference, the insertion viewing area 6605 in the exchange PIP demonstration 6682 and the example of background viewing area 6610.
Figure 68 is according to eight operational phases of the UI 6670 of the equipment among Figure 66 6800, and graphic extension exchange PIP operates.Preceding three phases among Figure 68 is identical with preceding three phases among Figure 66.In these stages, select by the touch-screen that utilizes local device, the user appears in the UI 6670 menu 6675.
The quadravalence section 6840 of Figure 68 is illustrated in the user and selects " Flip PIP " button 6640 (for example, dubbing button 6640 by the finger 6695 with him) UI 6670 afterwards.Highlight button 6640 by showing on 6670, indicate this selection at UI.Some embodiment utilize different indicated number (for example, highlighting the frame of selected item or the text in the selected item).
Five-stage 6845 is illustrated in and begins to exchange PIP operation UI 6670 afterwards.Some embodiment are by rotary movement, and flash demo inserts demonstration 6605 and background shows 6610 exchange.An example of this flash demo of Figure 68 graphic extension.In this example, show 6682 (before carrying out swap operation) on its one side by PIP, new PIP shows the upset of 6684 (after carrying out swap operation) the observation pane on its another side, and this animation can be described.Observe pane around being positioned at vertical axis 6686 Rotate 180s ° that PIP shows 6682 center.At five-stage 6845, observe pane and begin around vertical axis 6686 rotations.
In the 6th stage 6850, observe pane and be expressed as rotating about 90 °.This is to indicate with the fine rule 6688 that is presented at screen center (that is, observing the edge of pane).The rotation that pane is observed in the graphic extension of the 7th stages 6855 approaches to finish.New PIP shows that 6684 begin to occur from the another side of observing pane, and horizontal extension is to the screen that is full of equipment.PIP shows that 6684 are included in two viewing areas 6605 and 6610 of carrying out after the swap operation.The viewing area 6605 of video (from the camera of local device) that presents the man that attaches the names of pre-determined candidates is now in PIP shows 6684 background, the demonstration 6610 of video (from the camera of remote equipment) that presents the mountain range is now in PIP shows 6684 prospect, and described prospect overlaps and shows on 6605.Finishing of the 8th stage 6860 expression exchange display operations.
Those of ordinary skill will appreciate that the animation shown in Figure 68 is a kind of in may animation of PIP insertion/background swap operation multiple.For example, different embodiment can observe that pane, moment exchange these two viewing areas, viewing area of expansion dwindles another viewing area simultaneously along horizontal rotational shaft, or the like.Some embodiment provide a kind of animation that always is used for swap operation, and other embodiment allows the user to select from several animations, perhaps use different animation (for example, by selecting at random).In addition, swap operation can cause the variation of the image processing operations of two camera movement equipment, and such as the input that causes video conference manager 1604 response users, the proportional zoom that changes video is with synthetic.
4. lock onto the corner
Some embodiments of the present invention allow the user of two camera movement equipment to form the synthetic one or more viewing areas that show by moving around, and revise synthetic the demonstration.In the IV.A.1 joint, a this mobile example has been described above.When PIP shows that when comprising more than one insertion viewing area, it also is possible inserting this moving that shows.
Figure 69 is illustrated in this example of carrying out during the video conference.The example of graphic extension is similar with the example of graphic extension in Fig. 3 in Figure 69, except moving around, Figure 69 graphic extension comprises that two are inserted viewing area 6905 and 6910, rather than outside the insertion viewing area 6910 of the only PIP demonstration 6965 of a this insertion viewing area.
In Figure 69, with the long-distance user's of another equipment video conference during, the UI 6960 of mobile device 6900 presents PIP and shows 6965.PIP among Figure 69 shows that 6965 comprise that three videos show: background master shows that 6915 and two prospects insertions show 6905 and 6910.In this example, background master shows that 6915 present the personage's who plays the guitar while singing video, and described video is assumed to be it is video by the back side camera of remote equipment.Prospect insert to show that 6905 present the personage's who holds racket video, and in this example, this video is assumed to be it is video by the back side camera of local device.Another prospect is inserted and is shown that 6910 present the personage's who wears a hat video, and in this example, this personage's supposition is the personage by positive its video of camera of local device.PIP is the optional UI project 6970 (for example, button 6970) that is labeled as " End Conference " below showing 6965, optional UI project 6970 allow user by selecting it and finish video conference.
PIP shows that 6965 just present a kind of mode of the synthetic view of the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view.For example, replace having from the video of remote equipment show 6915 than overall background, bigger background shows that 6915 can be the video from local device, less prospect is inserted and is shown that 6905 and 6910 can be the video from remote equipment.In addition, some embodiment allow local video and long-distance video to appear at and insert demonstration 6905 and 6910 in a side, and background shows 6915 at opposite side, perhaps among the UI 6910 that all these three demonstrations are arranged side by side.In other embodiments, PIP shows that 6965 can comprise bigger background demonstration 6915 and/or less prospect insertion demonstration.In certain embodiments, the mode of PIP demonstration 6965 or acquiescence display mode can be specified by the user.
Figure 69 is by five different operational phases 6920,6925,6930,6935 and 6940 of reference, and two among the UI 6960 of graphic extension equipment 6900 are inserted moving of one of viewing areas.Phase I 6920 is illustrated in the UI 6960 during the video conference between the long-distance user of the local user of equipment 6900 and remote equipment.
Second stage 6925 graphic extensions are by selecting to insert viewing area 6910, and the user begins to lock onto the corner operation.In this example, insert in the viewing area 6910 Anywhere, make a choice by finger 6950 is placed on.As shown in the figure, show this selection with the thick frame 6962 that inserts viewing area 6910.Different embodiment can point out such selection with different modes, such as inserting viewing area 6910 by highlighting, inserts viewing area 6910 swings by making, or the like.
Phase III 6930 is illustrated in the user and begins PIP is shown that 6965 insertion viewing area 6910 shows that from PIP a zone 6965 moves to PIP and shows UI 6960 after another zone in 6965.In this example, insert viewing area 6910 and begun to move to the upper right corner of PIP demonstration 6965, shown in arrow 6955 from the lower right corner of PIP demonstration 6965.After selecting to insert viewing area 6910, the user drags its finger 6950 by show 6965 the upper right corner towards PIP, moves and inserts viewing area 6910.Some embodiment are provided at other technology that PIP shows the insertion viewing area 6910 that moves around in 6965.
Quadravalence section 6935 be illustrated in the user the screen of slave unit 6900 remove UI 6960 under the state of its finger after 6950.In this state, inserting viewing area 6910 still moves towards point the upper right corner of moving the PIP demonstration of discerning 6965 according to the user in the phase III.In other words, start to insert viewing areas 6910 show moving of 6965 the upper right corner towards PIP after at finger 6950, even after removing finger 6950, UI 6960 still keeps this and moves.Move in order to keep described, the UI 6960 of some embodiment required before the user removes its finger, and user's drag operation is greater than specific threshold quantity (for example, greater than specific distance, perhaps being longer than the specific time); Otherwise these embodiment can make this insertion viewing area remain on its initial position, the lower right corner after moving the insertion viewing area a little, perhaps move this and insert viewing area.
But, although even the user had stopped his drag operation before the insertion viewing area arrives its reposition, some embodiment still allow to insert the viewing area to be continued to move, but other embodiment requires the user to keep drag operation, till the insertion viewing area arrives its reposition.Some embodiment provide other technology of inserting the viewing area that moves.For example, some embodiment require the user viewing area 6910 is actual begin to move before, appointment will be directed to viewing area 6910 where, or the like.Some embodiment are also by only with different angles inclination mobile device, can be slided in the viewing area and lock onto the corner.
Five-stage 6940 is illustrated in and inserts the reposition UI 6960 afterwards that viewing area 6910 arrives its upper right corner that is positioned at PIP viewing area 6965.The elimination of thick frame 6962 indication locks onto the corner operation and finishes in the five-stage.
For make the above-mentioned the 3rd, the 4th and five- stage 6930,6935 and 6940 in graphic extension move easier, in case the UI 6960 of some embodiment adopts the user that insertion viewing area 6910 is moved towards certain corner of PIP demonstration 6965, just allows to insert the locking discipline that viewing area 6910 quick lock ins arrive this corner.For example, when dragging towards particular corner, the user inserts viewing area 6910 when surpassing threshold quantity, the direction of motion of viewing area 6910 is inserted in UI 6960 identifications of some embodiment, determine that described motion has surpassed threshold quantity, thereby automatically move subsequently and insert viewing area 6910, and further import the next grid point that inserts among the UI 6960 that viewing area 6910 can be locked into without the user.In certain embodiments, the unique grid point that provides for locking insertion viewing area 6910 is the grid point that is positioned at four corners of PIP demonstration 6965.Other embodiment provides other grid point of (for example, PIP shows in 6965) among the insertion UI 6960 that viewing area 6910 can be locked into.
Additional embodiments can not adopt grid point, makes insertion viewing area 6910 can be placed in the arbitrfary point in the PIP demonstration.Again additional embodiments provide allow that the user opens or closes UI lock onto the grid point feature.In addition, except the video that slave unit is taken, different embodiment can allow the user to various projects, locks onto the corner operation such as execution such as icons.As mentioned above, the mobile meeting of the synthetic one or more viewing areas that show causes the variation of the image processing operations of two camera movement equipment, such as the input that causes video conference manager 1604 response users, and the viewing area in synthetic again the demonstration.
5. push and locking
Example graphic extension among Figure 69 allows the user of two camera movement equipment to insert one of viewing areas to two, move to from the corner that PIP shows be not inserted into show another corner of occupying lock onto the corner operation.Some embodiment can realize that inserting demonstration to first moves to the position that second insertion shows, also inserts the propelling movement feature that demonstration is pushed to reposition to second.Figure 70 is illustrated in a this example of carrying out during the video conference.
Figure 70 is by with reference to six different phases 7020,7025,7030,7035,7040 and 7045 of UI 6960, and graphic extension is inserted and shown from a corner that PIP shows to not inserted by another and show moving of another corner that the PIP that occupies shows.Phase I 7020 is illustrated in the UI 6960 during the video conference between the long-distance user of the local user of this equipment and remote equipment.UI 6960 among Figure 70 represents and after the beginning video conference, the PIP identical at the PIP shown in the phase I of Figure 69 shows 6965.In this example, the video of being taken by local user's equipment is displayed in the insertion viewing area 6905 and 6910, and the video of being taken by long-distance user's equipment is displayed in the background viewing area 6915.
Second stage 7025 graphic extension user by selecting are inserted viewing area 6905, begin to lock onto the corner operation.In this example, by finger 7055 is placed on any place of inserting in the viewing area 6905, select.As shown in the figure, show this selection with the thick frame 7065 that inserts viewing area 6905.Different embodiment indicates such selection with different modes, such as by highlighting district 6905, swings by making viewing area 6905, or the like.
Phase IIIs 7030 graphic extension is shown in arrow 7050, begin to shift to the lower right corner of PIP demonstration 6965 (by during phase III from the lower left corner of PIP demonstration 6965 the user inserting viewing area 6905, after select inserting viewing area 6905, show that towards PIP 6965 the lower right corner drags his finger 7055) afterwards UI 6960.Some embodiment are provided at other technology that PIP shows the insertion viewing area 6905 that moves around in 6965.
Quadravalence section 7035 is illustrated in inserts viewing area 6905 contact insertion viewing areas 6910 UI 6960 afterwards.When contact, insert viewing area 6910 and move towards the nearest corner of the next one.In this example, 6910 beginnings of insertion viewing area are moved along the direction (shown in arrow 7075) in the upper right corner of PIP demonstration 6965.Thick frame 7070 with inserting viewing area 6910 shows the activation of this push operation.Different embodiment can indicate this activation with different modes, such as by highlight the district 6910, or the like.
Five-stage 7040 is illustrated in and inserts viewing area 6905 and be locked into UI under the state that before is inserted into after the lower right corner that viewing area 6910 occupies.In this example, still move towards the upper right corner of PIP demonstration 6965 this insertion viewing area.In addition, no longer show thick frame 7065.As long as user's drag operation of phase III 7030 is just removed from its corner inserting viewing area 6910, and is locked onto next minimum corner greater than making insertion viewing area 6905 lock onto the threshold value that right corner falls always.
Some embodiment comprise that determining to adopt which kind of mode to push second inserts one group of rule of viewing area 6910.Under the situation of graphic extension, some embodiment manage to continue to keep inserting the rotation of viewing area in Figure 70.That is be to move in the counterclockwise direction owing to insert viewing area 6905, so viewing area 6910 is also moved counterclockwise.Some embodiment provide the classification of the possible position that pushed insertion viewing area 6910 can move to, and select first position that is not occupied in this tabulation.For example, when the insertion viewing area that is positioned at the lower right corner was pushed by the insertion viewing area from the lower left corner, the upper right corner may be first position in this tabulation.But, if the 3rd insertion viewing area has been positioned at the upper right corner, some embodiment can move to the next option (for example, the upper left corner, center or the lower left corner) in the tabulation so.Other embodiment can insert the viewing area along with the second insertion viewing area pushes the 3rd together, and the equipment that makes does not need to determine that the 3rd inserts the reposition of viewing area.
The 6th stage 7045 was illustrated in inserts the reposition UI 6960 afterwards that viewing area 6910 arrives its upper right corner that is positioned at PIP viewing area 6965.In this stage, the elimination of thick frame 7070 indication locks onto corner-push operation and finishes.Be similar to reference to the corner that is pushed to of Figure 68 explanation and operate, the variation of moving the image processing operations that can cause two camera movement equipment of the synthetic one or more viewing areas that show, such as the input that causes video conference manager 1604 response users, the viewing area in synthetic again the demonstration.
6. rotation
When the user of the mobile device that is used for video conference rotated this equipment in the session, some embodiment were rotated in the PIP that presents during the video conference and show.Figure 71 graphic extension when equipment 7100 when the upright position is rotated to horizontal level, the UI of this equipment shows 7175 rotation.When the long limit of screen when being vertical, equipment 7100 is held by perpendicular, and when the level of the long limit of screen, equipment 7100 is held by horizontal.In Figure 71 in the example of graphic extension, UI shows that 7175 rotate to the horizontal transverse views of optimizing of holding for equipment 7100 from hold vertical view of optimizing for equipment perpendicular.This spinfunction makes is held or during horizontal holding, the user can both watch the UI 7175 with the stand up position demonstration by perpendicular when mobile device 7100.The example of graphic extension is similar to the example of graphic extension among Figure 34 among Figure 71, except Figure 71 graphic extension rotation comprises that two are inserted the viewing area and not just PIP demonstration of inserting the viewing area.
In Figure 71, with the long-distance user's of another mobile device video conference during, the UI 7175 of mobile device presents PIP and shows 7180.PIP among Figure 71 shows that 7180 comprise that three videos show: background master shows that 7115 and two prospects insertions show 7110 and 7160.In this example, background master shows that 7115 present the video on mountain range, and described video supposition is by the front of remote equipment or the video of back side camera.Prospect is inserted and is shown that 7110 present the video of the smiling face in the room, and described video supposition is by the front or the back side camera of local device.Another prospect is inserted and is shown that 7160 present the guitar player's of singing video, and described guitar player supposition is just by the guitar player of another its video of camera of local device.PIP is " closing session " button 7155 below showing 7180, and the user can select this button to finish video conference (for example, by singly refer to dub).This PIP shows a kind of mode of the synthetic view that just presents the video of being taken by remote equipment and local device.Some embodiment can provide other synthetic view, show such as tiled view or different PIP.
Figure 71 is according to six different operational phases 7120,7125,7130,7135,7140 and 7145, the rotation of graphic extension UI 7175.Phase I 7120 is illustrated in the UI7175 during the video conference between the long-distance user of the local user of this equipment and remote equipment.
Second stage 7125 is illustrated in the user and begins banking equipment 7100 UI 7175 afterwards.In this example, equipment 7100 has begun equipment 7100 is held from erecting to hold to tilt to by horizontal, shown in arrow 7185.The outward appearance of UI 7175 does not change.In other cases, the user may change into want equipment 7100 from by horizontal hold to tilt to by perpendicular hold, in these cases, UI shows that 7175 views from level optimization switch to the view of vertical optimization.
Phase III 7130 is illustrated in equipment 7100 from being tilted to by the UI under the state after horizontal the holding 7175 by perpendicular holding.In this state, UI shows that 7175 outward appearance does not still change.In certain embodiments, be tilted above threshold quantity, and keep surpassing threshold quantity after a period of time, trigger rotary manipulation at equipment 7100.In Figure 71, in the example of graphic extension, suppose that threshold quantity and rotary speed can not make UI show 7175 rotations, till the short period interval after equipment is placed on horizontal level.Different embodiment has different threshold quantities and the stand-by period that is used to trigger rotary manipulation.For example, some embodiment can have the threshold value of so low triggering rotary manipulation, make UI 7175 to be seemed the not orientation of tube apparatus 7100 always show with stand up position as it.In other embodiments, the user of equipment 7100 can stipulate when to trigger rotary manipulation (for example, preferentially selecting to be provided with by menu).In addition, some embodiment can not postpone described rotation after equipment is tilted above threshold quantity.In addition, different embodiment can allow to trigger rotary manipulation with different modes, such as by switch switch on the mobile device, by send voice commands, according to the selection of being undertaken by menu, or the like.
Quadravalence section 7135 is illustrated in the UI 7175 after the beginning rotary manipulation.Some embodiment flash demo rotation viewing areas are to provide the feedback about rotary manipulation to the user.The example of a kind of animation that Figure 71 graphic extension is such.Specifically, Figure 71 begins rotation together in its quadravalence section 7135 performance viewing areas 7110,7115 and 7160. Viewing area 7110,7115 and 7160 centers on axle 7165 (that is z the axles) rotation through the center of UI demonstration 7175. Viewing area 7110,7115 is rotated identical quantity with 7160, but along with the opposite direction rotation of the rotation of equipment 7100 inclination of equipment 7100 (for example, by).In this example, because equipment 7100 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), so rotary manipulation can make viewing area 7110,7115 and 7160 half-twist in the counterclockwise direction.When viewing area 7110,7115 and 7160 rotations, viewing area 7110,7115 and 7160 dwindles pari passu, shows 7175 to be fit to UI, makes viewing area 7110,7115 and 7160 still can appear at fully in the UI 7175.Some embodiment can provide the message (for example, by showing words " Rotating (rotation) ") of the state of indicating equipment 7100.
Five-stage 7140 is illustrated in viewing area 7110,7115 and 7160 and is rotated counterclockwise UI 7175 after 90 ° from vertical view to transverse views.In this stage, viewing area 7110,7115 and 7160 has been rotated, but also is not deployed into the whole width of UI 7175.The ending that arrow 7170 is pointed out at five-stage, viewing area 7110,7115 and 7160 will begin side direction and launch, to be fit to the whole width of UI 7175.Different embodiment can not comprise this stage, because can side by side carry out described expansion with the rotation in the quadravalence section 7135.
The 6th stage 7145 was illustrated in viewing area 7110,7115 and 7160 and has been unfolded, with the UI 7175 after the whole demonstration that occupies UI 7175.As mentioned above, other embodiment can differently realize this rotation.Concerning some embodiment, only the screen of equipment rotation is surpassed threshold quantity and just can trigger described rotary manipulation, and the orientation of tube apparatus 7100 not.
In addition, other embodiment can be provided for indicating the different animations of rotary manipulation.The rotary manipulation of carrying out in Figure 71 relates to UI and shows the 7175 center rotations around UI demonstration 7175.On the other hand, can make the viewing area center on they separately the central shaft of viewing area rotate separately.Represented a kind of such method among Figure 72, Figure 72 represents the alternative approach of rotation of the PIP viewing area 7180 of flash demo UI 7175.Among Figure 72 the PIP of graphic extension show with Figure 71 in the PIP of graphic extension show that 7180 is identical.
Figure 72 is according to six different operational phases 7120,7125,7130,7220,7225 and 7230, and graphic extension PIP shows 7180 rotation.The operation of the preceding three phases of UI 7175 is with identical as the operation of the preceding three phases of explanation among the UI 7175 in Figure 71.In the phase III of Figure 71 and 72, equipment all becomes horizontal holding from perpendicular holding, and the rotation of UI 7175 does not also begin.
The alternative approach of the described rotation of quadravalence section 7220 graphic extension flash demos.In the quadravalence section, rotary manipulation begins.Specifically, quadravalence section 7220 shows the beginning of the rotation of viewing areas 7110,7115 and 7160. Viewing area 7110,7115 and 7160 rotates around axle 7250 (that is the z axles) at the center of passing each viewing area respectively.Viewing area 7110,7115 is rotated identical amount with 7160, but rotates along the direction opposite with the rotation (for example passing through the inclination of equipment 7100) of equipment 7100.In this example, because equipment 7100 has been rotated in a clockwise direction 90 ° (by becoming horizontal holding from perpendicular holding), so rotary manipulation can make viewing area 7110,7115 and 7160 half-twist in the counterclockwise direction.When viewing area 7110,7115 and 7160 rotations, they are also scaled, show 7175 to be fit to UI, make viewing area 7110,7115 and 7160 still can appear at fully on the UI 7175.
Five-stage 7225 is illustrated in viewing area 7110,7115 and 7160 and all is rotated counterclockwise UI 7175 after 90 ° from vertical view to transverse views.In this stage, viewing area 7110,7115 and 7160 has been rotated, but also is not deployed into the whole width of UI 7175, perhaps arrives their final position.The viewing area PIP show in 7115 the final position by as show that at PIP the position in 7115 is definite (for example in the viewing area as shown in the phase I 7120, insert and show that 7110 the lower left corner and insertion demonstrations 7160 in PIP demonstration 7180 show 7180 the lower right corner at PIP).
The ending that arrow 7170 is pointed out at five-stage, viewing area 7115,7110 and 7160 will begin side direction and launch, till main viewing area 7115 is fit to the whole width of UI 7175.In addition, arrow 7255 is pointed out to insert viewing area 7110 and 7160 and will be moved, thereby arrives their final positions in PIP demonstration 7180.In other words, insertion viewing area 7110 will be moved towards the lower left corner of PIP demonstration 7180 downwards, and another inserts the lower right corner that PIP demonstration 7180 is shifted in viewing area 7160.Different embodiment can differently realize this animation, for example by utilizing the locking and the push operation of graphic extension in Figure 71.Thereby the 6th stage 7230 was illustrated in the final position UI 7175 afterwards that viewing area 7110,7115 and 7160 has been unfolded the whole demonstration that occupies UI 7175 and has moved to them.
As mentioned above, other embodiment can differently realize this rotation.For example, similar with graphic extension in Figure 36 and 37, some embodiment provide a kind of rotary manipulation, the directed change that wherein shows the viewing area of the video of taking by local device, to be reflected in local device is rotated the operation orientation of local device afterwards, some embodiment provide a kind of rotary manipulation, the directed change that wherein shows the viewing area of the video of taking by remote equipment, to be reflected in remote equipment is rotated the operation orientation of remote equipment afterwards, some embodiment provide viewing area 1155 wherein to be retained in the rotary manipulation of same position, some embodiment provide in the viewing area different layouts (for example, the layout of the viewing area 1155 of Figure 12), perhaps their combination.
Concerning some embodiment, only make the screen rotation of equipment surpass threshold quantity and just can trigger rotary manipulation, and the orientation of tube apparatus 7100 not.In addition as mentioned above, local device and remote equipment can be by the control communication ports, and notice is carried out the modification of any correspondence to a rotary manipulation that equipment carries out in two equipment to allow another equipment to the video of a described equipment mutually.In addition, the animation of rotary manipulation also can cause the variation of the image processing operations of the operation of camera or two camera movement equipment, be presented at image in one or more viewing areas such as the one or more viewing areas that make the one-tenth different angles among the synthetic again UI 1105 of video conference manager 1604 and proportional zoom.
7. select to watch remote view
As mentioned above, some embodiment allow the user of two camera movement equipment before the beginning video conference, perhaps when the beginning video conference, select which camera is used for video conference.Replace this ability, perhaps combine with this ability, some embodiment allow the user of this equipment to select between two videos, described two videos be in video conference, show and or from two video cameras of remote equipment, perhaps from two video cameras of user's local device.Figure 73 is illustrated in a video of selecting in the meeting in two long-distance videos, and Figure 74 is illustrated in a video of selecting in the meeting in two local videos.
Figure 73 is according to six operational phases 7335,7340,7345,7350,7355 and 7360 that are presented at the UI 7375 on the local device 7300, the selection of the described long-distance video of graphic extension.Phase I 7335 graphic extension have with the long-distance user's of mobile device video conference with two cameras during, the initial p IP that presents shows 7390 UI 7375.
As shown in the phase I 7335, initial p IP shows that 7390 comprise three demonstrations: background master shows that 7315 and two prospects insertions show 7305 and 7310.Background shows that 7315 occupy most PIP viewing area 7390, and the prospect insertion shows that the 7305 and 7310 part backgrounds that overlap respectively on the UI 7375 show on 7315.In this example, background shows that 7315 are presented on microphone personage's before video, and described video supposition is the video by the back side camera of remote equipment.First prospect is inserted and is shown that 7305 present the video of man's face, and in this example, this video supposition is the video that one of camera by local device 7300 is taken.Second prospect is inserted and is shown that 7310 present the personage's who wears a hat video, and in this example, this video supposition is the video by the positive camera of remote equipment.
Initial p IP demonstration 7390 just presents a kind of mode by the synthetic view of the video of the camera of local device and remote equipment.Some embodiment can provide other synthetic view.For example, the background demonstration can present the video of one of camera from local device, and less prospect is inserted demonstration and can be presented from the front camera of remote equipment and the video of back side camera.In addition, in some cases, PIP shows that a background video can including only all from remote equipment shows and a prospect video demonstration.In certain embodiments, the mode of PIP demonstration or acquiescence display mode can be specified by the user.
The beginning of second stage 7340 graphic extension video selection operations.In this example, by calling one group of optional UI project that will be presented in the PIP demonstration 7390, initiate this operation.Described one group of optional UI project presents and is used to select to show the option of using long-distance video.In certain embodiments, by selecting any viewing area of the broadcast long-distance video on (for example) UI 7375, call described one group of optional UI project by touching.In other embodiments, by selecting any place on (for example) UI 7375, can call described each project by touching.Replace this call operation, perhaps combine with this call operation, some embodiment also allow the user to pass through other operation, such as by different touch screen operations, perhaps utilize one or more other physics inputs of equipment, call described one group of optional UI project.
Phase III 7345 shows UI 7375, and UI 7375 has one group of optional UI project 7380 that is used to select long-distance video of calling.In this example, the one group of optional UI project 7380 that is the pop-up menu form is displayed in the PIP viewing area 7390, overlaps in this PIP demonstration.Described one group of optional UI project 7380 (they can be realized as optional button) (for example comprises " Select R1 " optional UI project 7320, button 7320), " Select R2 " optional UI project 7325 (for example, button 7325), " Select Both " optional UI project 7330 (for example, button 7330) and " Cancel " optional UI project 7385 (for example, button 7385).In this example, select " Select R1 " button 7320 can make the video (be presented on background show 7315 in) of 7375 demonstrations of UI by the back side camera of remote equipment.Select " SelectR2 " button 7325 can make the video (be presented on prospect insert show 7310 in) of 7375 demonstrations of UI by the positive camera of remote equipment.Select " Select Both " button 7330 can make UI 7375 continue to show by the front camera of remote equipment and two videos of back side camera.Select " Cancel " button 7385 can cancel this operation.In certain embodiments, the video of being taken by local device is not subjected to the influence of the selection carried out on this menu.
Quadravalence section 7350 is illustrated in the user and has selected " Select R1 " button 7320 (for example, dubbing button 7320 by the finger 7365 with him) UI 7375 afterwards.By highlighting the button 7320 on the UI 7375, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the frame of selected item, perhaps the text in the selected item).
Five-stage 7355 is illustrated in the user and has selected from the video of R1 so that the animation of the UI 7375 after showing.In this example, by shown in arrow 7370, make unwanted prospect insert the right hand edge that viewing area 7310 skids off PIP demonstration 7390, UI 7375 removes unwanted prospect and inserts viewing area 7310.Other embodiment utilizes different animations to eliminate unwanted insertion viewing area, such as making this insertion viewing area fade out or fade out, move this insertion viewing area or eliminate this insertion viewing area simply immediately along different directions.
The 6th stage 7360 was presented at and finishes after the video selection operation UI 7375 during the video conference.Video viewing area 7310 no longer is presented on the UI 7375.In this stage, UI7375 presents new PIP and shows 7395, and PIP demonstration 7395 comprises the video viewing area 7315 of main demonstration as a setting and the video viewing area 7305 that demonstration is inserted in conduct.
In certain embodiments, this video selection operation also can make remote equipment only show the video of selection, but in other embodiments, the video selection operation to remote equipment without any influence.In certain embodiments, this video selection operation can make remote equipment stop to transmit unwanted video to local device.In fact, in certain embodiments, lid video selection operation can make the camera of remote equipment stop to take unwanted video.In certain embodiments, the user of remote equipment can veto these influences to remote equipment.
The remote view that top example illustrates selection is the situation that has been presented at the remote view in the background master demonstration.In certain embodiments, when the user selected to be presented at the remote view that inserts in one of demonstration, the remote view of selection was displayed in the background master demonstration.In this case, animation like some such embodiment utilizations and the animated type that in Figure 68, shows.In addition, the selection of long-distance video can cause the variation of the image processing operations of local two camera movement equipment, such as the input that makes video conference manager 1604 response users, and only synthetic one or more long-distance videos of selecting in synthetic the demonstration.
8. select to watch local view
Figure 74 is according to six operational phases 7435,7440,7445,745,7455 and 7460 that are presented at the UI 7475 on the local device 7400, the selection of graphic extension local video.Phase I 7435 graphic extension UI 7475, UI 7475 have with the long-distance user's of mobile device video conference with at least one camera during the initial p IP that presents show 7490.PIP demonstration 7490 and the PIP in the phase I in Figure 73 7335 show similar, except being different from Figure 73, background demonstration 7415 presents the video by the man of the camera of remote equipment, the prospect in left side insert to show that 7410 present the video by the personage who holds guitar of the back side camera of local mobile device, and the prospect on right side is inserted and shown outside 7405 videos that present by the man who wears a hat of the positive camera of local mobile device 7400.Thereby, only show a long-distance video, and show two local videos.
The beginning of second stage 7440 graphic extension video selection operations.In this example, to be presented in the PIP demonstration 7490 by calling, be used to select to show one group of optional UI project using long-distance video, initiate this operation.In certain embodiments, show any viewing area of the broadcast local video on 7475, can call described one group of optional UI project by selecting (for example) UI by touching.In other embodiments, show any place on 7475, can call described each project by selecting (for example) UI by touching.Replace this call operation, perhaps combine with this call operation, some embodiment also allow the user to pass through other operation, such as by different touch screen operations, perhaps utilize one or more other physics inputs of equipment, call described one group of optional UI project.
Phase III 7445 shows UI 7475, and UI 7475 has one group of optional UI project 7480 that is used to select local video of calling.In this example, the one group of optional UI project 7480 that is the pop-up menu form is displayed in the PIP viewing area 7490, overlaps in this PIP demonstration.Described one group of optional UI project 7480 (for example comprises " Select L1 " optional UI project 7420, button 7420), " Select L2 " optional UI project 7425 (for example, button 7425), " Select Both " optional UI project 7430 (for example, button 7430) and be used to cancel " Cancel " optional UI project 7485 (for example, button 7485) of this operation.In this example, select " Select L1 " button 7420 can make the video (be presented on prospect insert show 7410 in) of 7475 demonstrations of UI by the back side camera of local device.Select " SelectL2 " button 7425 can make the video (be presented on prospect insert show 7405 in) of 7475 demonstrations of UI by the positive camera of local device.Select " Select Both " button 7430 can make UI 7475 continue two videos of demonstration, and select 7485 these operations of meeting cancellation of " Cancel " button by two camera of local device.In certain embodiments, the video of being taken by remote equipment is not subjected to the influence of the selection undertaken by this menu.
Quadravalence section 7450 is illustrated in the user and has selected " Select L2 " button 7425 (for example, dubbing button 7425 by the finger 7465 with him) UI 7475 afterwards.By highlighting the button 7425 on the UI 7475, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the frame of selected item, perhaps the text in the selected item).
Five-stage 7455 is presented at the user and has selected from the video of L2 so that the animation of the UI 7475 after showing.In this example, by shown in arrow 7470, unwanted prospect is inserted shown that 7410 skid off the left hand edge of PIP demonstration 7490, UI 7475 removes unwanted prospect and inserts demonstration 7410.Other embodiment utilizes different animations to eliminate unwanted insertion viewing area, such as making this insertion viewing area fade out or fade out, move this insertion viewing area or eliminate this insertion viewing area simply immediately along different directions.
The 6th stage was presented at finishes after the video selection operation UI7475 during the video conference.Video viewing area 7410 is no longer on UI 7425.In this stage, UI 7475 presents new PIP and shows 7495, and PIP shows that 7495 comprise the main as a setting long-distance video demonstration 7415 that shows and show 7405 as inserting the local video that shows.In certain embodiments, this video selection operation only influences local the demonstration, because two videos still are transmitted to remote equipment.Other embodiment stops from the camera video of removing.
The local view that top example illustrates selection is the situation that has been presented at the view in the background master demonstration.In certain embodiments, when the user selected to be presented at the local view that inserts in one of demonstration, the local view of selection was displayed in the background master demonstration.In this case, animation like some such embodiment utilizations and the animated type that in Figure 68, shows.When the local view in the background master shows is removed, other embodiment will utilize and insert remote view.
The remote view selection operation that illustrates with reference to Figure 73 above being similar to, the selection of local video can cause the variation of the image processing operations of local two camera movement equipment, such as the input that makes video conference manager 1604 response users, only synthetic one or more long-distance videos of selecting in synthetic the demonstration.The selection of local video also can cause the variation of operation of one or more cameras of local device.For example, some embodiment make the camera of unwanted video stop to transmit unwanted video to remote equipment, and other embodiment makes this camera stop to take unwanted video.
9. select to transmit local view
Each top height joint is for example understood the modification that in the meeting video is shown.Some embodiment also allow the user of two camera movement equipment before the beginning video conference, select which camera is used for video conference.Before Figure 75 is illustrated in meeting, from two videos taking by two camera movement equipment of user, select a video, be used for the operation of video conference.
Figure 75 is according to eight operational phases of UI 7500, and graphic extension is used for the selection of the local video of video conference.The UI 7500 of the two camera movement equipment 7518 of phase I 7502 graphic extension, UI 7500 have the initial p IP demonstration 7542 that presents after the user asks to begin video conference with the long-distance user of mobile device.
As shown in the phase I 7502, initial p IP shows that 7542 comprise that two videos show: background master show 7520 and prospect insert and show 7522.Background master show 7520 and prospect insert to show 7522.Background master shows that 7520 occupy most of display screen of equipment, and prospect insert to show that 7522 is less and overlap in the background master demonstration.In this example, background shows that 7520 present the personage's who holds guitar video, and this video supposition is the video by the back side camera of equipment.Prospect is inserted and is shown that 7522 present the personage's who wears a hat video, and in this example, this video supposition is the video by the positive camera of equipment.
Initial p IP demonstration 7542 just presents a kind of mode by the synthetic view of the video of the camera of local device.Some embodiment can provide other synthetic view.For example, background shows the video that can present from the front camera of equipment, and less prospect is inserted and shown the video that can present from the back side camera of equipment.In addition, some embodiment allow two videos to appear among the UI 7500 two viewing area (for example, left and right sides display window, perhaps display window up and down) side by side, perhaps in the viewing area that two diagonal angles are arranged.In certain embodiments, the mode of PIP demonstration or acquiescence display mode can be specified by the user.Below the PIP demonstration is the optional UI project 7540 (for example, button 7540) that is labeled as " End Conference ", and optional UI project 7540 allows these projects of user by selecting and finishes video conference.
In the phase I 7502, the user of mobile device 7518 has asked to carry out video conference with the long-distance user, and is waiting for that the long-distance user responds." Preview, Waiting for response... (preview, positive wait-for-response ...) " in the bottom surface of display frame explain graphic extension should the waiting period.
The beginning of second stage 7504 graphic extension video selection operations.In this example, by calling one group of optional UI project that will be presented in the PIP demonstration 7542, initiate this operation.Described one group of optional UI project presents and is used to select local video, with the variety of option of the remote equipment that sends video conference to.In certain embodiments, when waiting for long-distance user's response, show any place on 7500, can call described one group of optional UI project by selecting (for example, touching) UI in the time before meeting.Replace this call operation, perhaps combine with this call operation, some embodiment also allow the user to pass through other operation, such as by different touch screen operations, perhaps utilize one or more other physics inputs of equipment, call described one group of optional UI project.
Phase IIIs 7506 graphic extension has one group of user that supplies who calls and selects the UI 7500 of the optional UI project 7526 of video.In this example, the one group of optional UI project 7526 that is the pop-up menu form is presented in the PIP viewing area 7542, and overlaps in the PIP demonstration.In this example, described one group of optional UI project comprises: " Transmit L1 " project 7528 (for example, button 7528); " Transmit L2 " project 7530 (for example, button 7530); " Transmit Both " project 7532 (for example, button 7532); " Cancel " project 7534 (for example, button 7534).In this example, select " Transmit L1 " button 7528 can make UI 7500 during video conference, only handle sends remote equipment to by the video of the back side camera of equipment.Select " Transmit L2 " button 7530 can make UI 7500 during video conference, only handle sends remote equipment to by the video of the positive camera of equipment.Select " Transmit Both " button 7532 can make 7500 two videos of UI send the long-distance user of video conference to, and select " Cancel " button 7534 can cancel these operations by the front and back camera of equipment.
Quadravalence section 7508 is illustrated in the user and has selected " Transmit L1 " button 7528 (for example, dubbing button 7528 by the finger 7524 with him) UI 7500 afterwards.By highlighting the button 7528 on the PIP viewing area 7542, indicate this selection.Some embodiment utilize different indicated number (for example, highlighting the frame of selected item or the text in the selected item).
Five-stage 7510 is illustrated in the video that the user has selected the back side camera of equipment, to send the animation of remote equipment UI 7500 afterwards to.In this example, by shown in arrow 756, unwanted prospect is inserted shown that 7522 skid off the right hand edge of PIP demonstration 7542, UI 7500 removes unwanted prospect and inserts demonstration 7522.In the 6th stage 7512, insert demonstration 7522 and from PIP viewing area 7542, removed fully.Different embodiment utilizes different animations to remove unwanted viewing area, such as making this viewing area fade out or fade out, move this viewing area or eliminate this viewing area simply immediately along different directions.
The 7th stage 7514 was illustrated in the animation that the long-distance user has accepted video conference request UI 7500 afterwards.By eliminating " Preview, the Waiting for response... " note in the display frame, highlight the acceptance of video conference request.In this stage, narrow down to the lower left corner of PIP viewing area 7542 gradually as background viewing area 7520, shown in arrow 7538 from the video of the back side camera of equipment.Background shows that 7520 dwindle, and makes UI 7500 to show after viewing area 7520 to comprise viewing area 7544 from the video of long-distance user's camera.Some embodiment narrow down to different positions to local camera, utilize synthetic demonstration of tiling of two videos that show, perhaps make remote view become the insertion viewing area that PIP shows.
The 8th stage 7516 was illustrated in finishes video selection operation UI 7500 afterwards.UI 7500 presents new PIP and shows 7546, PIP show 7546 insertions that comprise the video of taking from local device show 7520 and the background of the video that sends from remote equipment show 7544.
B. Dai Kuan ﹠amp; Frame rate
In certain embodiments, local mobile device is redistributed distribute to the bandwidth of each video of taking by two cameras (that is, front camera and back side camera) of local mobile device during the video conference to the adjustment of the size of the viewing area of long-range mobile device.Figure 76 is illustrated in two examples that this bandwidth between two cameras of local device is redistributed.
Each example among Figure 76 relates to the back side camera sensor 7605 of local device, the positive camera sensor 7610 of local device, the video conference module 7615 of local device and the UI 7635 of long-range mobile device 7620.Back side camera sensor 7605 and positive camera sensor 7610 are caught from the respective back surfaces camera of local device and the video of front camera.Video captured is sent to video conference module 7615, and video conference module 7615 is handled video captured, and they are sent to remote equipment, to show in UI 7635.
In Figure 76, the UI 7635 of remote equipment presents synthetic the demonstration.The synthetic demonstration represented by the front camera of local device and the video of back side camera.From the video capture of front camera tree and the man that wears a hat, and from the video capture mountain range landscape of back side camera.As shown in Figure 76, according to the arrangement of the viewing area that is used for display video and the size of viewing area, available many different modes show these two videos in UI 7635.In each example, video conference module 7615 according to the relative size of the viewing area in the remote equipment, is distributed total output bandwidth at first between each video.Specifically, the video that is presented in the big viewing area among the UI 7635 is assigned with the total bandwidth of major part, and is presented at the bandwidth that video in the less viewing area among the UI7635 is assigned with smaller portions.In certain embodiments, when video is displayed in the viewing area of identical size, between these two videos, divide total output bandwidth equally.
The amount of distributing to the bandwidth of each video in these two videos can influence the mode of handling each video most.For example, video may need the bandwidth bigger than the bandwidth of distributing to this video.In this case, the frame rate of video is adjusted, and perhaps the size of video image is dwindled by ratio, so that adapt to less bandwidth.The frame rate that reduces video causes video seem " discontinuous ", and ratio is dwindled the zone that the size of video image can reduce to show this video.So when video was assigned with a certain amount of bandwidth, some embodiment adjusted the frame rate of this video, ratio is dwindled the size of video image, perhaps carries out the two combination, so that guarantee to transmit this video in the bandwidth of distributing.Those of ordinary skill in the art will appreciate that the adjustment of frame rate and average frame size can change, and to obtain best overall video quality, still guarantees to transmit this video simultaneously in the bandwidth of distributing.
The example of Figure 76 (1) illustrates a kind of situation that bandwidth is redistributed with two operational phases of UI 7635.The UI 7635 of the remote equipment 7620 in the phase I 7670 presents the synthetic demonstration that comprises two demonstrations: one at the top of UI 7635, and another is in the bottom of UI 7635.In this example, the video that the viewing area 7625 at top shows by the positive camera of local device, the video that the viewing area 7630 of bottom shows by the back side camera of local device.As shown in the phase I 7670, the viewing area 7625 at top is greater than the viewing area 7630 of bottom.Thereby, be assigned with 80% bandwidth from the video of the front camera of local device, be assigned with 20% bandwidth from the video of the back side camera of local device.In order to ensure can be in the bandwidth of distributing, the video from the back side camera of local device is sent to remote equipment from local device, adjust the frame rate and/or the scaling size of this video.
The user that second stage 7675 is illustrated in remote equipment increases the size of viewing area, bottom, makes the size of viewing area, top 7625 and viewing area, bottom 7630 be similar to identical UI 7635 afterwards.As a result, each video is redistributed 50% total bandwidth by video conference module 7615.
The example of Figure 76 (2) illustrates the another kind of situation that bandwidth is redistributed with two operational phases of UI 7635.In the phase I 7680 of example (2), UI 7635 presents PIP and shows.This PIP demonstration comprises two demonstrations: background master viewing area 7650 and prospect are inserted viewing area 7655.Background master viewing area 7650 occupies most PIP and shows, and that prospect is inserted viewing area 7655 is less and overlap on the background master viewing area 7650.In this example, background viewing area 7650 presents the video by the positive camera of this equipment.Insertion viewing area 7655 presents the video by the back side camera of this equipment.As shown in this stage, background viewing area 7650 is greater than inserting viewing area 7655.Thereby, be assigned with 80% bandwidth from the video of the front camera of equipment, be assigned with 20% bandwidth from the video of the back side camera of equipment.In order to ensure in the bandwidth of distributing, can send video to remote equipment from local device from the back side camera of local device, adjust the frame rate and/or the scaling size of video.
The user that second stage 7685 is illustrated in remote equipment has exchanged the UI 7635 after the demonstration of these two videos.Specifically, background viewing area 7660 presents the video by the back side camera of equipment now, and insertion viewing area 7665 presents the video by the positive camera of equipment now.Because the size of the viewing area of these two videos changes, therefore be assigned with 80% bandwidth from the video of the back side camera of equipment, be assigned with 20% bandwidth from the video of the front camera of equipment.Thereby, be reduced from the frame rate and/or the big young pathbreaker of scaling of the video of the front camera of local device.Those of ordinary skill in the art will appreciate that the bandwidth distribution that illustrates is an example in Figure 76, during video conference, it also is possible distributing other technology of bandwidth between two cameras.
1. frame rate control
Be similar to the frame rate control operation in the meeting described above, some embodiment may wish to adjust separately or remain on another equipment in video conference and transmit speed by the video image of every camera of two camera movement equipment.Some such embodiment provide similar techniques described above.For example, some embodiment control the frame rate of every camera by the VBI of the sensor assembly 415 of every camera of adjustment.Other embodiment also provides other technology, such as being abandoned by the frame that the sensor assembly 415 and/or the general transmission buffer 3120 of every camera are carried out.
2. the bandwidth of passing ratio convergent-divergent control
As mentioned above, during the video conference between two camera movement equipment and another equipment, in the certain hour amount, the quantity (that is, network connects bandwidth) that connects the view data that can transmit by one or more networks is limited.In order to maximize and keep the throughput that network connects, the different embodiment of two camera movement equipment provide different modes to be controlled in the special time amount, connect the quantity of the view data that transmits by network.In certain embodiments, throughput is the Mean Speed by the success message transmission of communication port (for example, network connects).
When the image that transmits by two camera of two camera movement equipment, a kind of such method adjustment connects the quantity of the view data that transmits from the size of the image of one one or two camera of two camera movement equipment by network with control.Some embodiment ratios are dwindled the size of the image of being taken by two camera movement equipment, to reduce the quantity that connects the view data that transmits by network, and the size of other embodiment ratio enlarged image, to increase the quantity that connects the view data that transmits by network.
Some embodiment keep the depth-width ratio (that is even convergent-divergent) of image when convergent-divergent.Other embodiment zoomed image makes the depth-width ratio of scaled images be different from the depth-width ratio of original image (that is distortion convergent-divergent).
In addition, can carry out proportional zoom in the different phase of image processing process.The proportional zoom of some embodiment can be undertaken by camera sensor.In such embodiments, the multirow of the discardable image of camera sensor or multi-column data (that is pixel value).In some such embodiment, remaining view data is by interpolation, with the profile of smoothed image.
The proportional zoom of other embodiment is carried out by the scaler module 455 of CIPU 400.In certain embodiments, proportional zoom is carried out by video conference manager 1604, and as mentioned above, in other embodiments, proportional zoom is carried out by encoder.Thereby the different embodiment of two camera movement equipment differently carry out proportional zoom.
3. Bit-Rate Control Algorithm
Some embodiment provide different mechanism to manage the bit rate by the video coding of the camera of two camera movement equipment.In certain embodiments, two camera movement equipment comprise the rate controller that is used for every camera.Some embodiment provide the fixed bit rate Managed Solution.According to this scheme, each rate controller is set to fixed bit rate, makes that the gross bit rate from the video of two cameras on the mobile device is constant.Other embodiment provides precedence scheme, wherein when requiring to reduce gross bit rate, always obtains the priority higher than another video from one of two videos of the camera of equipment.
In certain embodiments, two rate controllers of two cameras of arbiter module management.The example of this arbiter module of Figure 77 graphic extension.As shown in Figure 77, rate controller 7700 is set the bit rate of front camera, and rate controller 7705 is set the bit rate of back side camera.Rate controller sends to encoder 7715 to the image from camera sensor.Arbiter module 7710 is connected with two rate controllers, and according to such as the video size of each video in available bandwidth, two videos etc., controls the setting of the bit rate of each rate controller 7700 and 7705 in any multiple mode.To guarantee under available bandwidth, can all send two videos to remote equipment.In addition, arbiter 7710 can be configured to realize above mentioned fixed rate scheme, perhaps precedence scheme.
In some other embodiment, two rate controllers that are used for two cameras can intercom mutually.According to this scheme, rate controller can exchange their information of video separately, and sets the bit rate of video in view of the above.Several examples of rate controller speed administrative mechanism are provided.But, many other different mechanisms also are possible.
4. Video processing
Some embodiment of two camera movement equipment differently handle the image by two camera of two camera movement equipment under different situations.For example, when processing comprised PIP composograph by the image of two camera of two camera movement equipment, some embodiment carried out TNR to the PIP composograph selectively and handle 2000.Some this embodiment only carry out TNR to the master image in the PIP composograph and handle 2000, and other this embodiment only carries out TNR processing 2000 to the insertion image in the PIP composograph.
As processing another example by the image of two camera of mobile device, some embodiment are according to the various changes of video conference, such as the user (for example to the adjustment of the viewing area of display video, main viewing area/insertion the viewing area that amplify the insertion viewing area of PIP demonstration, in the video that shows, define area-of-interest, exchange PIP shows, or the like), to the change of total available bandwidth, or the like, proportional zoom is by the image of two camera of two camera movement equipment.Some such embodiment are the proportional zoom image in the manner described above.That is, image can be by encoder 1655, video conference manager 1604, scaler module 455 with camera sensor (that is, 405a or the 405b) proportional zoom of its photographic images.
5. encode
As mentioned above, some embodiment send the video from two cameras of two camera movement equipment.Thereby these embodiment can be to the video coding by two camera, so that send remote equipment to during video conference.Different embodiment provides different modes to come encoded video so that transmission.Figure 78 graphic extension utilizes multiplexer (MUX) 7815, coder module 7825, buffer 7830 and composite module 7835 to handle video, so that the method for transmission.
According to selecting signal, MUX 7815 obtains an input signal, and the input signal of selecting is exported to encoder 7825.For example, if the input signal of selecting signal indication MUX 7815 to obtain from C1, MUX 7815 selects this input signal so, and exports this input signal.Available multiple mode such as by the instruction from video conference manager 1604, provides described selection signal.By MUX 7815, encoder 7825 alternately becomes bitstream format to the image encoding that receives from MUX7815, and image encoded is kept in the buffer 7830.Composite module 7835 combinations (that is, multiplexed) are kept at the one or more bit streams in the buffer 7830, and export single bit stream.
According to three phases 7860,7865 and 7870, the operation of this coding method is described now.In the phase I 7860, MUX 7815 is configured to receive the image of being taken by camera C 1 7805, and exports to encoder 7825 so that coding.The image encoding of 7825 pairs of receptions of encoder also produces bit stream 7850, and bit stream 7850 is stored in the buffer 7830 subsequently.Second stage 7865 is similar to the phase I 7860, except MUX 7815 is configured to receive the image of being taken by camera C 2 7810, and exports to encoder 7825, so that outside the coding.Similarly, encoder is to the image encoding that receives and produce bit stream 7855, and bit stream 7855 is stored in the buffer 7830.In the phase III 870, composite module 7835 is fetched bit stream 7850 and 7855 from buffer 7830, and they are combined into a bit stream, so that send remote equipment to.
Figure 79 graphic extension is to two video codings from two camera movement equipment, so that send the another kind of method of remote equipment to during video conference.In this method, from the frame of video of first camera of mobile device (promptly, image) and to be synthesized from another frame of video of second camera of mobile device be a frame of video, He Cheng frame of video is encoded into bit stream afterwards, so that send to remote equipment.As shown in Figure 79, this method comprises synthesizer 7915, buffer 7920 and encoder 7925.
As shown in the figure, synthesizer 7915 synthetic from first camera image 7905 and from the image 7910 of second camera, to form composograph 7955.Different embodiment is composograph 7905 and 7910 differently.For example, the synthesizer 7915 of some embodiment can be by being arranged in (as shown in Figure 80) adjacent one another are, composograph to two images.Composograph 8030 and 8035 graphic extensions utilize two example composographs of this technology.In composograph 8030, be arranged on the image 7910 from second camera from the image 7905 of first camera.And composograph 8035 presentation videos 7905 are arranged in the left side of image 7910.
At some among other the embodiment, synthesizer 7915 can by two images 7905 and 7910 are superimposed upon bigger background image above, synthetic two images 7905 and 7910.Composograph 8040 graphic extensions of Figure 80 utilize the example composograph of this technology.In composograph 8040, image 7905 and 7910 diagonal angles are arranged, and are superimposed upon (that is, image 7905 is positioned at the upper left corner of background image, and image 7910 is positioned at the lower right corner of background image) on the blank image.In certain embodiments, camera sensor may vary in size, thereby takes the image with different pixels resolution.In such an embodiment, synthesizer 7915 can according to as by the similar manner composograph 7905 and 7910 of composograph 8045 graphic extensions of Figure 80.After synthetic these two images, synthesizer 7915 is kept at composograph in the buffer 7920.Encoder 7925 is fetched composograph from buffer 7920, and composograph is encoded into bit stream, and sends it to the remote equipment of video conference.
With reference now to graphic extension among Figure 79, synthesizer 7915, buffer 7920 and encoder 7925 illustrate each operation.At first, first camera sends to synthesizer 7915 to the image 7905 as the part of a series of images in the video.Simultaneously, second camera sends to synthesizer 7915 to the image 7910 as the part of a series of images in the video.Synthesizer 7915 is composograph 7905 and 7910 in the manner described above subsequently, to form composograph 7955.Afterwards, synthesizer 7915 sends to buffer 7920 to composograph 7955.Buffer 7920 is preserved composograph subsequently, afterwards composograph is sent to encoder 7925.At last, encoder 7925 is encoded into bit stream to composograph, and sends it to the remote equipment of video conference.
Figure 81 graphic extension is to two video codings from two camera movement equipment, so that send another method of remote equipment to during video conference.In this method, be shown with the synthetic form that shows from two videos of this equipment, obtain the synthetic screenshot capture that shows, and it is encoded into bit stream, so that send to remote equipment.As shown in Figure 81, this method comprises encoder 8115.In certain embodiments, 8115 pairs of synthetic image encodings of encoder, and send to remote equipment.
With reference now to graphic extension among Figure 81, encoder 8115 illustrates each operation.At first, the video from two cameras of two camera movement equipment is displayed on the screen of equipment with the synthetic form that shows.Described synthetic demonstration can present video by any-mode.For example, the synthetic demonstration among some embodiment can be presented on PIP to two videos and show, in the PIP demonstration 8105 such as graphic extension in Figure 81.In other embodiments, synthetic demonstration can be presented on two viewing areas side by side to two videos, perhaps in the viewing area of two diagonal angle arrangements.Obtain PIP and show 8105 screenshot capture,, and send it to encoder 8115 such as image 8110.Encoder is encoded into bit stream 8120 to a series of screenshot capture subsequently, sends it to the remote equipment of video conference afterwards.Although several distinct methods to two video codings have been described above, but other method also is possible.
6. decoding
Some embodiment of two camera movement equipment can receive with top method bitstream encoded with reference to figure 78-81 explanation.In such embodiments, two camera movement equipment can receive (for example, by video conference control channel) and indicate the information that is used for the method for video coding.Figure 82 graphic extension is a kind of to by communication network, and the bit stream decoding of two videos that receive from another equipment so that during video conference, is presented at the method on two camera movement equipment.Specifically, this method is used for decoding with top coding method bitstream encoded with reference to figure 78 explanations.
As shown in Figure 82, this scheme is used separation module 8235, buffer 8230 and 8290 and decoder module 8225.Separation module 8235 separates bit stream (that is, the multichannel decomposition) and becomes one or more bit streams, and bit stream is kept in the buffer 8230.Decoder 8225 is fetched bitstream encoded, and their decodings to produce video, are kept at video in the buffer 8290 subsequently.
With reference now to the separation module 8235 of graphic extension among Figure 82, buffer 8230 and 8290 and decoder module 8225,, the operation of this method is described.At first, two camera movement equipment are by communication network, and another equipment from video conference receives bit stream 7845 (that is, at networking manager 1614).Separation module 8235 is separated into two bit streams 8255 and 8260 to the bit stream that receives, because the bit stream that receives is the multiplexed bit stream of two bit streams.The video data that on behalf of one of two cameras of slave unit, each bitstream encoded take.Subsequently, separation module 8235 is kept at bit stream 8255 and 8260 in the buffer 8230.
Afterwards, decoder 8225 is fetched bit stream 8250 (it be two bit streams 8255 and 8260) from buffer 8230, and 8250 decodings of 8225 pairs of bit streams of decoder producing video 8280, and are kept at video 8280 in the buffer 8290.Decoder 8225 is also to another decoding in bit stream 8255 and 8260, and a video that produces is kept in the buffer 8290.Now, two videos can be retrieved from buffer 8290, and are saved or are presented on two camera movement equipment.
Figure 83 graphic extension is to using the method with reference to the method bitstream encoded decoding of Figure 79 explanation.As shown in Figure 83, this method comprises decoder 8325, buffer 8320 and decombiner 8315.
In certain embodiments, the method bitstream encoded that decoder 8325 receives with graphic extension among Figure 79 becomes one or more composographs to bit stream decoding, and described composograph is stored in the buffer 8320 subsequently.Decombiner 8315 extracts these two images from each composograph.In order from composograph, to extract these two images, the information that decombiner 8315 also receives the indication position of each image in composograph (for example, by video conference communication control channel, the information that the equipment of the synthetic and coded image from video conference receives).
With reference now to graphic extension among Figure 83, decoder 8325, buffer 8320 and decombiner 8315 illustrate the operation of this method.At first, decoder 325 is from the bit stream of another mobile device receiver, video of video conference, such as the bit stream of the video of using the method generation that illustrates with reference to Figure 79.Decoder 8325 becomes to comprise one or more composographs of composograph 7955 to this bit stream decoding, and they are kept in the buffer 8320.Buffer 8320 is preserved composograph subsequently, afterwards they is sent to decombiner 8315.When decombiner when buffer 8320 receives composograph 7955, it composograph 7955 be separated into Figure 79 in image 7905 and 7910 identical two images 7905 and 7910.
When the system from the system that illustrates such as Figure 81 received bit stream, decoder became a series of screenshot capture to bit stream decoding such as the decoder among Figure 83 8325.Described a series of screenshot capture is displayed on the screen of equipment as video, and does not further handle.
VI. many sources
As mentioned above, video can be by two camera of two camera movement equipment, and send another equipment in the video conference to.Some embodiment can be in company with from the video of a camera of two camera movement equipment together, transmission is presented at different media contents or the arbitrary content on two camera movement equipment, rather than transmits from the video of two camera of two camera movement equipment.In other words, these embodiment can send the content from a plurality of sources in company with by the video of the camera of two camera movement equipment together.
Figure 84 is the video conference of two camera movement equipment of some embodiment of graphic extension and the another kind of software architecture of processing module conceptually.The video conference of Figure 84 and processing module are similar to video conference and the processing module 1600 of Figure 16, except this video conference and processing module comprise display driver 8485 and memory 8475, and exchange of media module 1620 comprises that medium come outside source module 8470 and the screen capture module 8480.
The medium of some embodiment come source module 8470 route media content between video conference module 8402 and memory 8475.The example of media content comprises video, image, document and music.Other embodiment is kept at the media content of other type in the memory 8475.The memory 8475 of some embodiment is that (for example, RAM), and the memory 8475 of other embodiment is external memory storage (for example, compact flash (CF) card, secure digital (SD) card, or the like) to internal storage.
In certain embodiments, screen capture module 8480 is by display driver 8485, and route is presented at the image of the content on the display of two camera movement equipment.In certain embodiments, display driver 8485 is responsible for catching the content on the display, and content is converted to image.Different embodiment catches the different content that is presented on the display.For example, some embodiment catch all the elements that are presented on the display.Other embodiment catch display specific viewing area (for example, the viewing area of current active window, the viewing area that PIP shows, or the like).
With reference now to Figure 84,, some exemplary operations of video conference and processing module is described.For in company with from the video of the camera of two camera movement equipment transfers media content together, the video conference module 8402 of some embodiment is carried out with top in video conference module 1602 identical operations illustrated in fig. 16, except replacing fetching image from CIPU 1650, video conference manager 1604 comes source module 8470 by medium, fetches outside the media content from memory 8475.In order to transmit the image of the content on the display that is presented at two camera movement equipment, some embodiment of video conference manager 1604 fetch the image of the content on the display that is presented at two camera movement equipment by display driver 8485.Some embodiment carry out similarly handling (for example, perspective correction, convergent-divergent etc.) with the processing that the image of fetching from CIPU 1650 is carried out to the image that is presented at media content on the display or content, and other embodiment does not carry out any processing.
Above discussion described in company with together by the video of a camera of two camera movement equipment, send several examples from the content in various sources.But, other embodiment can transmit other dissimilar content.For example, in the video conference that relates to a plurality of participants, the video that some embodiment receive an equipment from video conference and send another equipment to by the video of the camera of two camera movement equipment.Thereby, in company with together, can send content from any number of different types in any a plurality of sources by the video of a camera of two camera movement equipment.
VII. multipart video-meeting
Above-mentioned each joint that relates to video conference has been described the video conference with two participants.But, the multipart video-meeting (that is the participant more than three) by the mobile device of some embodiment also is possible.In certain embodiments, all participants in the multipart video-meeting can see and hear mutually.Other embodiment provides the meeting of a kind of broadcast video in many ways, one of them participant (for example, the broadcaster) can see and hear all other participant, and all other participant can see and hear this broadcaster, but other participant can not see or hear (unless for example obtaining broadcaster's approval) mutually.
A. the user interface of multipart video-meeting
During multipart video-meeting, some embodiment provide the various participants that are used for the display video meeting, and select the different UI of the one or more particular participant that will watch.For example, some embodiment of mobile device provide all participants that show multipart video-meeting simultaneously, and allow the user of mobile device to select one of participant so that watch the UI of (for example, by amplifying selected participant's image).Figure 85 illustrates the example of this UI.
Figure 85 is with reference to five different phases 8505,8510,8515,8520 and 8525 of UI 8530, be illustrated in all participants that show multipart video-meeting among the UI 8530 of mobile device 8500 simultaneously, and select one of participant so that the sequence of operations of watching.Phase I 8505 is illustrated in the multipart video-meeting UI 8530 afterwards between other three users that set up miscellaneous equipment.As shown in the figure, UI 8530 comprises synthetic demonstration 8535 and viewing area 1155.Synthetic demonstration 8535 comprises four viewing areas 8565,8570,8575 and 8580 of demonstration by the image of the participant's of multipart video-meeting camera.In this example, the user of viewing area 8565 expression mobile devices 8500 (that is, viewing area 8565 demonstrations are by the image of the positive camera of mobile device 8500).Viewing area 1155 and front are 1155 identical in viewing area illustrated in fig. 12.
One of viewing area of the synthetic viewing area 8530 of user by selecting of second stage 8510 expression mobile devices 8500, beginning participant selection operation.Especially, second stage 8510 expression users select viewing area 870 (for example, by dub finger 8550 on viewing area 8570).
The phase III 8515 of UI 8530 is illustrated in finishes the synthetic demonstration 8555 afterwards of participant's selection operation.Some embodiment provide the animation (not shown) that shows second stage 8510 and the transformation between the phase III 8515.Synthetic demonstration 8555 comprises that PIP shows 8560, PIP shows 8560 the participant's who selects viewing area (promptly in second stage 8510, viewing area 8570) be shown as the background viewing area, and the insertion viewing area that user's viewing area 8565 is shown as PIP demonstration 8560.In this example, PIP shows that 8560 show the image that is stretched horizontally with the selected viewing area 8570 that is fit to transversal orientation.In certain embodiments, this image is not stretched, and selects the image of viewing area to keep its portrait orientation (that is, as shown in Figure 36, being filled in the redundant space of both sides, background viewing area with secret note).In addition, synthesize and show that 8555 also comprise synthetic demonstration 8585, synthetic demonstration 8585 shows the image that the ratio of two unselected viewing areas 8575 and 8580 is dwindled.
The user by selecting PIP of quadravalence section 8520 expression mobile devices 8500 shows 8560 (for example, by dubbing finger 8550 in the PIP demonstration 8560), and the beginning participant cancels selection operation.Five-stage 8525 is illustrated in the participant and cancels the synthetic demonstration 8535 of selection operation after finishing.
Figure 85 represents to show simultaneously all participants of multipart video-meeting, carries out participant's selection operation and carries out the exemplary operations sequence that the participant cancels selection operation.Other sequence of operation also is possible.For example, after the phase III 8515, replace the beginning participant to cancel selection operation, the user can select to be presented at synthetic the demonstration and not select one of viewing area in 8585, show 8560 background viewing area (that is the previous viewing area of selecting) with newly select viewing area and PIP in the exchange viewing area 8585.Thereby, during multipart video-meeting, the user can be at any time and the viewing area and the PIP that repeatedly exchange arbitrarily in the viewing area 8585 show 8560 background viewing area.In addition, in the time of any during multipart video-meeting, the user can carry out the participant and cancel selection operation, syntheticly shows 8535 to return.In addition, different embodiment allows the user according to different modes, such as the switch that switches on the mobile device 8500, by sending voice commands etc., selects specific participant.
Some embodiment provide the technology of for example selecting the participant according to speech detection automatically.In such embodiments, when one of participant spoke, this participant's viewing area was elected as PIP automatically and is shown 8560 background viewing area.When a different participant spoke, this participant's viewing area was elected as PIP automatically and is shown 8560 background viewing area.In certain embodiments, when the participant of multipart video-meeting speaks, ormal weight mourn in silence (for example, 3 seconds) afterwards, display shows and syntheticly shows 8535.In certain embodiments, when the user of mobile device 8500 spoke, any operation did not take place yet on the UI 8530 of mobile device 8500.
Figure 86 graphic extension shows all participants of multipart video-meeting simultaneously and selects one of participant so that the sequence of operation of another example of watching.Figure 86 is with reference to seven different phases 8505,8605,8610,8615,8620,8625 and 8630 of UI 8645, this operation among the UI 8645 of graphic extension mobile device 850.The phase I 8505 of phase I 8505 and graphic extension in Figure 85 is identical, because it is presented at the multipart video-meeting UI 8645 afterwards between other three users that set up miscellaneous equipment.
Second stage 8605 graphic extensions are by selecting viewing area 8570 (for example, by two fingers are placed on the viewing area 8570), and the user of mobile device 8500 begins participant's selection operation.The conversion stage of phase III 8610 expression participant selection operations.In this stage, the user drags two fingers away from each other, makes viewing area 8570 become bigger simultaneously, and is full of the viewing area that is used to show synthetic demonstration 835.This example represents that viewing area 8570 is selected, but other viewing area 8565,8575 and 8580 also can be selected arbitrarily.In certain embodiments, do not allow the user of mobile device 8500 to select this user's viewing area (that is the viewing area 8565 in this example).
The quadravalence section 8615 of UI 8645 is presented at the PIP that finishes participant's selection operation UI8645 afterwards and shows 8635.Some embodiment require the user to continue to drag away from each other two fingers, be full of up to viewing area 8570 till the background viewing area 8640 of PIP demonstration 8635, and other embodiment only required before the user removes finger, user's drag operation greater than specific threshold quantity (for example, greater than specific distance, perhaps greater than specific time quantum).When user's drag operation satisfied or surpass the certain threshold level amount, UI 8645 continued the amplification of viewing areas 8570, till it is full of PIP and shows 8635 background viewing area 8640.Otherwise participant's selection operation is not finished, and UI 8645 returns synthetic demonstration 8535.As shown in the figure, the viewing area of selection (that is, viewing area 8570) is that PIP shows 8635 background viewing area 8640, and user's viewing area 8565 is that PIP shows 8635 insertion viewing area.Some embodiment provide the animation (not shown) that shows the transformation between phase III 8610 and the quadravalence section 8615.
Five-stage 8620 graphic extensions show 8635 background viewing area 8640 (for example, by two fingers are placed on the background viewing area 8640) by selecting PIP, and the user of mobile device 8500 begins the participant and cancels selection operation.The 6th stages 8625 expression participant cancels the conversion stage of selection operation.This stage graphic extension user closely drags finger mutually, to dwindle the viewing area of the background viewing area 8640 that is used to PIP demonstration 8635.Be similar to the operation of explanation in the phase III 8610, some embodiment required before the user removes finger, and user's drag operation is greater than certain threshold level amount (for example, greater than specific distance, perhaps greater than specific time quantum).Otherwise the participant cancels selection operation and does not finish, and UI 8645 returns PIP and shows 8635.The 7th stage 8630 of UI 8645 is illustrated in finishes the participant and cancels synthetic demonstration 8535 after the selection operation.
Figure 86 graphic extension shows all participants of multipart video-meeting simultaneously, carries out participant's selection operation and carries out another exemplary operations sequence that the participant cancels selection operation.But, some embodiment allow the user of mobile device 8500 to repeat participant's selection operation and the participant cancels selection operation.Such embodiment of Figure 87 graphic extension.
Specifically, Figure 87 is with reference to seven different phases 8505,8705,8615,8710,8715,8720 and 8725 of UI 8730, and the exemplary sequence that participant's selection operation and participant cancel selection operation is repeatedly carried out in graphic extension.Phase I 8505 is identical with 86 phase I 8505 with above mentioned Figure 85.Second stage 8705 is similar to the second stage 8605 of Figure 86, except the user by dubbing viewing area 8570 once (rather than be placed on the viewing area 8570) to two fingers, select outside the viewing area 8570.The quadravalence section 8615 of phase III 8615 and Figure 86 is identical, finishes the PIP demonstration 8635 afterwards of participant's selection operation because it is illustrated in.Quadravalence section 8710 is similar to the five-stage 8620 of Figure 86, except the user by dubbing background viewing area 8640 once (rather than be placed on the background viewing area 8640) to two fingers, select PIP to show outside 8645 the background viewing area 8640.
The 7th stage 8630 of five-stage 8715 and Figure 86 is identical, finishes the synthetic demonstration 8535 afterwards of cancellation selection operation because it is illustrated in the participant.The 6th stage 8720 was similar to second stage 8510, except viewing area 8575 is carried out participant's selection operation.Similarly, the 7th stage 8725 was similar to the phase III 8705, showed 8635 background viewing area 8640 because it is expressed as PIP to the viewing area of selecting (that is, the viewing area 8575).Although Figure 87 has only represented that a little participant selects and the participant cancels selection operation, but during multipart video-meeting, can carry out any this operation repeatedly.
In addition, some embodiment are provided at during the video conference, can show the participant's of different numbers UI.For example, when mobile device was uprightly held (that is, portrait orientation), the UI of some embodiment only showed some participants of multipart video-meeting, when mobile device is laterally held (that is, transversal orientation), showed other participant.When other embodiment is laterally held at mobile device, show all participants.In addition, some embodiment provide animation indicate and Figure 34,35,36 and 37 in the diverse location of similar mobile device of graphic extension and/or the transformation between the orientation.Other different animation also is possible.
As during video conference, show another example of UI of the participant of different numbers, some embodiment allow the user of mobile device to select the multidigit participant that will watch simultaneously during video conference.For the purpose of illustrating, referring to the phase I 8505 of Figure 85, some such embodiment allow the user of mobile device 8500 to select two or more viewing areas (for example, by dubbing the synthetic corresponding viewing area that shows in 8535) in viewing area 8565,8570,8575 and 8580.The viewing area of selecting can be shown according to variety of way subsequently, any one in arranging such as the demonstration of synthesizing graphic extension among demonstration, PIP demonstration, Figure 65, and other various multi-players show arrangement.In addition, although the example of some embodiment has been described, but, those of ordinary skill will appreciate that different embodiment can select and show the multidigit participant of multipart video-meeting according to any multiple mode.
B. the user interface of broadcast video meeting in many ways
As mentioned above, the broadcast video meeting only allows a participant to hear and sees all other participants in many ways, and other participant can not hear or see mutually.In many ways the broadcast video meeting is easier in order to make, and some embodiment are provided for showing the broadcaster of broadcast video meeting in many ways and other participant's numerous different UI.For example, some embodiment provide student-teacher's formula UI layout, are similar to the layout of the phase III 8515 of graphic extension in Figure 85.Thereby, with reference now to the described phase III 8515, student-teacher's formula UI layout of some embodiment is described.
In these embodiments, have only the broadcaster to be displayed on PIP and show in 8560 the whole viewing area (that is, do not show and insert the viewing area).In many ways other participant of broadcast video meeting is displayed on below the PIP demonstration 8560, is similar in the synthetic viewing area that shows in 8585 that shows.With top described similar, in certain embodiments, when mobile device is in vertical pattern, synthetic other participant who shows demonstration defined amount in 8585, and when mobile device was in transverse mode, the synthetic demonstration can show other or all participants in 8585.In addition, other embodiment provides the broadcaster that shows broadcast video meeting in many ways and other participant's different UI.
C. control the audio frequency of multipart video-meeting
In addition, the mobile device of some embodiment provides the different technologies of a plurality of participants' that control multipart video-meeting audio frequency.For example, some embodiment of mobile device allow the user of mobile device by the single group volume control (for example, the volume sliding shoe) on the UI that is presented at this embodiment, the audio frequency of every participant in the control multipart video-meeting.In other embodiments, mobile device allows the user of mobile device by one group of volume control, such as the volume sliding shoe in the viewing area that is presented at each participant, controls the volume of the audio frequency of every participant in the multipart video-meeting respectively.Some embodiment only provide mute button, rather than one group of volume control.Thereby, in some such embodiment, the user of mobile device is merely able to all participants quiet or that cancel quiet multipart video-meeting, and in other this embodiment, the user of mobile device can distinguish each participant quiet or that cancel quiet multipart video-meeting.In addition, other technology of audio frequency of the participant of control multipart video-meeting also is possible, such as by switch switch on the mobile device, by sending voice commands or the like.
VIII. electronic system
Many above-mentioned features and application are realized as software process, and described software process is defined as the one group of instruction that is recorded on the computer-readable recording medium (being also referred to as computer-readable medium).When these instructions were carried out by one or more processing units (for example, one or more processors, the nuclear core of processor, perhaps other processing unit), they made processing unit carry out the action of indicating in instruction.The example of computer-readable medium includes but not limited to CD-ROM, flash drive, RAM chip, hard disk drive, EPROM or the like.Computer-readable medium does not comprise the carrier wave and the signal of telecommunication wireless or that transmit by wired connection.
In this manual, term " software " is intended to comprise and perhaps is kept at the application in the magnetic storage by the firmware that resides in the read-only memory that they can be read in the memory, so that by processor processing.In addition, in certain embodiments, a plurality of software inventions can be realized as the more subdivision of large program, remain different software inventions simultaneously.In certain embodiments, a plurality of software inventions also can be realized as independently program.At last, together realize software described herein invention separable programming any combination also within the scope of the invention.In certain embodiments, when being installed to when operating on one or more electronic systems, the software program definition is carried out and is realized that one or more particular machines of the operation of software program realize.
In the caller code environment mutual by one or more interfaces and invoked other program code, some embodiment are realized as the software process that comprises one or more API (API).Various function calls, message or other various calling that further can comprise various parameters can be passed through API, at caller be called and transmit between the code.In addition, API can provide use to define in API and the data type that realizes in invoked program code or the ability of classification to the caller code.
At least some embodiment comprise and call component software by API and the mutual environment of invoked component software.A kind of method by the API operation in this environment comprises by API, transmits one or more function calls, message, other various calling or parameter.
In certain embodiments, can use one or more API (API).For example, some embodiment of exchange of media module 310 (perhaps 910) provide one group of API to other component software, are used for visiting at Fig. 3 and 9 various Video processing and the encoding functions of describing, such as the function of the TNR module of describing in Figure 19 1900.
API is the interface of being realized by program code assembly or nextport hardware component NextPort (below be called " API realizes assembly "), allows different program code assemblies or nextport hardware component NextPort (below be called " API Calls assembly ") visit and one or more functions, method, process, data structure, classification and/or other service that is provided by API realization assembly is provided.AIP can be defined in the one or more parameters that transmit between API Calls assembly and the AIP realization assembly.
API allows developer's (can be third party developer) of API Calls assembly to utilize by API and realizes the regulation feature that assembly provides.Can there be an API Calls assembly, perhaps can has API Calls assembly more than one.API can be the source code interface that computer system or program library provide for the service request of supporting self-application.Operating system (OS) can have a plurality of API, and with the one or more such API of application call that allow to move on OS, service (such as program library) can have a plurality of API, to allow to use the one or more such API of application call of this service.Can utilize when setting up application, the programming language that can be explained or compile is specified API.
In certain embodiments, API realizes that assembly can provide the API more than, and each API provides the different views of visit by the different situations of the different aspect of the function of API realization assembly realization, the different views that perhaps has described different situations.For example, API realizes that an API of assembly can provide first group of function, and can be exposed to third party developer, API realizes that another API of assembly can be hidden (not being exposed), and provide the subclass of first group of function, also provide another group function, such as test or the debug function in first group of function not.In other embodiments, API realizes that assembly itself can pass through one or more other assemblies of API Calls of lower floor, thereby is the API Calls assembly, is again that API realizes assembly.
API defines when visiting and utilize API to realize the specific characteristic of assembly, language and parameter that the API Calls assembly uses.For example, one or more API Calls that the API Calls assembly exposes by API or enable (for example specializing) with function or method call, visit API realizes the specific characteristic of assembly, and via API Calls or enable, utilizes parameter to transmit data and control information.API realizes that assembly can respond the API Calls from the API Calls assembly, returns numerical value by API.Although the grammer of API definition API Calls and result (for example, how to enable API Calls and what API Calls does) call the function of how finishing by the API Calls appointment but API can not represent AIP.Each API Calls is to realize that by called side (API Calls assembly) and API the one or more API between the assembly transmit.Transmitting API Calls can comprise and send, just open, enable, call, receive, return or response function calls or message; In other words, transmission can be described the action of API Calls assembly or API realization assembly.The function call of API or other are enabled and can be passed through parameter list or other structure, send or receive one or more parameters.Parameter can be the pointer of constant, key (key), data structure, object, object type, variable, data type, pointer, array, tabulation or function or method or quote data or other modes of another purpose that will transmit by API.
In addition, data type or classification can be provided by API, and realize the assembly realization by API.Thereby the definition that provides in API can be provided the API Calls assembly, the constant of explanatory variable, the pointer that uses this type or classification, use or this type of illustration or classification.
Usually, API can be used to visit by API and realize service or the data that assembly provides, and perhaps initiates to realize the operation that assembly provides or the execution of calculating by API.For example, API realizes that assembly and API Calls assembly can be operating system, program library, device driver, API, application program or other module one of any (should understand that API realizes that assembly and API Calls assembly can be the modules of same type, perhaps the module that differs from one another of type).In some cases, can come imbody API to realize assembly with firmware, microcode or other hardware logic to small part.In certain embodiments, API can allow the service that client-side program uses to be provided by software development kit (SDK) storehouse.In other embodiments, the API that is provided by application architecture can be provided for application or other client-side program.The data type or the object that perhaps use in SDK definition and provided by API in these embodiments, are provided or client-side program can comprise the calling of the function that provided by SDK and provided by API or method.In these embodiments, application architecture can be the part circulation of taking charge that program provides the variety of event of response by described architecture definition.API allows the applications exploiting application architecture, allocate event and to the response of incident.In some implementations, API Calls can be to the ability or the state of application report hardware device, comprise ability relevant or state with various aspects, such as input capability and state, fan-out capability and state, disposal ability, power supply status, memory capacity and state, communication capacity or the like, and other the rudimentary logic realization that can partly move on nextport hardware component NextPort with firmware, microcode or part of API.
The API Calls assembly can be local component (that is, realizing that with API assembly is on identical data handling system), or realizes the remote component (that is, realizing that with API assembly is on different data handling systems) of component communication through API and API by network.Should understand that API realizes that assembly also can serve as API Calls assembly (that is, it can carry out the API accent to realized the AIP that assembly exposes by different API), by realizing being exposed to the API of different API Calls assemblies, the API Calls assembly also can serve as API and realize assembly.
API also allows a plurality of API Calls assemblies and the API that write with different programming languages to realize component communication (thereby API can be included in the feature that API realizes that conversion is called and replied between assembly and the API Calls assembly); But, can utilize specific programming language to realize API.In one embodiment, the API Calls assembly is adjustable uses API from different providers, such as from one group of API of OS provider, from another group API of plug-in unit provider with from another group API of the founder of another provider (for example, the provider of software library) or another group API.
Figure 88 is the block diagram that graphic extension can be used example API architecture in some embodiments of the invention.As shown in Figure 88, API architecture 8800 comprises that the API that realizes API 8820 realizes assembly 8810 (for example, operating system, program library, device driver, API, application program, software or other module).API 8820 specifies the further feature that can be realized assembly by one or more functions, method, classification, object, agreement, data structure, form and/or the API that API Calls assembly 8830 uses.API 8820 can stipulate at least a calling convention, and described calling convention regulation API realizes how parameter and this function that how the function in the assembly 8810 receives from API Calls assembly 8830 return to the API Calls assembly to the result.API Calls assembly 8830 (for example, operating system, program library, device driver, API, application, software or other module) sends API Calls by API 8820, is realized the feature of assembly 8810 by the API of API 8820 appointments with visit and utilization.API realizes that assembly 8810 can respond API Calls, returns numerical value by API 8820 to API Calls assembly 830.
Recognize that API realizes that assembly 8810 can comprise and specifies by API 8820 and can not be for other function, method, classification, data structure and/or the further feature of API Calls assembly 8830 uses.Should understand that API Calls assembly 8830 can realize that assembly 8810 in identical system, perhaps can be positioned at remote location with API, and realize assemblies 8810 by network utilisation API 8820 visit API.Although the single API Calls assembly 8830 that Figure 88 graphic extension and API 8820 are mutual, but should understand that other API Calls assembly can use API8820, described other API Calls assembly can use the language (perhaps use identical language) different with API Calls assembly 8830 to write.
API realizes that assembly 8810, API 8820 and API Calls assembly 8830 can be stored in the machine readable media, and described machine readable media comprises any mechanism of the information of preserving with the readable form of machine (for example computer or other data handling system).For example, machine readable media comprises disk, CD, random access memory, read-only memory, flash memory device or the like.
Figure 89 is an example of two camera movement computing equipment architectures 8900.The realization of mobile computing device can comprise one or more processing units 8905, memory interface 8910 and Peripheral Interface 8915.These assemblies that constitute the computing equipment architecture can be independent assemblies, perhaps are integrated in one or more integrated circuits.Also available one or more communication buss of these each assemblies or holding wire are coupled in together.
Peripheral Interface 8915 can with various transducers and subsystem, comprise that camera subsystem 8920, radio communication subsystem 8925, audio subsystem 8930, I/O subsystem 8935 etc. couple.Peripheral Interface 8915 can be realized the communication between processor and the ancillary equipment.Ancillary equipment such as orientation sensor 8945 or acceleration transducer 8950 can couple with Peripheral Interface 8915, with convenient directed and acceleration function.
Camera subsystem 8920 can with one or more optical pickocffs 8940, for example charge-coupled device (CCD) optical pickocff, complementary metal oxide semiconductors (CMOS) (CMOS) optical pickocff couple.The camera subsystem 8920 that couples with transducer can convenient camera-enabled, such as image and/or video data capture.Radio communication subsystem 8925 can be used for convenient communication function.Radio communication subsystem 8925 can comprise radio frequency receiver and reflector and optical receiver and reflector.They can be realized as by one or more communication networks, such as work such as GSM network, Wi-Fi network, blueteeth networks.Audio subsystem 8930 couples with loud speaker and microphone, with the function of facility permission speech, such as speech recognition, digital record or the like.
I/O subsystem 8935 relates between the data/address bus of I/O ancillary equipment such as display, touch-screen and CPU the transmission by Peripheral Interface.I/O subsystem 8935 can comprise touch screen controller 8955 and other input controller 8960, with convenient these functions.Touch screen controller 8955 can couple with touch-screen 8965, and utilizes multiple in the quick technology any one of touching, and detects the contact on the screen and moves.Other input controller 8960 can with other input-output apparatus, couple such as one or more buttons.
Memory interface 8910 can couple with memory 8970, and memory 8970 can comprise high-speed random access memory and/or nonvolatile memory, such as flash memory.Memory can be preserved operating system (OS) 8972.OS 8972 can comprise the instruction of handling basic system services and carrying out hardware dependent tasks.
Memory also can comprise to be made and the easier communication instruction 8974 of communicating by letter of one or more other equipment; The graphic user interface instruction 8976 that convenient graphic user interface is handled; Facility relates to the processing of image/video and the image/video processing instruction 8978 of function; Facility relates to the processing of phone and the telephone order 8980 of function; The convenient processing relevant with processing and the exchange of media and the processing instruction 8982 of function with media communication; Facility relates to the processing of camera and the camera instruction 8984 of function; Video conference instruction 8986 with convenient video conference processing and function.Above-mentioned instruction does not need to be realized as independently software program or module.Available hardware and/or software comprise the various functions that realize mobile computing device with one or more signal processing and/or application-specific integrated circuit (ASIC).
The foregoing description can comprise receiving and touch input, passing through wired or wireless communication passage 9002 as Figure 90 as shown in, with the mutual touch I/O equipment 9001 of computing system 9003.Touch I/O equipment 9001 can be used to replace or in conjunction with other input equipment, such as keyboard, mouse etc., user's input be offered computing system 9003.One or more touch I/O equipment 9001 can be used for user's input is offered computing system 9003.Touching I/O equipment 9001 can be the part (for example, the touch-screen of laptop computer) of computing system 9003, perhaps is independent of computing system 9003.
Touch I/O equipment 9001 and can comprise transparent wholly or in part, translucent, opaque touch sensitive panel, perhaps their combination in any.Touch that I/O equipment 9001 can be embodied as touch-screen, touch pad, the touch-screen that plays the touch pad effect (for example, the touch-screen that replaces the touch pad of laptop computer), combine with any other input equipment or the touch-screen that merges or touch pad (for example, place on the keyboard touch-screen or touch pad) or have and be used to receive any multidimensional object that touches sensitive surfaces that touch is imported.
In an example, the touch I/O equipment 9001 that is embodied as touch-screen can comprise the transparent and/or translucent touch sensitive panel on the display that partly or entirely places at least a portion.According to this embodiment, touch I/O equipment 9001 and be used for showing the graph data that transmits from computing system 9003 (and/or another source), and be used to receive user's input.In other embodiments, touch I/O equipment 9001 can be embodied as wherein to make and touch the integrated touch screen that quick assembly/equipment combines with display module/equipment.In other embodiment, touch-screen can be used as and show the assistant images data, and perhaps identical with basic display unit graph data and reception touch the auxiliary or additional display of input.
Touching I/O equipment 9001 can be configured to according to electric capacity, resistance, optics, acoustics, induction, machinery or chemical measurement result, perhaps the one or many about nearby device 9001 touches or the approaching measurable any phenomenon of generation that touches, and the one or many on the checkout equipment 9001 touches or the approaching position that touches.Software, hardware, firmware or their combination in any can be used to handle the measurement result of the touch that is detected, to discern and to follow the tracks of one or more gestures.Gesture can touch or approaching the touch corresponding to the fixing or revocable one or many that touches on the I/O equipment 9001.By substantially side by side, incessantly or one after the other according to predetermined way,, pressure directed such as dubbing, push, swing, wipe, twist, changing is pushed etc. with changing, touching mobile one or more finger or other object on the I/O equipment 9001, can make gesture.Gesture can be with between the finger, perhaps the pinching of any other one or more finger, slide, hit, rotate, crooked, drag or dub action and characterize (but being not limited thereto).Single gesture can be with a hand or two hands, realized by one or more users or their combination in any.
Computing system 9003 can be used the graph data driving display, with display graphics user interface (GUI).GUI can be configured to receive the touch input by touching I/O equipment 9001.The touch I/O equipment 9001 that is embodied as touch-screen can show GUI.On the other hand, GUI can be displayed on touch the display that I/O equipment 9001 separates on.GUI can comprise the graphic elements that is presented at the ad-hoc location in the interface.Graphic elements can include but not limited to the virtual input device of various demonstrations, comprises virtual scroll wheel, dummy keyboard, virtual knob, virtual push button, any virtual UI or the like.The user can do gesture at the touch I/O equipment one or more ad-hoc locations related with graphic elements GUI 9001.In other embodiments, the user can do gesture with one or more positions of the location independent of the graphic elements of GUI.The gesture of making on touching I/O equipment 9001 can be handled directly or indirectly, control, revises, moves, encourages, starts or usually influence graphic elements in the GUI, such as cursor, icon, media file, tabulation, text, all or part of image or the like.For example, with regard to touch-screen, the user can be by doing gesture on the graphic elements on the touch-screen, directly and graphic elements mutual.On the other hand, touch pad generally provides indirect mutual.Gesture also influences the GUI element (for example, causing user interface to occur) that does not show, perhaps can influence other action (for example, influencing the state or the pattern of GUI, application or operating system) in the computing system 9003.Can on touch I/O equipment 9001, do or not do gesture in conjunction with the cursor that shows.For example, just do the situation of gesture on touch pad, cursor (perhaps designator) can be displayed on display screen or the touch-screen, by the touch on touch pad input controllable cursor, with display screen on Drawing Object mutual.Directly be on touch-screen therein among other embodiment of gesture, the user can show or not be presented under the situation on the touch-screen at cursor or designator, directly and the object interaction on the touch-screen.
Response or according to touch on the I/O equipment 9001 touch or near touching, can provide feedback to the user by communication port 9002.Feedback can be changeably or immutablely with modes such as optics, machinery, electric, sense of smell, acoustics, perhaps their combination in any transmits.
Above-mentioned these function available digital electronic circuits, realize with computer software, firmware or hardware.Various technology can utilize one or more computer programs to realize.Programmable processor and computer can be included in or be packaged into mobile device.Processing and logic flow can be carried out by one or more programmable processors with by one or more Programmable Logic Device.General and dedicated computing equipment and memory device can pass through interconnection of telecommunication network.
Some embodiment comprise computer program instructions are kept at electronic building brick in machine readable or the computer-readable medium (calling computer-readable recording medium, machine readable media or machinable medium on the other hand), such as microprocessor, memory and memory storage.The example of this computer-readable medium (for example comprises RAM, ROM, read-only optical disc (CD-ROM), CD-R (CD-R), CD-RW (CD-RW), read-only digital versatile disc, DVD-ROM, but DVD-dual layer-ROM), various record/rewritable DVD (for example, DVD-RAM, DVD-RW, DVD+RW etc.), flash memory (for example SD card, small-sized SD card, miniature SD card etc.), magnetic and/or solid-state hard drive, read-only and can write down blue light
Figure BSA00000402183701661
CD, super-high density CD, other optics or magnetic medium and floppy disk arbitrarily.Computer-readable medium can be preserved and can comprise many groups of instructions that are used to realize various operations by the computer program of at least one processing unit execution.The example of computer program or computer code comprises machine code, such as the machine code that is produced by compiler with comprise by computer, electronic building brick or utilize the file of the high-level code that the microprocessor of interpreter carries out.
Although top discussion relates generally to the microprocessor or the multinuclear die processor of executive software, but some embodiment are carried out such as application-specific integrated circuit (ASIC) (ASIC) or field programmable gate array (FPGA) by one or more integrated circuits.In certain embodiments, such integrated circuit is carried out and is kept at originally on one's body instruction of circuit.
The term " computer " that uses in any claim of specification and the application, " server ", " processor " and " memory " all refer to electronic equipment or other technical equipment.People or crowd got rid of in these terms.Concerning specification, term " demonstration " meaning shows on electronic equipment.The term " computer-readable medium " that uses in any claim of specification and the application is confined to the tangible entity of the information of preserving with computer-readable form fully.Any wireless signal, wired download signal and any other instant signal got rid of in these terms.
Figure 91 graphic extension conceptually is used to connect some participants' of video conference example communication system 9100 according to some embodiment.As shown in the figure, communication system 9100 comprises some mobile devices 9115, some cellular basestations (Node B in other words) 9110, some radio network controllers (RNC) 9105 and core network 9125.Cellular basestation and RNC are called Universal Mobile Telecommunications System (UMTS) grounding wireless access network network (UTRAN) 9130 by collective.Each RNC9105 connects with the one or more cellular basestations 9110 that are called as Radio Access Network (RAN).
Each cellular basestation 9110 covers a service area 9120.As shown in the figure, the mobile device 9115 in each service area is by the Uu interface, with service cellular basestation 9110 wireless connections of service area 9120.The Uu interface uses the dihedral protocol stack of tool: control plane and user plane.User plane support circuit-switched, packet switching and broadcast data stream.Control plane carries the signaling message of network.
Each cellular basestation is connected with RNC by Iub interface.Each RNC 9105 is connected with core network 9125 with the Iu-ps interface by Iu-cs.The Iu-cs interface is used for circuit switched service (for example, speech), and the Iu-ps interface is used for packet-switched services (for example, data).The Iur interface is used for two RNC are linked together.
Therefore, communication system 9100 support circuit-switched service and packet-switched services.For example, circuit switched service allows to transmit communicating data (for example, speech) by the circuit switching equipment via communication system 9100, converses.Packet-switched services allows by utilizing the transportation protocol layer (such as UDP or TCP) on Internet protocol layer (such as IP), and the PSE transmission videoconference data via communication system 9100 carries out video conference.In certain embodiments, the front is provided with circuit switched service and the packet-switched services that the conversation-video conference that illustrates in the chapters and sections changes the communication system support of (for example, switching) utilization such as communication system 9100 in video conference.That is, in such embodiments, conversation is that the circuit switching equipment by communication system 9100 carries out, and video conference is that the PSE by communication system 9100 carries out.
Although the example communication system among Figure 91 illustrates the third generation (3G) technology UTRAN mobile radio system, but should note in certain embodiments, the second generation (2G) communication system, other 3G communication system, such as 3GPP2 Evolution-Data Optimized or evolution-just data (EV-DO) and third generation partner program 2 (3GPP2) code division multiple access access 1X (CDMA1X), the 4th generation (4G) communication system, wireless lan (wlan) can be used to be connected some participants of meeting with micro-wave access global inter communication (WiMAX) communication system.The example of 2G system comprises global system for mobile communications (GSM), general packet radio service (GPRS) and enhancing data rate GSM evolution (EDGE).The 2G communication system architecture is similar to the architecture shown in Figure 91, replace Node B 9110 except the 2G communication system architecture uses base station transceiver (BTS), and base station controller (BSC) replaces outside the RNC 9105.In the 2G communication system, the A interface between BSC and the core network is used for circuit switched service, and the Gb Interface between BSC and the core network is used for packet-switched services.
In certain embodiments, communication system 9100 is utilized the service provider operation of communication system 9100 by initial supply mobile device 9115 to allow mobile device 9115.Some embodiment provide mobile device 9115 by configuration and registered user's identification module (SIM) in mobile device 9115.In other embodiments, change memory configurations and the registration mobile device 9115 that utilizes mobile device 9115 into.In addition, can provide other service (after the client buys mobile device 9115), such as the data, services of similar GPRS, Multimedia Message sending and receiving services (MMS) and instant messaging.In case be supplied, mobile device 9115 activates with regard to serviced operator, thereby allows to use communication system 9100.
In certain embodiments, communication system 9100 is dedicated communications networks.In such embodiments, mobile device 9115 can be in (for example, the mobile device 9115 that provides for communication system 9100) communication (for example, carrying out voice call, swap data) each other.In other embodiments, communication system 9100 is public communication networks.Thereby except the mobile device 9115 that provides for communication system 9110, mobile device 9115 can also be communicated by letter with the miscellaneous equipment outside communication system 9100.In the miscellaneous equipment outside communication system 9100 some comprise by other network, such as public switch telephone network or another kind of cordless communication network, and the telephone set, computer and the miscellaneous equipment that are connected with communication system 9100.
Long Term Evolution (LTE) standard is used to define the 4G communication system.Figure 92 graphic extension conceptually is used to connect some participants' of video conference the example of 4G communication system 9200 in certain embodiments.As shown in the figure, communication system 9200 comprises some mobile devices 9115, some evolved Node B (eNB) 9205, mobile management entity (MME) 9215, gateway (S-GW) 9220, packet data network (PDN) gateway 9225 and home subscriber server (HSS) 9235.In certain embodiments, communication system 9200 comprises one or more MME 9215, one or more S-GW 9220, one or more PDN Gateway 9225 and one or more HSS 9235.
ENB 9205 provides air interface for mobile device 9115.As shown in the figure, each eNB9205 covers service area 9210.Mobile device equipment 9115 in each service area 9210 is by the LTE-Uu interface, with eNB 9205 wireless connections of service area 9210.Figure 92 represents that also eNB 9205 interconnects by X2 interface.In addition, eNB 9205 is connected with MME 9215 by S1-MME interface, is connected with S-GW 9220 by S1-U interface.ENB 9205 collectives are called as the UTRAN (E-TRAN) 9230 of evolution.
ENB 9205 provides various functions, such as provided for radio resources management (for example, radio bearer control, the control of connection mobility or the like), user plane data is towards the route of S-GW 9220, signal measurement and measurement report, the MME when mobile device connects selects, or the like.The function of MME9215 comprises idle pulley mobile device tracking and paging, the enabling and stopping using of radio bearer, the selection of the S-GW 9220 when mobile device connects, the termination of non-access aspect (NAS) signaling, by with the mutual subscription authentication of HSS 9235, or the like.
The function of S-GW 9220 comprises (1) route and transmits user data packets and (2) management and preservation mobile device context, such as the parameter and the network internal routing iinformation of IP carrying service.The function of PDN Gateway 9225 comprises by the exit point of the traffic that becomes mobile device and entrance, and the connectedness from mobile device to external packet data network (not shown) is provided.Connectedness when mobile radio station can have more than a PDN Gateway is so that insert a plurality of packet data network.PDN Gateway 9225 also serves as 3GPP and non-3GPP technology, such as the ambulant anchor point (anchor) between the WiMAXt 3GPP2 (for example, CDMA 1X and EV-DO).
As shown in the figure, MME 9215 is connected with S-GW 9220 by the S11 interface, is connected with HSS 9235 by the S6a interface.S-GW 9220 is connected by the S8 interface with PDN Gateway 9220.MME 9215, S-GW 9220 and PDN Gateway 9225 collectives are called as block core evolution (EPC).EPC is the primary clustering of system architecture evolution (SAE) architecture, and system architecture evolution (SAE) architecture is the Core Network Architecture of 3GPP LTE wireless communication standard.EPC is pure grouping system.For example, EPC does not have the speech media gateway.Service such as speech and SMS is the packet switching route, and is provided by the application function that utilizes the EPC service.So utilize the conversation-video conference that illustrates previously to change as an example, in certain embodiments, conversation and video conference all are that the PSE by communication system 9200 carries out.In some such embodiment, after end of conversation, the packet switched channels that is used to converse continues to be used to the voice data of video conference.But, in other this embodiment, create different packet switched channels (for example, when setting up video conference), the packet switched channels by new establishment rather than when end of conversation utilizes the packet switched channels of conversation to transmit voice data.
In addition, the quantity of the bandwidth that provides of these different technologies surpasses 10 mbit/(Mbps) from 44 kilobits/second (kbps) of GPRS to LTE.Following the expectation downloading rate is 100Mbps concerning LTE, and uploading rate is 50Mbps.
Although about numerous details the present invention has been described, but those of ordinary skill in the art will appreciate that imbody the present invention in other specific forms, and does not break away from spirit of the present invention.In addition, numerous accompanying drawings illustrate various processing conceptually.Can carry out the concrete operations of these processing not according to the strict order of described expression and description.In continuous sequence of operations, specific operation can be do not carried out, and different specific operations can be in different embodiment, carried out.In addition, processing can utilize several height to handle and realize, perhaps is realized as the part of bigger grand processing.
In addition, about the video conference between two camera movement equipment many embodiment have been described above.But, those of ordinary skill in the art will appreciate that in these embodiments many embodiment can be used on and relates to two camera movement equipment and another equipment, such as single camera movement equipment, computer, have in the situation of the video conference between the telephone set etc. of video conference capabilities.In addition, at single camera movement equipment and having in other computing equipment of video conference capabilities, can use many embodiment described above.Thereby those of ordinary skill in the art understands the present invention and not limited by top example details, but is limited by additional claim.

Claims (20)

1. mobile device comprises:
Present the synthetic display screen that shows on first equipment, described synthetic demonstration has first video of being taken by first equipment and second video of being taken by participation and second equipment of the video conference of first equipment;
Communication interface is used for during first equipment is received in video conference, revises the synthetic input that shows; With
Image processor is used for revising synthetic the demonstration according to the input that receives.
2. according to described first mobile device of claim 1,
Wherein display screen is to touch quick display screen,
Wherein synthetic the demonstration is that picture-in-picture (PIP) shows, described PIP shows that the insertion with performance first video shows and shows the main of second video and show, described insertion shows less than described main the demonstration, and overlaps on main demonstration of at least a portion,
Wherein receiving described input comprises (i) and makes the position that insert to show and touch the position that just is being touched on the quick display screen and be associated, (ii), select to insert demonstration, contacting the maintained while with (iii) determining in response to described association, touch contact position on the quick display screen mobile and
Wherein revise synthetic demonstration and comprise, move and insert demonstration along the moving direction of contact.
3. according to described first mobile device of claim 2, comprise with after contacting of screen is terminated even wherein move to insert show, insert the latched position that shows in the synthetic demonstration of also no show simultaneously, continue to move insert show that described latched position has been designated as the position that is used to place the end that described insertion shows in synthetic the demonstration.
4. according to described first mobile device of claim 2, wherein move the insertion demonstration and comprise:
If moving of contact position surpasses specified quantitative,, continue to move to insert and show even so with after contacting of screen is terminated; And
If moving of contact position do not surpass specified quantitative, after contact has been terminated, does not move insertion and show so.
5. according to described first mobile device of claim 1, wherein insert first corner that shows from the PIP demonstration and be moved to second corner that PIP shows.
6. according to described first mobile device of claim 1,
Wherein when the rotation of the display screen of first equipment, produce described input and
Wherein modification is synthetic shows that comprising rotation synthesizes demonstration.
7. according to described first mobile device of claim 6, wherein the display screen of first equipment can irrespectively rotate with the fuselage of first equipment.
8. according to described first mobile device of claim 6, wherein when first equipment rotated, the display screen of first equipment rotated.
9. according to described first mobile device of claim 6, wherein rotation synthetic show to comprise present animation, described animate is synthetic to be shown around the rotation of the axle by the synthetic center that shows.
10. according to described first mobile device of claim 9, wherein said animation comprises synthetic dwindling of showing, makes during the synthetic rotation that shows, shows whole synthetic demonstration.
11. according to described first mobile device of claim 10, wherein in the ending of animation, synthetic demonstration is unfolded to be fit to the orientation of screen.
12. according to described first mobile device of claim 6, the screen of first equipment is rotated along first direction, and makes synthetic the demonstration along the second direction rotation opposite with first direction.
13. according to described first mobile device of claim 6,
Wherein synthetic the demonstration is picture-in-picture (PIP) demonstration, and described PIP shows that the insertion with performance first video shows and show the main demonstration of second video,
Wherein said insertion shows less than described main the demonstration, and overlaps on main demonstration of at least a portion,
Wherein first mobile device also comprises network interface, is used to the notice of the rotation of second equipment that receives, and becomes second orientation with the orientation that the master is shown from first directed change.
14. according to described first mobile device of claim 1,
Wherein synthetic the demonstration is picture-in-picture (PIP) demonstration, and described PIP shows that the insertion with performance first video shows and show the main demonstration of second video,
Wherein said insertion shows less than described main the demonstration, and overlaps on main demonstration of a part,
Wherein receive described input comprise reception to the selection of the part that insert to show and to described part move and
Wherein said modification comprises to change inserts the size that shows.
15. according to described first mobile device of claim 14, wherein said part mobile comprises described part towards inserting moving of the center that shows, and described modification comprises to dwindle and inserts the size that shows.
16. according to described first mobile device of claim 14, wherein said part mobile comprises described part away from inserting moving of the center that shows, and wherein said modification comprises to amplify and inserts the size that shows.
17. according to described first mobile device of claim 1,
Wherein synthetic the demonstration is picture-in-picture (PIP) demonstration, and described PIP shows that the insertion with performance first video shows and show the main demonstration of second video,
Wherein said insertion shows less than described main the demonstration, and overlaps on main demonstration of a part,
Wherein said modification comprises exchange and inserts demonstration and main the demonstration, and make and insert demonstration performance second video, and main performance first video that shows.
18. according to described first mobile device of claim 1,
Wherein receive described input comprise select first video and
Wherein said modification comprises with respect to synthetic demonstration moves first video.
19. one kind adjust first mobile device with the video conference of second mobile device in method of operating, described method comprises:
Reception is by the video of the camera of second mobile device;
The picture-in-picture of the video that shows the video taken by second equipment and taken by first equipment on first equipment shows;
Detect the variation of the screen orientation of first mobile device; With
According to the variation of the screen orientation of first equipment, the PIP that rotates on the screen of first equipment shows.
20. a mobile device comprises:
Display screen;
Reception is by the device of the video of second mobile device shooting, and wherein said video is displayed on the display screen of first mobile device;
The demonstration of the video that generation is taken by second mobile device, and another device shown of the video of taking by first mobile device;
The device of user's input that a demonstration that receives the instruction video is moved towards the direction of the display screen of first mobile device; With
According to user's input, a described demonstration of video is moved to the device of another location of the display screen of first mobile device.
CN201010602653.9A 2010-04-07 2010-09-25 In conference display adjustments Active CN102215373B (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US32187110P 2010-04-07 2010-04-07
US61/321,871 2010-04-07
US12/794,768 2010-06-06
US12/794,768 US8502856B2 (en) 2010-04-07 2010-06-06 In conference display adjustments

Publications (2)

Publication Number Publication Date
CN102215373A true CN102215373A (en) 2011-10-12
CN102215373B CN102215373B (en) 2014-12-24

Family

ID=44746473

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201010602653.9A Active CN102215373B (en) 2010-04-07 2010-09-25 In conference display adjustments

Country Status (1)

Country Link
CN (1) CN102215373B (en)

Cited By (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102625129A (en) * 2012-03-31 2012-08-01 福州一点通广告装饰有限公司 Method for realizing remote reality three-dimensional virtual imitated scene interaction
CN103118242A (en) * 2012-11-16 2013-05-22 佳都新太科技股份有限公司 Video call image rectification method
CN103702064A (en) * 2013-12-27 2014-04-02 三亚中兴软件有限责任公司 Video conference method, video conference terminal and video conference system
CN103841353A (en) * 2014-02-24 2014-06-04 广州华多网络科技有限公司 Video interaction method, terminal, server and system
CN103929747A (en) * 2013-01-16 2014-07-16 中兴通讯股份有限公司 Dimension to dimension (D2D) discovery application and completion method and corresponding device
CN104094593A (en) * 2012-02-03 2014-10-08 三星Sds株式会社 Terminal, image communication control server, and system and method for image communication using same
CN104735467A (en) * 2015-03-31 2015-06-24 北京奇艺世纪科技有限公司 Video picture-in-picture advertisement generating method and device
CN106797448A (en) * 2014-10-02 2017-05-31 英特尔公司 Interactive video meeting
CN107197356A (en) * 2017-06-06 2017-09-22 青岛海信电器股份有限公司 A kind of method, device and the TV of display screen menu on television
CN108965899A (en) * 2017-11-22 2018-12-07 北京视联动力国际信息技术有限公司 A kind of method of video image processing and device based on view networking
CN109257546A (en) * 2018-09-26 2019-01-22 北京大米未来科技有限公司 A kind of image display method, device, electronic equipment and medium
US10491861B2 (en) 2014-12-05 2019-11-26 Intel IP Corporation Interactive video conferencing
CN110505428A (en) * 2019-08-28 2019-11-26 联想(北京)有限公司 A kind of information processing method and electronic equipment
CN110502198A (en) * 2018-09-29 2019-11-26 国核自仪系统工程有限公司 Interface display system based on FPGA
CN111784615A (en) * 2016-03-25 2020-10-16 北京三星通信技术研究有限公司 Method and device for processing multimedia information
CN111971949A (en) * 2018-03-28 2020-11-20 富士胶片株式会社 Image processing system, image processing method, and information processing apparatus
CN112969083A (en) * 2021-02-20 2021-06-15 深圳市颍创科技有限公司 Method and system for simultaneously displaying main picture and sub-picture in PIP picture in vertical screen mode
CN113556462A (en) * 2020-04-23 2021-10-26 安讯士有限公司 Zoom camera for controlling pan and tilt
CN114020198A (en) * 2021-09-30 2022-02-08 荣耀终端有限公司 Display method and electronic equipment
CN114375471A (en) * 2019-09-17 2022-04-19 夏普Nec显示器解决方案株式会社 Display control device, display device, and display control method
CN115002384A (en) * 2021-12-24 2022-09-02 荣耀终端有限公司 Method for transmitting data, electronic device and readable storage medium
CN115118921A (en) * 2022-08-29 2022-09-27 全时云商务服务股份有限公司 Method and system for video screen-combining self-adaptive output in cloud conference
EP4052176A4 (en) * 2020-02-22 2023-01-04 Corephotonics Ltd. Split screen feature for macro photography
US11703668B2 (en) 2014-08-10 2023-07-18 Corephotonics Ltd. Zoom dual-aperture camera with folded lens
US11733064B1 (en) 2018-04-23 2023-08-22 Corephotonics Ltd. Optical-path folding-element with an extended two degree of freedom rotation range
US11770616B2 (en) 2015-08-13 2023-09-26 Corephotonics Ltd. Dual aperture zoom camera with video support and switching / non-switching dynamic control
US11809066B2 (en) 2017-11-23 2023-11-07 Corephotonics Ltd. Compact folded camera structure
US11832008B2 (en) 2020-07-15 2023-11-28 Corephotonics Ltd. Image sensors and sensing methods to obtain time-of-flight and phase detection information
US11856291B2 (en) 2013-08-01 2023-12-26 Corephotonics Ltd. Thin multi-aperture imaging system with auto-focus and methods for using same
US11852845B2 (en) 2013-07-04 2023-12-26 Corephotonics Ltd. Thin dual-aperture zoom digital camera
US11852790B2 (en) 2018-08-22 2023-12-26 Corephotonics Ltd. Two-state zoom folded camera
US11962901B2 (en) 2020-05-30 2024-04-16 Corephotonics Ltd. Systems and methods for obtaining a super macro image

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1473430A (en) * 2000-11-01 2004-02-04 ���˹���Ѷ��� Mixed-media telecommunication call set-up
US7148911B1 (en) * 1999-08-09 2006-12-12 Matsushita Electric Industrial Co., Ltd. Videophone device
US20070115349A1 (en) * 2005-11-03 2007-05-24 Currivan Bruce J Method and system of tracking and stabilizing an image transmitted using video telephony
US20080074550A1 (en) * 2006-09-25 2008-03-27 Samsung Electronics Co., Ltd. Mobile terminal having digital broadcast reception capability and pip display control method
US20100073455A1 (en) * 2008-09-25 2010-03-25 Hitachi, Ltd. Television receiver with a TV phone function

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7148911B1 (en) * 1999-08-09 2006-12-12 Matsushita Electric Industrial Co., Ltd. Videophone device
CN1473430A (en) * 2000-11-01 2004-02-04 ���˹���Ѷ��� Mixed-media telecommunication call set-up
US20070115349A1 (en) * 2005-11-03 2007-05-24 Currivan Bruce J Method and system of tracking and stabilizing an image transmitted using video telephony
US20080074550A1 (en) * 2006-09-25 2008-03-27 Samsung Electronics Co., Ltd. Mobile terminal having digital broadcast reception capability and pip display control method
US20100073455A1 (en) * 2008-09-25 2010-03-25 Hitachi, Ltd. Television receiver with a TV phone function

Cited By (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104094593B (en) * 2012-02-03 2019-04-19 三星Sds株式会社 Terminal, video calling control server and video call system and method using them
CN104094593A (en) * 2012-02-03 2014-10-08 三星Sds株式会社 Terminal, image communication control server, and system and method for image communication using same
CN102625129A (en) * 2012-03-31 2012-08-01 福州一点通广告装饰有限公司 Method for realizing remote reality three-dimensional virtual imitated scene interaction
CN103118242A (en) * 2012-11-16 2013-05-22 佳都新太科技股份有限公司 Video call image rectification method
CN103929747A (en) * 2013-01-16 2014-07-16 中兴通讯股份有限公司 Dimension to dimension (D2D) discovery application and completion method and corresponding device
US11852845B2 (en) 2013-07-04 2023-12-26 Corephotonics Ltd. Thin dual-aperture zoom digital camera
US11856291B2 (en) 2013-08-01 2023-12-26 Corephotonics Ltd. Thin multi-aperture imaging system with auto-focus and methods for using same
US9794513B2 (en) 2013-12-27 2017-10-17 Xi'an Zhongxing New Software Co., Ltd. Video conference method, terminal, and system
CN103702064A (en) * 2013-12-27 2014-04-02 三亚中兴软件有限责任公司 Video conference method, video conference terminal and video conference system
WO2014161404A3 (en) * 2013-12-27 2015-01-15 中兴通讯股份有限公司 Video conference method, terminal, and system
CN103841353A (en) * 2014-02-24 2014-06-04 广州华多网络科技有限公司 Video interaction method, terminal, server and system
US11703668B2 (en) 2014-08-10 2023-07-18 Corephotonics Ltd. Zoom dual-aperture camera with folded lens
CN106797448A (en) * 2014-10-02 2017-05-31 英特尔公司 Interactive video meeting
US10791261B2 (en) 2014-10-02 2020-09-29 Apple Inc. Interactive video conferencing
US10491861B2 (en) 2014-12-05 2019-11-26 Intel IP Corporation Interactive video conferencing
CN104735467A (en) * 2015-03-31 2015-06-24 北京奇艺世纪科技有限公司 Video picture-in-picture advertisement generating method and device
US11770616B2 (en) 2015-08-13 2023-09-26 Corephotonics Ltd. Dual aperture zoom camera with video support and switching / non-switching dynamic control
CN111784615A (en) * 2016-03-25 2020-10-16 北京三星通信技术研究有限公司 Method and device for processing multimedia information
CN107197356A (en) * 2017-06-06 2017-09-22 青岛海信电器股份有限公司 A kind of method, device and the TV of display screen menu on television
CN107197356B (en) * 2017-06-06 2020-08-14 海信视像科技股份有限公司 Method and device for displaying screen menu in television and television
CN108965899A (en) * 2017-11-22 2018-12-07 北京视联动力国际信息技术有限公司 A kind of method of video image processing and device based on view networking
US11809066B2 (en) 2017-11-23 2023-11-07 Corephotonics Ltd. Compact folded camera structure
CN111971949B (en) * 2018-03-28 2021-07-13 富士胶片株式会社 Image processing system, image processing method, and information processing apparatus
CN111971949A (en) * 2018-03-28 2020-11-20 富士胶片株式会社 Image processing system, image processing method, and information processing apparatus
US11733064B1 (en) 2018-04-23 2023-08-22 Corephotonics Ltd. Optical-path folding-element with an extended two degree of freedom rotation range
US11867535B2 (en) 2018-04-23 2024-01-09 Corephotonics Ltd. Optical-path folding-element with an extended two degree of freedom rotation range
US11852790B2 (en) 2018-08-22 2023-12-26 Corephotonics Ltd. Two-state zoom folded camera
CN109257546A (en) * 2018-09-26 2019-01-22 北京大米未来科技有限公司 A kind of image display method, device, electronic equipment and medium
CN110502198B (en) * 2018-09-29 2023-06-16 国核自仪系统工程有限公司 Interface display system based on FPGA
CN110502198A (en) * 2018-09-29 2019-11-26 国核自仪系统工程有限公司 Interface display system based on FPGA
US11303825B2 (en) 2019-08-28 2022-04-12 Lenovo (Beijing) Co., Ltd. Information processing method and electronic apparatus
CN110505428A (en) * 2019-08-28 2019-11-26 联想(北京)有限公司 A kind of information processing method and electronic equipment
CN114375471A (en) * 2019-09-17 2022-04-19 夏普Nec显示器解决方案株式会社 Display control device, display device, and display control method
EP4052176A4 (en) * 2020-02-22 2023-01-04 Corephotonics Ltd. Split screen feature for macro photography
CN113556462A (en) * 2020-04-23 2021-10-26 安讯士有限公司 Zoom camera for controlling pan and tilt
US11962901B2 (en) 2020-05-30 2024-04-16 Corephotonics Ltd. Systems and methods for obtaining a super macro image
US11832008B2 (en) 2020-07-15 2023-11-28 Corephotonics Ltd. Image sensors and sensing methods to obtain time-of-flight and phase detection information
CN112969083A (en) * 2021-02-20 2021-06-15 深圳市颍创科技有限公司 Method and system for simultaneously displaying main picture and sub-picture in PIP picture in vertical screen mode
CN114020198B (en) * 2021-09-30 2023-01-17 荣耀终端有限公司 Display method and electronic equipment
CN114020198A (en) * 2021-09-30 2022-02-08 荣耀终端有限公司 Display method and electronic equipment
CN115002384B (en) * 2021-12-24 2023-01-31 荣耀终端有限公司 Method for transmitting data, electronic device and readable storage medium
CN115002384A (en) * 2021-12-24 2022-09-02 荣耀终端有限公司 Method for transmitting data, electronic device and readable storage medium
CN115118921B (en) * 2022-08-29 2023-01-20 全时云商务服务股份有限公司 Method and system for video screen-combining self-adaptive output in cloud conference
CN115118921A (en) * 2022-08-29 2022-09-27 全时云商务服务股份有限公司 Method and system for video screen-combining self-adaptive output in cloud conference

Also Published As

Publication number Publication date
CN102215373B (en) 2014-12-24

Similar Documents

Publication Publication Date Title
CN102215217B (en) Establishing a video conference during a phone call
CN102215373B (en) In conference display adjustments
JP6949917B2 (en) Establishing a video conference during a call
CN102215372B (en) Remote control operations in a video conference
CN102215374B (en) Camera is switched during the video conference of multi-camera mobile device
KR102660942B1 (en) Establishing a video conference during a phone call

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant