JP2020017995A - Establishment of video conference during call - Google Patents

Abstract

To provide a camera image switching process of a mobile device having a plurality of cameras used for a video conference.SOLUTION: A first mobile device transmits an image captured by a camera of a first mobile device to a second device in a video conference, and receives and displays an image captured by a camera of the second device. The first mobile device selects a graphical user interface option indicating a camera switching request, and the first mobile device transmits a command to instruct to the second device to perform a camera switching operation that adjusts a display of the image captured by the camera of the second device.SELECTED DRAWING: Figure 50

Description

  Many of today's mobile devices, such as smartphones, provide video capture capabilities. The user of the mobile device can capture both still images and video through the phone's camera. However, to transmit the captured video to another party, the user typically sends the video directly to another party after the video capture is complete, or sends the video to another location (eg, Internet video hosting site). Unfortunately, this does not allow other parties to view the live video stream while capturing video with the mobile device.

  Furthermore, a standard portable device is equipped with only one camera, and it is quite difficult to process information from this one camera. An ideal device would have multiple cameras and be able to transmit live video, which is a composite of video from at least two cameras. This means that the device handles multiple captured video streams, given the limited resources available to the mobile device, and that the network to which the device is connected handles the transmission of live video streams From both perspectives, it is a particularly difficult problem.

  Some embodiments of the present invention provide a portable device having two cameras capable of taking pictures and videos. The mobile device of some embodiments has a display screen for displaying captured photographic and video images. The portable device also includes storage for storing the captured image for later transmission to another device. The device further has a network interface that can transmit the captured images to one or more devices during a real-time communication session between the users of the device. The device also includes an encoder that can be used to store the captured image locally or to encode it for transmission to another device. The mobile device further includes a decoder that allows the device to decode an image captured by another device during a real-time communication session or to decode a locally stored image.

  An example of a real-time communication session involving the transmission of a captured video image is a video conference. In some embodiments, the portable device can transmit only the captured video images of one camera at any given time during a video conference. However, in other embodiments, the portable device can simultaneously transmit captured video images from both of its cameras during a video conference or other real-time communication session.

  During a videoconference with another device, the portable device of some embodiments may transmit other types of content along with video captured by one or both of its cameras. An example of such other content includes a low-resolution or high-resolution photographic image captured by one of the device's cameras, while another camera of the device captures video used in a video conference. Other examples of such other content include (1) files and other content stored on the device, (2) device screen display (ie, content displayed on the device screen), and (3) television. Such as content received from another device during a conference or other real-time communication session.

  The mobile device of some embodiments employs a new in-meeting coordination technique for making adjustments during a video conference. For example, the portable device of some embodiments dynamically switches to transmit video captured by one camera while transmitting only video captured by one camera during a video conference. Can be. In such a situation, the mobile device of some embodiments may have a video conference call so that the other device can achieve a smooth transition between the video captured by the two cameras on the device side. Notify any other participating devices of this switch.

  In some embodiments, the camera switch request may originate from a “local” device that switches its camera during a video conference, as well as other “remote” devices receiving video captured by the local device. Can also be derived from Further, the ability of one device to instruct another device to switch cameras is only one example of the remote control capabilities of the device in some embodiments. Examples of other operations that can remotely instruct the device in some embodiments include an exposure adjustment operation (eg, auto exposure), a focus adjustment operation (eg, auto focus), and the like. Another example of a new in-meeting adjustment that can be specified locally or remotely is the specification of a region of interest (ROI) in the captured video, where the purpose of the ROI identification is to change the behavior of the capturing camera. To change the image processing operation of the device having the capturing camera, or to change the encoding operation of the device having the capturing camera.

  Yet another example of a new in-meeting adjustment of some embodiments includes real-time modification of a composite video display generated by a device. Specifically, in some embodiments, the portable device generates a composite display that simultaneously displays multiple videos captured by multiple cameras of one or more devices. In some cases, the composite display places the video in an adjacent display area (eg, an adjacent window). In other cases, the composite display is a picture-in-picture (PIP) display that includes at least two display areas showing two different videos, one of the display areas being a background main display area and the other being a background main display area. This is the foreground insertion display area overlapping with.

  In some embodiments, the real-time modification of the composite video display includes moving one or more of the display areas within the composite display in response to user selection and movement of the display area. Also, some embodiments rotate the composite display during a video conference if the screen of the device that provided the composite display rotates. Also, the portable device of some embodiments allows the user of the device to swap the video in the PIP display (ie, display the video in the background main display in the foreground insert display and replace the video in the foreground insert display with the background main display). Display on the display) is possible.

  The above summary of the invention is intended to serve as a brief description of some embodiments of the invention. This does not imply a description or general description of the subject matter of any invention disclosed herein. In the following embodiments for carrying out the invention and the drawings referred to in the embodiments for carrying out the invention, not only the embodiment described in the summary of the invention but also other embodiments will be further described. Therefore, in order to understand all the embodiments described by this specification, a thorough examination of the summary of the invention, the modes for carrying out the invention, and the drawings is required.

  The appended claims set forth the novel features of the invention. However, for illustrative purposes, some embodiments of the present invention are shown in the following figures.

FIG. 14 is a diagram illustrating a composite display according to some embodiments. FIG. 10 is a diagram illustrating another composite display of some embodiments. FIG. 4 conceptually illustrates a software architecture for a video processing and encoding module of a dual camera portable device of some embodiments. FIG. 7 is a diagram conceptually illustrating a captured image processing unit according to some embodiments. FIG. 5 is a diagram conceptually illustrating an example of different frame rates based on different vertical blanking periods (VBI). FIG. 3 is a diagram conceptually illustrating an example of different interleave frame rates based on different VBIs. FIG. 9 is a diagram conceptually illustrating another captured image processing unit according to some embodiments. FIG. 9 is a diagram conceptually illustrating another captured image processing unit according to some embodiments. FIG. 4 conceptually illustrates a software architecture for video conferencing and processing modules of a dual camera portable device of some embodiments. FIG. 4 conceptually illustrates an example video conference request message sequence of some embodiments. FIG. 4 illustrates a user interface of some embodiments for a video conference setup operation. FIG. 4 illustrates a user interface of some embodiments for accepting an invitation to a video conference. FIG. 8 illustrates another user interface of some embodiments for accepting an invitation to a video conference. FIG. 8 illustrates another user interface of some embodiments for a video conference setup operation. FIG. 4 conceptually illustrates a process of some embodiments for setting a video conference bit rate. FIG. 6 conceptually illustrates another software architecture for the video conferencing and processing module of the dual camera portable device of some embodiments. FIG. 4 conceptually illustrates another software architecture for a dual camera portable device of some embodiments. FIG. 17 conceptually illustrates a process performed by the video conference manager of some embodiments, such as the process illustrated in FIG. 16. FIG. 4 conceptually illustrates a software architecture for the temporal noise reduction module of some embodiments. FIG. 4 conceptually illustrates a process of some embodiments for reducing noise in a video image. FIG. 10 conceptually illustrates a process performed by the image processing manager of some embodiments, such as the process illustrated in FIG. 9. FIG. 9 illustrates a user interface of some embodiments for an exposure adjustment operation. FIG. 9 illustrates a user interface of some embodiments for a focus adjustment operation. FIG. 17 conceptually illustrates a perspective correction process performed by the image processing manager of some embodiments, such as the process illustrated in FIG. 16. FIG. 7 conceptually illustrates an exemplary perspective correction operation of some embodiments. FIG. 17 conceptually illustrates a software architecture for the encoder driver of some embodiments, such as the software architecture shown in FIG. 16. FIG. 27 conceptually illustrates an image resizing process performed by an encoder driver of some embodiments, such as the encoder shown in FIG. 26. FIG. 17 conceptually illustrates a software architecture for the decoder driver of some embodiments, such as the software architecture shown in FIG. 16. FIG. 29 conceptually illustrates an image extraction process performed by a decoder driver of some embodiments, such as the decoder driver shown in FIG. 28. FIG. 4 illustrates an encoder driver of some embodiments including two rate controllers. FIG. 17 conceptually illustrates a software architecture for a networking manager of some embodiments, such as the networking manager shown in FIG. 16. FIG. 9 illustrates a user interface of some embodiments for a snap-to-corner operation. FIG. 9 illustrates another user interface of some embodiments for a corner snap operation. FIG. 9 illustrates a user interface of some embodiments for a PIP display rotation operation. FIG. 10 illustrates another user interface of some embodiments for a PIP display rotation operation. FIG. 10 illustrates another user interface of some embodiments for a PIP display rotation operation. FIG. 10 illustrates another user interface of some embodiments for a PIP display rotation operation. FIG. 9 illustrates a user interface of some embodiments for resizing a foreground insertion display area of a PIP display. FIG. 10 illustrates another user interface of some embodiments for resizing an insertion display area of a PIP display. FIG. 10 illustrates another user interface of some embodiments for resizing an insertion display area of a PIP display. FIG. 10 illustrates another user interface of some embodiments for resizing an insertion display area of a PIP display. FIG. 4 illustrates a user interface of some embodiments for identifying a region of interest in a display. FIG. 6 illustrates another user interface of some embodiments for identifying a region of interest in a display. FIG. 6 illustrates another user interface of some embodiments for identifying a region of interest in a display. FIG. 9 illustrates a process of some embodiments for performing a local camera switching operation on a dual camera portable device. FIG. 10 illustrates a user interface of some embodiments for a camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a camera switching operation. FIG. 9 illustrates a process of some embodiments for performing a remote camera switching operation on a dual camera portable device. FIG. 7 illustrates a user interface of some embodiments for a remote control camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a remote control camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a remote control camera switching operation. FIG. 9 illustrates another user interface of some embodiments for a remote control camera switching operation. FIG. 4 conceptually illustrates a process of some embodiments for performing an exposure adjustment operation. FIG. 9 illustrates a user interface of some embodiments for an exposure adjustment operation. FIG. 10 illustrates another user interface of some embodiments for an exposure adjustment operation. FIG. 10 illustrates another user interface of some embodiments for an exposure adjustment operation. FIG. 17 conceptually illustrates an exposure adjustment process performed by the image processing manager of some embodiments, such as the process illustrated in FIG. 16. FIG. 7 is a diagram conceptually illustrating an exemplary exposure adjustment operation of some embodiments. FIG. 7 conceptually illustrates a process of some embodiments for performing a focus adjustment operation. FIG. 9 illustrates a user interface of some embodiments for a focus adjustment operation. FIG. 9 illustrates another user interface of some embodiments for a focus adjustment operation. FIG. 9 illustrates another user interface of some embodiments for a focus adjustment operation. FIG. 9 illustrates different display configurations of some embodiments for video captured from one or more dual camera portable devices. FIG. 7 illustrates a user interface of some embodiments for superimposing a foreground of an insertion video on a background video in a PIP display. FIG. 4 illustrates a technique of some embodiments for determining a foreground of a video image. FIG. 7 illustrates a user interface of some embodiments for exchanging an inserted display of a PIP display with a background display during a video conference. FIG. 9 illustrates a user interface of some embodiments for a corner snap operation. FIG. 9 illustrates a user interface of some embodiments for corner snap and push operations. FIG. 9 illustrates a user interface of some embodiments for a PIP display rotation operation. FIG. 10 illustrates another user interface of some embodiments for a PIP display rotation operation. FIG. 4 illustrates a user interface of some embodiments for selecting one video from two remote videos during a video conference. FIG. 4 illustrates a user interface of some embodiments for selecting one video from two local videos during a video conference. FIG. 4 illustrates a user interface of some embodiments for pre-conference selection of a video to use for a video conference. FIG. 4 illustrates an example of a bandwidth allocation between two videos captured by a dual camera portable device of some embodiments. FIG. 4 conceptually illustrates an arbitrator module of some embodiments for managing a rate controller of a dual camera portable device. FIG. 4 conceptually illustrates a method of some embodiments for encoding an image captured by a camera of a dual camera portable device. FIG. 4 conceptually illustrates another embodiment of some methods for encoding images captured by a camera of a dual camera portable device. FIG. 80 illustrates an exemplary image composition for the method illustrated in FIG. 79. FIG. 4 conceptually illustrates another embodiment of some methods for encoding images captured by a camera of a dual camera portable device. FIG. 4 conceptually illustrates a method of some embodiments for decoding an image captured by a camera of a dual camera portable device. FIG. 7 conceptually illustrates another embodiment of some methods for decoding images captured by a camera of a dual camera portable device. FIG. 6 conceptually illustrates another software architecture for the video conferencing and processing module of the dual camera portable device of some embodiments. FIG. 6 illustrates a user interface of some embodiments for a multi-participant video conference. FIG. 6 illustrates another user interface of some embodiments for a multi-participant video conference. FIG. 6 illustrates another user interface of some embodiments for a multi-participant video conference. FIG. 4 conceptually illustrates an application programming interface (API) architecture of some embodiments. FIG. 4 illustrates an architecture for a dual camera mobile computing device of some embodiments. FIG. 4 conceptually illustrates a touch input / output (I / O) device of some embodiments. 1 conceptually illustrates an exemplary communication system of some embodiments. FIG. 4 conceptually illustrates another exemplary communication system of some embodiments.

  In the following description, numerous details are set forth for explanation. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order not to obscure the description of the present invention with unnecessary detail.

  Some embodiments of the present invention provide a portable device having two cameras capable of taking pictures and videos. Examples of a mobile device include a mobile phone, smart phone, personal digital assistant (PDA), laptop, tablet personal computer, or any other type of mobile computing device. As used herein, a photo refers to still photo images taken with a camera, one at a time in single photo mode, or multiple at once in fast-action mode. Video, on the other hand, refers to a series of video images captured by a camera at a particular speed, which is often referred to as the frame rate. Typical frame rates for capturing video are 25 frames per second (fps), 30 fps and 60 fps. The camera of the portable device of some embodiments may capture video images (ie, video frames) at these and other frame rates.

  The mobile device of some embodiments can (1) display captured photographic and video images, (2) store the captured images for later transmission to another device, and (3) device The captured image can be transmitted to one or more devices during a real-time communication session between the users of the plurality of users, and (4) encoding the captured image for local storage or transmission to another device. it can.

  An example of a real-time communication session involving the transmission of a captured video image is a video conference. In some embodiments, the mobile device can transmit only the captured video images of one camera at any given time during a video conference. However, in other embodiments, the portable device can simultaneously transmit video images captured from both of its cameras during a video conference or other real-time communication session.

  In some embodiments, the mobile device generates a composite display that includes a simultaneous display of multiple videos captured by multiple cameras of one or more devices. In some cases, the composite display places the video in an adjacent display area (eg, an adjacent window). FIG. 1 shows two adjacent display areas 105 and 110 that simultaneously display two videos captured by two cameras on one device or captured by two cameras on two different devices during a video conference. An example of such a composite display 100 is shown.

  In other cases, the composite display is a PIP display that includes at least two display areas showing two different videos, one of the display areas being a background main display area and the other being a foreground insertion overlapping the background main display area. The display area. FIG. 2 shows one such example of a composite PIP display 200. The composite PIP display 200 includes a background main display area 205 and a foreground insertion display area 210 overlapping the background main display area. The two display areas 205 and 210 simultaneously display two videos captured by two cameras on one device or by two cameras on two different devices during a video conference. The exemplary composite PIP display shown and described herein is similar to the composite PIP display 200 in which the entire foreground insert display area 210 is displayed inside the background main display area 205, except that the background main display area 205 Other composite PIP displays are possible that have a foreground insert display area 210 that overlaps but is not completely inside.

  In addition to transmitting video content during a video conference with another device, the portable device of some embodiments can transmit other types of content along with the video content of the conference. An example of such other content is a low-resolution or high-resolution photographic image captured by one of the device's cameras while another camera of the device is capturing video used in a video conference. There is. Other examples of such other contents include (1) files and other contents stored in the device, (2) screen display of the device (that is, content displayed on the screen of the device), and (3) television. Such as content received from another device during a conference or other real-time communication session.

  The mobile device of some embodiments uses a novel in-conference coordination technique for making adjustments during a video conference. For example, the portable device of some embodiments can dynamically switch to transmit video captured by another camera while transmitting only video captured by one camera during a video conference. . In such a situation, the mobile device of some embodiments notifies any other device participating in the video conference of this switch, so that the other device is captured by the two cameras Smooth transitions between videos can be achieved on this device.

  In some embodiments, a camera switch request may occur at a "local" device that switches its camera during a video conference, as well as other "remote" devices receiving video captured by the local device. Can also occur. Further, the ability of one device to instruct another device to switch cameras is only one example of the remote control functionality of some embodiments of the device. Examples of other operations that can remotely instruct the device in some embodiments include an exposure adjustment operation (eg, automatic exposure), a focus adjustment operation (eg, autofocus), and the like. Another example of a new in-meeting adjustment that can be specified locally or remotely is the identification of a region of interest (ROI) in the captured video, where the purpose of the ROI identification is to change the behavior of the capturing camera To change the image processing operation of the device having the capturing camera, or to change the encoding operation of the device having the capturing camera.

  Yet another example of a new in-meeting adjustment of some embodiments involves real-time modification of the composite video display generated by the device. Specifically, in some embodiments, the real-time modification of the composite video display involves moving one or more of the display areas within the composite display in response to user selection and movement of the display area. . Also, in some embodiments, if the screen of the device that provided the composite display rotates, the composite display is rotated during the video conference. Also, the mobile device of some embodiments allows the user of the device to reverse the order of the videos in the PIP display (i.e., the background main display video is displayed in the foreground insert display while the foreground insert video is displayed). Is displayed in the background main display).

  Some more detailed embodiments are described below. Section I provides a description of the video processing architecture of some embodiments. Then, Section II describes the captured image processing units of some embodiments. In some embodiments, this unit is a component of the device that is responsible for processing RAW images captured by the device's camera.

  Next, Section III describes the video conferencing architecture of some embodiments. This section describes some schemes for setting up a single camera video conference, as well as the video conference module of some embodiments. Thereafter, Section IV describes the in-meeting coordination and control operations of some embodiments. Thereafter, Section V describes the video conferencing feature of an embodiment for transmitting and displaying multiple videos from individual devices during a video conference. Next, transmission of real-time video along with non-real-time content during a video conference in section VI will be described. Finally, Section VII describes the hardware architecture of the dual camera device of some embodiments.

I. Video Capture and Processing FIG. 3 conceptually illustrates a video processing and encoding module 300 of a dual camera portable device of some embodiments. In some embodiments, module 300 processes images captured by the camera of the dual camera mobile device and encodes video captured by the camera of the dual camera mobile device as well. As shown in FIG. 3, the module 300 includes a captured image processing unit (CIPU) driver 305, a media exchange module 310, an encoder driver 320, and a video processing module 325.

  In some embodiments, the media exchange module 310 allows programs on devices that are consumers and producers of media content to exchange instructions regarding media content and processing of the media content. In the video processing and encoding module 300, the media exchange module 310 of some embodiments transfers instructions and media content between the video processing module 325 and the CIPU driver 305 and between the video processing module 325 and the encoder driver 320. Route control. To facilitate the routing of such instructions and media content, the media exchange module 310 of some embodiments provides a collection of application programming interfaces (APIs) used by media content consumers and producers. In some of such embodiments, media exchange module 310 is a collection of one or more frameworks that are part of an operating system running on a dual camera portable device. One example of such a media exchange module 310 is the Core Media framework provided by Apple Inc.

  Video processing module 325 performs image processing on images and / or video captured by the device's camera. Examples of such operations include exposure adjustment operation, focus adjustment operation, perspective correction, dynamic range adjustment, image size change, image synthesis, and the like. In some embodiments, some image processing operations may be performed by the media exchange module 310. For example, as shown in FIG. 3, the media exchange module 310 of some embodiments performs a temporal noise reduction (TNR) operation (eg, by the TNR 315) that reduces noise in video images captured by the device's camera. I do. Further examples of such image processing operations of the video processing module 325 and the media exchange module 310 are provided below.

  As described above, the video processing module 325 interacts with the CIPU driver 305 and the encoder driver 320 through the media exchange module 310. The CIPU driver 305 functions as a communication interface between the captured image processing unit (CIPU) 330 and the media exchange module 310. As described further below, CIPU 330 is a component of a dual camera device that is responsible for processing images captured during an image capture operation or a video capture operation of the device's camera. The CIPU driver 305 receives image and / or video requests from one or both of the device's cameras from the video processing module 325 through the media exchange module 310. CIPU driver 305 relays such a request to CIPU 330 and, in response, receives the requested image and / or video from CIPU 330, and CIPU driver 305 then forwards the image and / or video to the media exchange module. Send to video processing module 325 through 310. The video processing module 325 of some embodiments also modifies some of its operations through the CIPU driver 305 and the media exchange module 310 (eg, camera frame rate, exposure adjustment operation, focus adjustment operation, etc.). To the CIPU 330 to change

  The encoder driver 320 functions as a communication interface between the media exchange module 310 and the encoder hardware 335 (eg, an encoder chip, an encoding component on a system-on-chip, etc.). In some embodiments, encoder driver 320 receives an image and a request to encode the image from video processing module 325 through media exchange module 310. The encoder driver 320 sends the image to be encoded to the encoder 335, which then performs picture or video encoding on the image. If the encoder driver 320 receives the encoded image from the encoder 335, the encoder driver 320 sends the encoded image back to the video processing module 325 through the media exchange module 310.

  In some embodiments, video processing module 325 may perform various operations on the encoded images received from the encoder. Examples of such operations include storing the encoded image in a storage device of the device, transmitting the encoded image during a video conference through the device's network interface, and the like.

  In some embodiments, some or all of the modules of the video processing and encoding module 300 are implemented as part of an operating system. For example, in some embodiments, all four components 305, 310, 320 and 325 of this module 300 are implemented as part of the device operating system. In another embodiment, the media exchange module 310, CIPU driver 305, and encoder driver 320 are implemented as part of the operating system of the device, and the video processing module 325 has an application running on the operating system. Further, other implementations of module 300 are possible.

  The operation of the video processing and encoding module 300 during a video capture session is described below. To start a video capture session, video processing module 325 initializes several components required for the video capture session. In some embodiments, these components include (1) CIPU 330, (2) scaling and compositing module of video processing module 325 (not shown), and (3) image processing module of video processing module 325 (not shown). And (4) an encoder 335. Also, the video processing module 325 of some embodiments initializes a network manager (not shown) when joining a video conference.

  The video processing module sends its initialization request to CIPU 330 via media exchange module 310 and CIPU driver 305 to cause one or both of the device's cameras to begin capturing video. In some embodiments, the request specifies a particular frame rate, exposure level, and scaling size for each camera that needs to capture video. In response to this request, CIPU 330 begins returning video images from the requested camera at the specified rate (s), exposure level (s), and scaling size (s). These video images are returned to the video processing module 325 through the CIPU driver 305 and the media exchange module 310, which performs the TNR operation on the video images before supplying the video processing module 325 as described above. I do. At the video processing module 325, the video images are stored in a buffer (not shown) for further image processing.

  The image processing module of the video processing module 325 reads the buffered video image for further video processing. The scaling and compositing module then reads the processed video image for scaling if necessary for real-time display on the display screen of the device. In some embodiments, this module may be used by another device during a videoconference to provide a real-time display of the captured video image on the device or to create a composite video image for encoding. A composite image is created from images captured by two cameras of the device with the camera (s) or from images captured by the camera (s) of the device.

  The processed and / or synthesized video images are provided to encoder 335 through encoder driver 320 and media exchange module 310. Thereafter, the encoder 335 encodes the video image. The encoded image is then returned to video processing module 325 (also via encoder driver 320 and media exchange module 310) for storage at the device or transmission during a video conference. If the device is participating in a video conference, the network manager (initialized by the video processing module 325) will then read these encoded images and packetize these images into the network interface of the device. (Not shown) to one or more other devices.

II. Processing of captured image Single Pipeline The image captured by the camera of the dual camera portable device of some embodiments is a raw RAW image. These images need to be converted to a specific color space before they can be used for other operations, such as transmitting the image to another device (eg, during a video conference), storing the image, or displaying the image. And Further, images captured by the camera may need to be processed to correct errors and / or distortions and adjust the color, size, etc. of the images. Thus, some embodiments perform some processing operations on images before storing, transmitting, and displaying such images. Part of the processing of such an image is executed by the CIPU 330.

  An example of such a CIPU is shown in FIG. Specifically, this figure conceptually illustrates a captured image processing unit (CIPU) 400 of some embodiments. The CIPU 400 may process images from only one of the device's cameras at a time, or may simultaneously process images from both of the device's cameras in a time division multiplexed (ie, time interleaved) manner. One processing pipeline 485 is included. The processing pipeline 485 of CIPU 400 may be configured differently to accommodate different characteristics and / or operational settings of different cameras. Examples of different camera characteristics in some embodiments include different resolutions, noise sensors, lens types (fixed lens or zoom lens), and the like. Examples of different operation settings that allow the device to operate the camera in some embodiments include image resolution size, frame rate, zoom level, exposure level, and the like.

  As shown in FIG. 4, the CIPU 400 includes a sensor module 415, a line / frame buffer 417, a bad pixel correction (BPC) module 420, a lens shading (LS) module 425, a demosaicing module 430, and a white balance ( A WB) module 435, a gamma module 440, a color space conversion (CSC) module 445, a hue, saturation, and contrast (HSC) module 450, a scaler module 455, a filter module 460, a statistics engine 465, It includes two sets of registers 470 and a controller module 475. In some embodiments, all of the modules of CIPU 400 are implemented as hardware (eg, ASICs, FPGAs, SOCs with microcontrollers, etc.), while in other embodiments, some or all of the modules of CIPU 400 are implemented. Is implemented as software.

  As shown in FIG. 4, the sensor module 415 is communicatively coupled to two pixel arrays 410a and 410b and two sets of sensors 405a and 405b of the two cameras of the device. In some embodiments, this communicable coupling is facilitated by the Mobile Industry Processor Interface (MIPI) of each camera sensor.

  This communicable coupling allows the sensor module 415 to transfer instructions to the cameras to control various aspects of their operation, such as the power level, zoom level, focus, exposure level, etc. of each camera. In some embodiments, each camera has four operating power modes. In the first operating power mode, the camera is turned off. In the case of the second operating power mode, the camera is powered on but has not been configured. In the third operating power mode, the camera is powered on, the sensor of the camera is configured, and the pixels of the camera sensor collect photons and convert the collected photons to digital values. However, the camera sensor has not yet sent the image to the sensor module 415. Finally, in the fourth operating power mode, the camera is in the same operating power mode as the third power mode, except that the camera is currently sending an image to the sensor module 415.

  During operation of the device, the camera may switch from one operating power mode to another any number of times. When switching between operating power modes, some embodiments require that the camera switch between operating power modes in the order described above. Thus, in these embodiments, a camera in the first operating power mode can switch to only the second operating power mode. The camera can switch to the first operating power mode or the third operating power mode when in the second operating power mode. Similarly, the camera can switch from the third operating power mode to the second operating power mode or the fourth operating power mode. The camera can only switch back to the third operating power mode when in the fourth operating power mode.

  Furthermore, switching from one operating power mode to the next or previous operating power mode takes a certain amount of time. Therefore, switching between two or three operating power modes is slower than switching between one operating power mode. The amount of power consumed differs depending on the operating power mode. For example, the fourth operating power mode consumes the most amount of power, the third operating power mode consumes more power than the first operating power mode and the second operating power mode, and the second operating power mode Consumes more power than the first operating power mode. In some embodiments, the first operating power mode consumes no power.

  The camera may remain in one of the other operating power modes when not in the fourth operating power mode for capturing images. Determining the mode of operation to keep an unused camera in place depends on how much power the camera is allowed to consume, and how quickly the camera must respond to a request to start capturing images. It depends. For example, a camera configured to operate in a third operating power mode (eg, a standby mode) is better than a camera configured to be in a first operating power mode (ie, powered off). Consumes a lot of power. However, if the camera is commanded to capture an image, a camera operating in the third operating power mode will enter the fourth operating power mode more quickly than a camera operating in the first operating power mode. Can be switched. In this way, the camera may be configured to operate in different operating power modes based on various requirements (eg, response time to image capture request, power consumption) when no image is captured.

  As described further below, when the video processing module 325 requests one or both cameras to begin capturing images and the sensor module 415 receives this request through the controller module 475, the sensor module 415 Through a communicable coupling with a set of one or both camera sensors, one can instruct the set of camera sensors to begin capturing images. A Bayer filter is superimposed on each of the camera sensors, so that each camera sensor outputs a Bayer pattern image, which is stored in a pixel array associated with each camera sensor. The Bayer pattern image is an image in which each pixel stores only one color among red, blue, and green.

  The sensor module 415 reads out the Bayer pattern RAW image stored in the camera pixel arrays 410a and 410b through its own connection with the pixel arrays 410a and 410b. By controlling the rate at which sensor module 415 reads images from the camera's pixel array, sensor module 415 can control the frame rate of video images being captured by a particular camera. The sensor module 415 interleaves the fetch of images captured by different cameras to interleave the image processing of images captured by different cameras by the CIPU processing pipeline 485 by controlling its image readout speed. You can also. Regarding the control of the image reading of the sensor module 415, the following subsection II. A. 1 and II. A. This will be further described in Section 2.

  The sensor module 415 reads an image line (that is, a pixel row of an image) from the pixel arrays 410 a and 410 b and stores the image line in the line / frame buffer 417. Each image line in line / frame buffer 417 is processed by CIPU processing pipeline 485. As shown in FIG. 4, the CIPU processing pipeline 485 includes a BPC module 420, an LS module 425, a demosaicing module 430, a WB module 435, a gamma module 440, a CSC module 445, an HSC module 450, a scaler module 455, and a filter module 460. Formed by In some embodiments, CIPU processing pipeline 485 processes images from line / frame buffer 417 line by line (ie, line by line), while in other embodiments, CIPU processing pipeline 485 includes: The entire image from the line / frame buffer 417 is processed frame by frame.

  In the exemplary pipeline shown in FIG. 4, the BPC module 420 is a module that reads an image from the line / frame buffer 417. This module performs a bad pixel elimination operation that attempts to correct bad pixels in the read image, the bad pixels having one or more of the camera sensors being defective (e.g., having a defect). The light sensor does not detect the light at all or detects the light incorrectly). In some embodiments, the BPC module 420 detects bad pixels by comparing a particular pixel in the image with one or more adjacent pixels in the image. If the difference between the value of a particular pixel and the value of an adjacent pixel is greater than the threshold amount, then the value of the particular pixel is adjacent to the same color (ie, red, green and blue) as the particular pixel. It is replaced by the average of the values of some pixels.

  The operation of BPC module 420 is controlled in part by values stored in two sets of registers 470 of CIPU 400 for this module. Specifically, to process images captured by two different cameras of the device, in some embodiments, as described above, the CIPU processing pipeline 485 is configured differently for each camera. Is done. CIPU processing pipeline 485 is configured for two different cameras by storing two different sets of values in two different sets of registers 470a (Ra) and 470b (Rb) of CIPU 400. Each set of registers 470 contains one register (Ra or Rb) for each of the modules 420-460 in the CIPU processing pipeline 485. Each register in each register set stores a set of values that define the operation of one processing pipeline module. Thus, as shown in FIG. 4, register set 470a is to indicate the mode of operation of each processing pipeline module for one camera (Camera A) of a dual camera portable device, and register set 470b is This is to indicate a mode of operation of each module for the other camera (camera B) of the camera portable device.

  One example of configuring the CIPU processing pipeline 485 differently for each camera is to configure the modules of the CIPU processing pipeline 485 to process images of various sizes. For example, if the camera sensor 405a is 640 × 480 pixels and the camera sensor 405b is 2048 × 1536 pixels, the set of registers 470a instructs the module of the CIPU processing pipeline 485 to process a 640 × 480 pixel image. The set of registers 470 b is configured to store a value that instructs a module of the CIPU processing pipeline 485 to process a 2048 × 1536 pixel image.

  In some embodiments, different processing pipeline configurations (ie, register values) are stored in different profile settings. In some of such embodiments, the user of the mobile device configures one of the profile settings (e.g., via a user interface displayed on the mobile device) to configure the operation of the camera (s). You are allowed to choose. For example, the user may have profile settings to configure the camera to capture high-resolution video, profile settings to configure the same camera to capture low-resolution video, or both cameras to capture high-resolution still images. You can select the profile settings to configure. Various configurations are possible and can be stored in a number of different profile settings. In other such embodiments, instead of having the user select a profile setting, the profile setting is automatically selected based on the application or activity selected by the user. For example, if the user selects the video conferencing application, the profile that configures both cameras to capture video is automatically selected, and if the user selects the photo application, the camera is configured to capture still images. For example, a profile constituting one of them is automatically selected.

  After the BPC module 420, the LS module 425 receives the defective pixel corrected image. The LS module 425 performs a lens shading correction operation to correct image defects caused by a camera lens that produces a light fall-off effect (i.e., light decreases toward the edge of the camera sensor). Such an effect causes the image to be unevenly illuminated (eg, darker at corners and / or edges). To correct for these image defects, the LS module 425 of some embodiments estimates a mathematical model of the illumination falloff of the lens. The estimated model is then used to compensate for the lens falloff of the image to evenly illuminate the unevenly illuminated portion of the image. For example, if the brightness of a corner of the image is half of the center of the image, the LS module 425 of some embodiments doubles the pixel value of the corner to produce a uniform image.

  The demosaicing module 430 performs a demosaicing operation to generate a full color image from the sampled color image. As described above, the camera sensor outputs a Bayer pattern image. However, since each pixel of the Bayer pattern image stores only one color value, the Bayer pattern image is incomplete. The demosaicing module 430 reconstructs a red, green, blue (RGB) image from the Bayer pattern image by interpolating the color values for each color set of the Bayer pattern image.

  The WB module 435 performs a white balance operation on the RGB images received from the demosaicing module 430 such that the color of the content of the image is similar to the color of such content as perceived by the human eye in real life. . The WB module 435 adjusts the white balance by adjusting the colors of the image to properly render neutral colors (eg, gray, white, etc.). For example, a single blank image under incandescent light may appear yellow, but the human eye perceives the paper as white. To account for differences between the color of the image captured by the sensor and the color perceived by the human eye, the WB module 435 determines the image so that the captured image appropriately reflects the color perceived by the human eye. Adjust the color value of.

  The statistics engine 465 collects image data at various stages of the CIPU processing pipeline 485. For example, FIG. 4 illustrates that the statistics engine 465 collects image data after the LS module 425, demosaicing module 430, and WB module 435. In various embodiments, data is collected from any number of various stages of the CIPU processing pipeline 485. The statistics engine 465 processes the collected data and adjusts the operation of the camera sensors 405a and 405b through the controller module 475 and the sensor module 415 based on the processed data. Examples of such operations include exposure and focus. Although FIG. 4 shows that the statistics engine 465 controls camera sensors 405 a and 405 b through the controller module 475, other embodiments of the statistics engine 465 control camera sensors just through the sensor module 415.

  The processed data can also be used to coordinate the operation of various modules of CIPU 400. For example, the statistics engine 465 of some embodiments adjusts the operation of the WB module 435 based on data collected after the WB module 435. In some of such embodiments, the statistics engine 465 provides an automatic white balance (AWB) function by adjusting the white balance operation of the WB module 435 using the processed data. In other embodiments, processed data collected from any number of stages of the CIPU processing pipeline 485 can be used to coordinate the operation of any number of modules in the CIPU processing pipeline 485. Further, statistics engine 465 may also receive instructions from controller module 475 to coordinate the operation of one or more modules of CIPU processing pipeline 485.

  After receiving the image from the WB module 435, the gamma module 440 performs a gamma correction operation on the image to encode and decode the luminance or tristimulus values of the camera system. The gamma module 440 of some embodiments corrects gamma by converting a 10- to 12-bit linear signal to an 8-bit non-linear encoded signal to correct for gamma of an image. In some embodiments, gamma is corrected by using a look-up table.

  The CSC module 445 converts the image received from the gamma module 440 from one color space to another color space. Specifically, the CSC module 445 converts the image from the RGB color space to the luminance and color difference (YUV) color space. However, other embodiments of the CSC module 445 can convert an image from any number of color spaces to any number of color spaces.

  The HSC module 450 may adjust the hue, saturation, contrast, or any combination thereof of the image received from the CSC module 445. The HSC module 450 can, for example, adjust these properties to reduce noise or enhance images. For example, the saturation of an image captured by a low noise camera sensor can be increased to make the image appear sharper. In contrast, the saturation of images captured by high noise camera sensors can be reduced to reduce the color noise in such images.

  After the HSC module 450, the scaler module 455 can resize the image to adjust the pixel resolution of the image or adjust the data size of the image. The scaler module 455 may, for example, reduce the size of the image to accommodate a smaller display. The scaler module 455 can scale an image in several different ways. For example, the scaler module 455 can enlarge (ie, stretch) and shrink (ie, shrink) an image. The scaler module 455 can also scale the image proportionally or scale the image anamorphically.

  Filter module 460 applies one or more filtering operations to the image received from scaler module 455 to change one or more attributes of some or all pixels of the image. Examples of filters include a low-pass filter, a high-pass filter, a band-pass filter, a bilateral filter, and a Gaussian filter, among other examples. As such, the filter module 460 can apply any number of different filters to the image.

  The controller module 475 of some embodiments is a microcontroller that controls the operation of the CIPU 400. In some embodiments, the controller module 475 includes (1) operation of the camera sensor (eg, exposure level) through the sensor module 415, (2) operation of the CIPU processing pipeline 485, (3) of the CIPU processing pipeline 485. Control timing (eg, when to switch camera sensors, when to switch registers, etc.), and (4) flash / strobe (not shown) that is part of the dual camera portable device of some embodiments. .

  Some embodiments of the controller module 475 process instructions received from the statistics engine 465 and the CIPU driver 480. In some embodiments, the instructions received from CIPU driver 480 are instructions from a dual camera portable device (ie, received from a local device), while in other embodiments, the instructions received from CIPU driver 480 are Command from another device (eg, remote control during a videoconference). Controller module 475 can adjust the operation of CIPU 400 by programming the value of register 470 based on the processed instruction. Further, controller module 475 can dynamically reprogram the value of register 470 during operation of CIPU 400.

  As shown in FIG. 4, CIPU 400 includes a plurality of modules in CIPU processing pipeline 485. However, those skilled in the art will appreciate that CIPU 400 may be implemented with only a few of the described modules or different additional modules. Further, the processing performed by the different modules may be applied to the images in a different order than the order shown in FIG.

  Here, an exemplary operation of the CIPU 400 will be described with reference to FIG. For purposes of illustration, the set of registers Ra is used to process images captured by the camera sensor 405a of the dual camera portable device, and the set of registers Rb is captured by the camera sensor 405b of the dual camera portable device. Used to process images. Controller module 475 receives instructions from CIPU driver 480 to generate an image captured by one of the cameras of the dual camera portable device.

  Controller module 475 then initializes various modules of CIPU processing pipeline 485 to process images captured by one of the cameras of the dual camera portable device. In some embodiments, this includes the controller module 475 confirming that an appropriate set of registers of the registers 470 is being used. For example, if the CIPU driver 480 instructs the controller module 475 to generate an image captured by the camera sensor 405a, the controller module 475 may have a set of registers Ra that is a set of registers read by the modules of the CIPU 400. Make sure that Otherwise, the controller module 475 switches the set of registers such that the set of registers Ra is the set read by the modules of the CIPU 400.

  For each module of the CIPU processing pipeline 485, the mode of operation is indicated by the value stored in the set of registers Ra. As described above, the values of the set of registers 470 may be dynamically reprogrammed during operation of CIPU 400. Thus, processing of one image may be different from processing of the next image. Although the description of this exemplary operation of CIPU 400 describes that each module of CIPU 400 reads a value stored in a register indicating the mode of operation of the module, some software implemented embodiments Instead, parameters are passed to the various modules of CIPU 400 instead.

  In some embodiments, the controller module 475 initializes the sensor module 415 by instructing the sensor module 415 to delay a certain amount of time after reading an image from the pixel array 410a. In other words, the controller module 475 instructs the sensor module 415 to read an image from the pixel array 410a at a specific rate.

  Next, the controller module 475 instructs the camera sensor 405a via the sensor module 415 to capture an image. In some embodiments, controller module 475 also provides camera sensor 405a with exposure and other camera operating parameters. In another embodiment, camera sensor 405a uses default values for camera sensor operating parameters. The camera sensor 405a captures a RAW image stored in the pixel array 410a based on the parameters. The sensor module 415 reads the RAW image from the pixel array 410a and sends the image to the line / frame buffer 417 for storage before the CIPU processing pipeline 485 processes the image.

  In certain circumstances, the image may be discarded by line / frame buffer 417. If the camera sensors 405a and / or 405b are capturing images at high speed, the sensor module 415 may be able to read the images from the BPC module 420 from the line / frame buffer 417 (eg, capture high frame rate video). Also, the image can be received and stored in the line / frame buffer 417 at high speed, and the line / frame buffer 417 will be full. When this occurs, the line / frame buffer 417 of some embodiments discards the image (ie, frame) on a first-in first-out basis. That is, when line / frame buffer 417 discards an image, line / frame buffer 417 discards the image received before all other images in line / frame buffer 417.

  Processing of the image by the CIPU processing pipeline 485 begins when the BPC module 420 reads the image from the line / frame buffer 417 to correct for bad pixels in the image. After that, the BPC module 420 sends the image to the LS module 425 to correct uneven illumination of the image. After the illuminance of the image is corrected, the LS module 425 sends the image to a demosaicing module 430 that processes the RAW image and generates an RGB image from the RAW image. Next, the WB module 435 receives the RGB images from the demosaicing module 430 and adjusts the white balance of the RGB images.

  As described above, the statistics engine 465 may have collected some data at various points in the CIPU processing pipeline 485. For example, the statistics engine 465 collects data after the LS module 425, demosaicing module 430, and WB module 435, as shown in FIG. Based on this collected data, the statistics engine 465 may operate the camera sensor 405a, the operation of one or more modules of the CIPU processing pipeline 485, to adjust subsequent image capture from the camera sensor 405a. Or both can be adjusted. For example, based on the collected data, the statistics engine 465 may instruct the camera sensor 405a through the sensor module 415 to increase the exposure level for the current image that is too low, and therefore for subsequently captured images. May be determined. Accordingly, the statistics engine 465 of some embodiments operates as a feedback loop for some processing operations.

  After adjusting the white balance of the image, the WB module 435 sends the image to the gamma module 440 for gamma correction (eg, to adjust the gamma curve of the image). The CSC module 445 receives the gamma-corrected image from the gamma module 440 and performs color space conversion. In this example, the CSC module 445 converts an RGB image into a YUV image. In other words, the CSC module 445 converts an image represented in the RGB color space into an image represented in the YUV color space. HSC module 450 receives the YUV image from CSC module 445 and adjusts the hue, saturation, and contrast attributes of various pixels in the image. After HSC module 450, scaler module 455 resizes the image (eg, stretches or shrinks the image). After receiving the image from the scaler module 455, the filter module 460 applies one or more filters to the image. Finally, the filter module 460 sends the processed image to the CIPU driver 480.

  In this example of the operation of CIPU 400 described above, each module of CIPU processing pipeline 485 processed the image in some manner. However, other images processed by CIPU 400 may not require processing by all modules of CIPU processing pipeline 485. For example, an image may not require white balance adjustment, gamma correction, scaling, or filtering. Thus, CIPU 400 may process the image in any of a number of ways based on various received inputs, such as instructions from CIPU driver 480, or data collected by statistics engine 465.

  Different embodiments control the speed at which the image is processed (ie, the frame rate) differently. One way to control the frame rate is by manipulating the vertical blanking interval (VBI). In some embodiments, reading the image lines to process the image line by line, the VBI reads the last line of the image of the video captured by the camera of the dual camera portable device from the pixel array, This is the time difference from reading the first line of the next image from the pixel array. In another embodiment, the VBI is the time difference between reading one image of the video captured by the camera of the dual camera portable device from the pixel array and reading the next image of the video from the pixel array.

  One example where VBI can be used is between the sensor module 415 and the pixel arrays 410a and 410b. For example, some embodiments of the sensor module 415 read images from the pixel arrays 410a and 410b line by line, and other embodiments of the sensor module 415 read images from the pixel arrays 410a and 410b image by image. Accordingly, the frame rate can be controlled by adjusting the VBI of sensor module 415. Increasing VBI decreases the frame rate, and decreasing VBI increases the frame rate.

1. Using VBI for Single Camera: Frame Rate Control FIG. 5 conceptually illustrates examples of different frame rates 505, 510 and 515 based on different VBIs. Each sequence shows an image of a person with a guitar at various times 525-555 along timeline 520, captured by one of the cameras of the dual camera portable device. Further, the time between the times 525 to 555 is the same, and is called one unit time. For purposes of explanation, FIG. 5 will now be described with respect to sensor module 415 and pixel array 410a of FIG. Accordingly, each image represents the time at which the sensor module 415 reads the image from the pixel array 410a along the time line 520.

  At the exemplary frame rate 505, the VBI of the sensor module 415 for the pixel array 410a is set to 3 unit times (eg, by the controller module 475). That is, the sensor module 415 reads an image from the pixel array 410a every three times along the time line 520. As shown in the example frame rate 505, the sensor module 415 reads out images at times 525, 540, and 555. Thus, the exemplary frame rate 505 has a frame rate of one image every three unit times.

  Exemplary frame rate 510 is similar to exemplary frame rate 505 except that the VBI is set to two unit times. Therefore, the sensor module 415 reads an image from the pixel array 410a every two times along the time line 520. The example frame rate 510 indicates that the sensor module 415 reads an image from the pixel array 410a at times 525, 535, 545, and 555. Since the VBI of example frame rate 510 is smaller than the VBI of example frame rate 505, the frame rate of example frame rate 510 is higher than the frame rate of example frame rate 505.

  The example frame rate 515 is similar to the example frame rate 505, except that the VBI of the sensor module 415 for the pixel array 410a is set to one unit time. Accordingly, the sensor module 415 is instructed to read an image from the pixel array 410a at each time along the timeline 520. As illustrated, the sensor module 415 reads an image from the pixel array 410a at times 525 to 555. The VBI at the example frame rate 515 is smaller than the VBI at the example frame rates 505 and 510. Accordingly, the frame rate of example frame rate 515 is higher than example frame rates 505 and 510.

2. Use of VBI for Two Cameras In some embodiments, it is desirable to operate both cameras of a dual camera portable device simultaneously (eg, transmitting video from both cameras during a video conference). sell. Different embodiments of a dual camera portable device that include a single processing pipeline provide different mechanisms for operating both cameras of the dual camera portable device simultaneously.

  One such mechanism is to interleave processing of the images captured by both cameras by controlling the VBI of each camera. That is, one or more images captured by one camera are captured and processed during the VBI of the other camera, and one or more images captured by the other camera are processed during the VBI of one camera. Captured and processed. Since the CIPU 400 described above has a single processing pipeline 485, this mechanism may be implemented in the CIPU 400 in some embodiments. In such an embodiment, sensor module 415 reads an image from one of pixel arrays 410a and 410b, and the read image is read by CIPU 400 during VBI for the other pixel arrays of sensor module 415. It is processed.

  The VBI of the sensor module 415 for each pixel array can be set to a specific value. However, in some embodiments, the VBI is not set to a value less than the time it takes for the CIPU 400 to read and process one image. In some embodiments, the VBI of sensor module 415 for each pixel array is set to the same value. For example, when the VBI of the sensor module 415 for each pixel array is set to the same value, the sensor module 415 reads an image alternately from the pixel arrays 410a and 410b. In another embodiment, the VBI of the sensor module 415 for each pixel array is set to a different value. In some of such embodiments, the VBI of sensor module 415 for one pixel array is set to a multiple of the VBI of sensor module 415 for the other pixel array. For example, the VBI of the sensor module 415 for one pixel array is set to 2 unit times, and the VBI of the sensor module 415 for the other pixel array is set to 4 unit times. In this example, each time the sensor module 415 reads one image from the other pixel array, the sensor module 415 reads two images from one pixel array.

  FIG. 6 conceptually illustrates examples of different interleaved frame rates 605, 610, and 615 based on different VBIs. FIG. 6 is similar to FIG. 5, except that FIG. 6 includes thirteen times 625-685 along timeline 620. Further, the image of the person with the guitar represents the time at which the image is read from one of the pixel arrays along the timeline 620, while the image of the person wearing the college cap (ie, the mortarboard) is shown on the timeline 620. Along with the time when the image is read from the other pixel array.

  For purposes of illustration, it is assumed that the image of the person with the guitar was captured by the camera sensor 405a of the dual camera mobile device, and the image of the person wearing the college cap was captured by the camera sensor 405b of the dual camera mobile device. It is assumed that 6 will now be described with respect to the sensor module 415 and the pixel arrays 410a and 410b of FIG.

  At the exemplary interleaved frame rate 605, the VBI of the sensor module 415 for both pixel array 410a and pixel array 410b is set to two unit times. As shown in the example interleave frame rate 605, the sensor module 415 reads an image from the pixel array 410a along the timeline 620 at times 625, 635, 645, 655, 665, 675, and 685, and Reads an image from the pixel array 410b along the timeline 620 at times 630, 640, 650, 660, 670, and 680. That is, the sensor module 415 alternately reads an image from the pixel array every unit time.

  Exemplary interleave frame rate 610 is similar to exemplary interleave frame rate 605, except that the VBI of sensor module 415 for both pixel array 410a and pixel array 410b is set to 4 unit times. The exemplary interleaved frame rate 610 is such that the sensor module 415 reads an image from the pixel array 410a along the timeline 620 at times 625, 645, 665, and 685, and the sensor module 415 determines the times 635, 655 along the timeline 620. And 675 indicate that an image is read from the pixel array 410b. The exemplary interleaved frame rate 610 has a higher VBI than the exemplary interleaved frame rate 605 VBI, so the exemplary interleaved frame rate 610 has a lower frame rate than the exemplary interleaved frame rate 605.

  The exemplary interleave frame rate 615 is also similar to the exemplary interleave frame rate 605, except that the VBI of the sensor module 415 for both pixel array 410a and pixel array 410b is set to 6 unit times. As shown in FIG. 6, the sensor module 415 reads an image from the pixel array 410a at times 625, 655, and 685 along the timeline 620, and the sensor module 415 reads pixels at times 640 and 670 along the timeline 620. Read an image from array 410b. The VBI of exemplary interleave frame rates 615 is greater than the VBI of exemplary interleave frame rates 605 and 610. Accordingly, the frame rate of the exemplary interleaved frame rate 615 is less than the exemplary interleaved frame rates 605 and 610.

B. Multiple Pipelines FIG. 7 conceptually illustrates a captured image processing unit (CIPU) 700 in some embodiments. CIPU 700 performs the same functions as CIPU 400 described above, except that CIPU 700 is implemented by two front-end processing pipelines, storage and back-end processing pipelines instead of a single processing pipeline. Therefore, the description of the function of CIPU 700 will be described with reference to the module of CIPU 400.

  As shown, CIPU 700 includes a front-end processing pipeline 715 for camera sensor 405a and pixel array 410a, a front-end processing pipeline 720 for camera sensor 405b and pixel array 410b, a storage device 725, and a controller 725. It includes a module 730 and a back-end processing pipeline 735. In some embodiments, camera sensors 405a and 405b are sensors of a camera of a dual camera portable device.

  The front-end processing pipelines 715 and 720 in some embodiments perform a portion of the image processing of the CIPU 400. Thus, different embodiments may include different numbers of modules of CIPU 400. For example, each of the front-end processing pipelines 715 and 720 of some embodiments includes a CIPU 400 sensor module 415, a BPC module 420, an LS module 425, a demosaicing module 430, a WB module 435, and a statistics engine. 465.

  The front-end processing pipelines 715 and 720 perform the same type of image processing by having the same modules, but each module of the front-end processing pipelines 715 and 720 has different registers as described above with respect to CIPU 400. It can be configured differently depending on the value. Further, since each of the camera sensors 405a and 405b has its own front-end processing pipeline, the front-end processing pipelines 715 and 720 can process images independently of each other. For example, front-end processing pipelines 715 and 720 can process images in parallel (ie, simultaneously) at different times and at different speeds.

  In some embodiments, each of the front-end processing pipelines 715 and 720 can read an image from its corresponding camera sensor and pixel array. For example, front-end processing pipeline 715 reads an image captured by camera sensor 405a from pixel array 410a, and front-end processing pipeline 720 receives an image captured by camera sensor 405b from pixel array 410b. When one of the front-end processing pipelines 715 and 720 reads an image from its corresponding camera sensor and pixel array, the front-end processing pipeline processes the image and stores the processed image in storage 725. send. Similarly, each of the front-end processing pipelines 715 and 720 communicate with the controller module 730 as described above (eg, through the statistics engine of each front-end processing pipeline).

  The storage 725 of some embodiments stores the partially processed images for the back-end processing pipeline 735 to complete processing. In these embodiments, storage 725 receives the partially processed images from front-end processing pipelines 715 and 720 and sends the partially processed images to back-end processing pipeline 735. In some embodiments, storage 725 is implemented as volatile storage (eg, random access memory (RAM)), while in other embodiments, storage 725 is non-volatile (eg, flash memory). , Hard disk, optical disk, etc.). Further, the storage device 725 of some embodiments is an internal storage device (for example, a RAM), whereas the storage device 725 of another embodiment is an external storage device (for example, a compact flash (registered trademark) (CF)). Card, secure digital (SD) card, etc.).

  Some embodiments of the back-end processing pipeline 735 perform part of the image processing of the CIPU 700. In some embodiments, back-end processing pipeline 735 includes modules of CIPU 400 that do not include front-end processing pipelines 715 and 720. For example, referring to the above example, back-end processing pipeline 735 would include CSC module 445, gamma module 440, HSC module 450, scaler module 455, and filter module 460 of CIPU 400. Therefore, the back-end processing pipeline 735 of such an embodiment performs the remaining image processing of the CIPU 400 that is not performed by the front-end processing pipelines 715 and 720. Accordingly, the back-end processing pipeline 735 reads the partially processed image from the storage device 725 and performs the remaining image processing on the partially processed image. After processing the image, the back-end processing pipeline 735 sends the processed image to the CIPU driver 480.

  Controller module 730 performs the same functions described above with respect to FIG. As shown in FIG. 7, controller module 730 interacts with front-end processing pipelines 715 and 720 and back-end processing pipeline 735. In some embodiments, controller module 730 is included in back-end processing pipeline 735, while in other embodiments, controller module 730 is included in one of front-end processing pipelines 715 and 720. .

  Here, the operation of the CIPU 700 will be described with respect to the camera sensors 405a and 405b, the pixel arrays 410a and 410b, the front-end processing pipelines 715 and 720, the storage device 725, and the back-end processing pipeline 735 shown in FIG. When one of the front-end processing pipelines 715 and 720 reads an image from its corresponding camera sensor and pixel array, that front-end processing pipeline processes the image and stores the partially processed image. Send to device 725. For example, front-end processing pipeline 715 may read images captured by camera sensor 405a from pixel array 410a, or front-end processing pipeline 720 may store images captured by camera sensor 405b in pixel array 410b. May be read. As described above, each front-end processing pipeline 715 and 720 can process images in parallel.

  Back-end processing pipeline 735 reads the partially processed image from storage 725 and processes the partially processed image to complete image processing of the image. In some embodiments, the back-end processing pipeline 735 reads and processes images stored in the storage device 725 on a first-in, first-out basis. In other words, a particular image in storage device 725 is processed after all images received before and stored in storage device 725, but this particular image is processed after the particular image. It is processed before the images received and stored in the storage device 725. After processing the image, the back-end processing pipeline 735 sends the processed image to the CIPU driver 480.

  FIG. 8 conceptually illustrates another captured image processing unit (CIPU) 800 of some embodiments. CIPU 800 performs the same functions as CIPU 400 described above, except that CIPU 800 is implemented by two separate processing pipelines and each camera sensor has its own separate processing pipeline. Therefore, the function of CIPU 800 will be described with reference to the module of CIPU 400.

  As shown, CIPU 800 includes a processing pipeline 815 for camera sensor 405a and pixel array 410a, and a processing pipeline 820 for camera sensor 405b and pixel array 410b. Each of the processing pipelines 815 and 820 of some embodiments includes all modules included in the CIPU 400. Therefore, the operation of each of the processing pipelines 815 and 820 of these embodiments is the same as the operation of the CIPU 400.

  Since each of the camera sensors 405a and 405b has its own processing pipeline, the processing pipelines 815 and 820 can process images independently of each other. For example, processing pipelines 815 and 820 can process images in parallel (ie, simultaneously) at different times and at different speeds. Further, each of the processing pipelines 815 and 820 of some embodiments may be configured differently through different register values, as described above with reference to CIPU 400.

  In some embodiments, some modules of CIPU 400 include one or more line / frame buffers to perform some or all of the operations of the modules. For example, the filtering module 460 of some embodiments is implemented to perform a 3x3 low pass filter. In such an embodiment, the 3x3 low-pass filter processes three consecutive lines in the image such that a 3x3 low-pass filter is applied to the center line of the three consecutive lines. Thus, the filtering module 460 of such an embodiment requires at least three line / frame buffers to perform a 3x3 low pass filter. Other modules of CIPU 400 also include one or more line / frame buffers, such as, for example, BPC module 420 and LS module 425.

  Each of the processing pipelines of CIPU 800 may have different line / frame buffer sizes to customize the image processing to the characteristics of its corresponding camera. For example, if one camera of a dual camera portable device has a 2048 × 1500 pixel sensor, the processing pipeline of the 2048 × 1500 pixel sensor can include a line / frame buffer that is 2048 pixels wide. Similarly, if the other camera of the dual camera portable device has a 640x480 pixel sensor, the processing pipeline of the 640x480 pixel sensor can include a line / frame buffer that is 640 pixels wide. That is, the size of a line / frame buffer included in a module of one processing pipeline may be different from the size of a line / frame buffer included in a module of another processing pipeline.

III. Video Conference A. Video Conferencing Architecture FIG. 9 conceptually illustrates a software architecture for a video conferencing and processing module 900 of a dual camera portable device of some embodiments. Video conferencing and processing module 900 includes a CIPU driver 905, which is similar to corresponding modules and drivers 305, 310 and 320 described above with respect to FIG. 3, a media exchange module 910, and an encoder driver 920. The video conferencing and processing module 900 also includes a video conferencing module 925, a video conferencing client 945, and a network interface 950 for performing various video conferencing functions. Like the video processing and encoding module 300, the video conferencing and processing module 900 processes and encodes the images captured from the cameras of the dual camera portable device.

  As described above with respect to FIG. 3, by using the media exchange module 910, consumers and producers of the media content in the device exchange media content and instructions for processing the media content, and the CIPU driver 905 allows the The encoder driver 920 functions as a communication interface with the processed image processing unit (CIPU) 955, and functions as a communication interface with the encoder hardware 960 (for example, an encoder chip, an encoding component on a system-on-chip, and the like).

  The video conferencing module 925 of some embodiments handles various video conferencing functions, such as image processing, video conferencing management, and networking. As shown, video conferencing module 925 interacts with media exchange module 910, video conferencing client 945, and network interface 950. In some embodiments, video conferencing module 925 receives instructions from and sends instructions to video conferencing client 945. In some embodiments, the video conferencing module 925 may also connect to a network (eg, a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a network of networks, a code division Transmit data to and receive data from multiple access (CDMA) networks, GSM networks, etc.).

  The video conference module 925 includes an image processing layer 930, a management layer 935, and a network layer 940. In some embodiments, the image processing layer 930 performs image processing operations on images of the video conference. For example, the image processing layer 930 of some embodiments performs exposure adjustment, image resizing, perspective correction, and dynamic range adjustment, as described in more detail below. The image processing layer 930 of some embodiments sends a request for an image from the CIPU 955 through the media exchange module 910.

  The management layer 935 of some embodiments controls the operation of the video conferencing module 925. For example, in some embodiments, management layer 935 initializes one camera / multiple cameras of a dual camera portable device, processes images and audio for transmission to a remote device, and receives images from the remote device. And process audio. In some embodiments, management layer 935 generates a composite (eg, PIP) display for the device. Further, management layer 935 may change the operation of video conference module 925 based on the networking report received from network layer 940.

  In some embodiments, network layer 940 performs some or all of the networking functions for video conferencing. For example, the network layer 940 of some embodiments establishes a network connection (not shown) between a video conference dual camera portable device and a remote device, among other functions, as described below. Transmitting the image to the remote device and receiving the image from the remote device. In addition, the network layer 940 receives and processes networking data, such as packet loss, one-way latency, and round-trip latency, among other types of data, and processes such data. The data is reported to the management layer 935.

  The video conferencing client 945 in some embodiments is an application that may use the video conferencing features of the video conferencing module 925, such as a video conferencing application, a voice over IP (VOIP) application (eg, Skype), or an instant messaging application. is there. In some embodiments, video conference client 945 is a stand-alone application, while in other embodiments, video conference client 945 is integrated into another application.

  In some embodiments, the network interface 950 is configured such that the video conferencing module 925 and the video conferencing client 945 are connected via a network (eg, a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.) through the network interface 950. A communication interface for transmitting data and receiving data. For example, if video conferencing module 925 desires to transmit data (eg, images captured by a camera of a dual camera portable device) to another device over the Internet, video conferencing module 925 may transmit data to other devices via network interface 950. Send images to other devices.

B. Video Conference Setup FIG. 10 conceptually illustrates an exemplary video conference request message sequence 1000 of some embodiments. The figure shows a video conference request message sequence 1000 between a video conference client 1010 running on device 1005, a video conference server 1015, and a video conference client 1025 running on device 1020. In some embodiments, video conference clients 1010 and 1025 are the same as video conference client 945 shown in FIG. As shown in FIG. 10, one device (ie, device 1005) requests a video conference and another device (ie, device 1020) responds to such a request. The dual camera portable device described in the present application can perform both operations (ie, make a request and respond to the request).

  In some embodiments, the video conference server 1015 routes messages between multiple video conference clients. In some embodiments, videoconferencing server 1015 is implemented on one computing device, while in other embodiments, videoconferencing server 1015 is implemented on multiple computing devices. In some embodiments, the video conferencing server is a publicly accessible server that can handle and route messages for many conferences at once. Each of the video conferencing clients 1010 and 1025 of some embodiments is connected via a network (eg, a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.) through a network interface such as the network interface 950 described above. It communicates with the video conference server 1015.

  The video conference request message sequence 1000 of some embodiments starts when the video conference client 1010 receives a request to start a video conference with the device 1020 from the user of the device 1005 (operation 1). The video conference client 1010 of some embodiments receives a video conference start request when a user of the device 1005 selects a user interface (UI) item of a user interface displayed on the device 1005. Examples of such a user interface are shown in FIGS. 11 to 14 described below.

  After the video conference client 1010 receives the request, the video conference client 1010 sends (in operation 2) a video conference request indicating the device 1020 as a recipient based on input from the user. Video conference server 1015 forwards the video conference request to video conference client 1025 of device 1020 (operation 3). In some embodiments, video conference server 1015 forwards the video conference request to video conference client 1025 using push technology. That is, instead of waiting for the client 1025 to send some message request, the video conference server 1015 starts transmitting the video conference request to the video conference client 1025 upon receiving from the video conference client 1010.

  If video conference client 1025 of some embodiments receives the video conference request, it indicates to the user of device 1020 that the user of device 1005 has transmitted the video conference start request and accepts or rejects the video conference request. A user interface is displayed on the device 1020 to confirm with the user of the device 1020 what to do. An example of such a user interface is shown in FIG. 12 described below. In some embodiments, if videoconferencing client 1025 receives a videoconferencing request acceptance request from the user of device 1005 (at operation 4), videoconferencing client 1025 may request videoconferencing consent from videoconferencing server 1015 (operation 4). 5) Send. Video conference client 1025 of some embodiments receives a video conference request acceptance request, for example, when the user of device 1020 selects a user interface item of the user interface shown in FIG.

  After the video conference server 1015 receives the video conference agreement from the video conference client 1025, the video conference server 1015 forwards the video conference agreement to the video conference client 1010 (operation 6). Some embodiments of the video conferencing server 1015 transfer video consent to the video conferencing client 1010 using push technology as described above.

  Upon receiving the videoconference acceptance, in some embodiments, a videoconference is established between device 1005 and device 1020 (operation 7). In different embodiments, the video conference is established differently. For example, video conference establishment in some embodiments includes negotiating a connection between device 1005 and device 1020, determining a bit rate for encoding video, and exchanging video between device 1005 and device 1020.

  In the above example, the user of device 1020 accepts the videoconference request. In some embodiments, device 1020 may be configured to automatically accept incoming video conference requests without displaying a UI (e.g., via device preference settings). Further, the user of device 1020 may reject the video conference request (at operation 4) (e.g., by selecting a user interface item of a user interface displayed on device 1020). Instead of sending a videoconference acceptance, videoconference client 1025 sends a videoconference reject to videoconference server 1015, which forwards the videoconference rejection to videoconference client 1010. In that case, no video conference is established.

1. Video Conference Setup User Interface In some embodiments, a video conference is initiated based on an ongoing call. That is, while the user of the mobile device is engaged in a call with the second user, the user can turn the call into a video conference with the permission of the other party. In some embodiments of the present invention, FIG. 11 illustrates the initiation of such a video conference by a dual camera handheld portable device 1100. This figure shows the start of a video conference as five operational phases 1110, 1115, 1120, 1125 and 1130 of a user interface ("UI") 1105 of the device 1100.

  As shown in FIG. 11, the UI 1105 includes a name field 1135, a selection menu 1140, and a selectable UI item 1145. The name field 1135 displays the name of the person on the other end of the call for which the user may wish to request a video conference. In this example, selectable UI item 1145 (which can be implemented as a selectable button) provides a "end call" option that the user can select to end the call. The selection menu 1140 displays a menu of selectable UI items such as a speakerphone item 1142, a mute item 1144, a keypad item 1146, a telephone directory item 1148, a hold item 1152, and a video conference item 1154. Different embodiments display the selection menu differently. In the embodiment illustrated by FIG. 11, the selection menu 1140 includes a number of icons of equal size, each representing a different action. Other embodiments provide a scrollable menu or prioritize certain items (eg, by making the items larger).

  The operation of the UI 1105 will now be described with respect to the state of this UI in the five stages 1110, 1115, 1120, 1125 and 1130 shown in FIG. In a first stage 1110, a call is established between a handheld portable device user and Nancy Jones. The second step 1115 displays the UI 1105 after the user has selected a video conferencing option 1154 (eg, through a single finger tap with the finger 1150) to activate the video conferencing tool. In this example, a video conference option 1154 (which can be implemented as a selectable icon) allows a user to initiate a video conference during a call. In the second stage, the video conferencing option 1154 is highlighted to indicate that the video conferencing tool has been activated. Different embodiments may indicate such a selection in a different manner (eg, by highlighting the border or text of an item).

  The third stage 1120 displays the UI 1105 after the device 1100 has started the video conference process by selecting the video conference option 1154. The third stage is a transition waiting stage during which the device waits for a video conference to be established (eg, while the device waits for the device at the other end of the call to accept or reject the video conference). In a third stage 1120, the user of the device 1100 can still talk to the user of the other device (ie, Nancy Jones) while the video conference connection is established. Further, according to some embodiments, the user of device 1100 may select a selectable UI item (not shown) displayed on UI 1105 to cancel the video conferencing request, thereby causing a video conferencing in a third step 1120. Request can be canceled. During this wait phase, different embodiments use different displays on the UI 1105 to indicate the wait state.

  As shown in FIG. 11, in some embodiments, the third stage wait state is shown as a full screen display of the video captured by device 1100, with a "preview" notation at the bottom of the video. Specifically, in FIG. 11, the third stage 1120 indicates the start of the video conference process by displaying a full screen presentation of the video captured by the device camera in the display area 1160 of the UI 1105. In some embodiments, the front camera is the default camera selected by the device at the start of the video conference. In many cases, this front camera points to the user of the device at the start of the video conference. Thus, in the example shown in FIG. 11, the third stage 1120 shows the device 1100 as presenting a full screen video of the user of the device 1100. The waiting state of the device is further highlighted by a “preview” indicator 1165 below the video that appears in the display area 1160 during the third stage 1120.

  The third waiting phase 1120 of the transition may be represented differently in some embodiments. For example, according to some embodiments, a user of device 1100 may select a rear camera as the camera for initiating a video conference. To enable this selection, some embodiments allow the user to specify the rear camera (eg, via menu preferences) as the default camera for initiating a video conference, and / or After selecting the video conference option 1154, the rear camera can be selected from a menu that displays the rear camera and the front camera. In any of these situations, UI 1105 (eg, display area 1160) displays video captured by the rear camera during third standby phase 1120.

  Similarly, other embodiments provide a message that highlights a device's wait state by displaying a still image stored on device 1100 by displaying a small version of the video captured by device 1100. By doing so (e.g., by indicating "meeting is being established"), the activation of the video conferencing tool can be indicated, such as by not displaying a "preview" sign. Similarly, in a third stage 1120, the UI 1105 of some embodiments may indicate that the user does not enter the video conference at this stage (eg, while the user is waiting for the remote user to respond to the request). If so, an end button (not shown) is provided to allow you to cancel the video conference and return to the call.

  Fourth stage 1125 shows UI 1105 in a transition state after the remote user has accepted the video conference request and a video conference connection has been established. In this transition state, as shown by arrow 1175, the size of display area 1160 displaying the local user's video (captured in this example by the front camera) decreases stepwise (ie, stepwise). Shrink). The display area 1160 (i.e., the video of the local user) shrinks, so that the UI 1105 can display a display area 1170 (e.g., display window 1170) containing video from the camera of the remote device behind the display area 1160. In other words, the shrinking of the local user's video 1160 generates a PIP display 1180 having a foreground insert display 1160 of the local user's video and a background main display 1170 of the remote user. In this example, the background main display 1170 may be a woman being captured by the front camera of the remote device (eg, Nancy Jones who is a user of the remote device) or a woman being captured by the rear camera of the remote device (eg, , A woman whose video has been captured by Nancy Jones). It is noted that the fourth stage of the transition shown in FIG. 11 is only one exemplary approach used by some embodiments, and that other embodiments can animate the fourth stage of the transition differently. Those skilled in the art will appreciate.

  The fourth step 1125 also shows a selectable UI item 1132 in the lower display area 1155. A selectable UI item 1132 (which may be implemented as a selectable button) provides a selectable conference end option 1132 below the PIP display 1180. The user can select this end conference option 1132 (eg, through a single finger tap) to end the video conference. According to different embodiments, the user may end the meeting in different ways, such as by flipping a switch on the mobile device, giving a voice command, and the like. Further, different embodiments may allow the end-of-conference option 1132 to disappear during a video conference, whereby the PIP display 1180 may occupy the entire display area 1185. Thereafter, a single finger tap at the bottom of the display area 1185 causes the end conference option 1132 to be redisplayed, providing access to the end conference option 1132 to the user. In some embodiments, the layout of display area 1155 is the same as display area 1155 described in more detail below.

  The fifth stage 1130 shows the UI 1105 after the animation of the fourth transition state 1125 has ended. Specifically, the fifth stage 1130 shows a PIP display 1180 presented by the UI 1105 during a video conference. As described above, this PIP display 1180 includes two video displays, a larger background display 1170 from the remote camera and a smaller foreground insert display 1160 from the local camera.

  This PIP display 1180 is just one way to present a composite view of the video being captured by the remote and local devices. In addition to this composite view, the devices of some embodiments provide other composite views. For example, instead of having a larger background display 1170 for the remote user, the larger background display 1170 can be for the local user and the smaller foreground insert display 1160 can be for the remote user. As described further below, some embodiments allow a user to switch between a local camera and / or a remote camera as a camera for the inset view and main view of the PIP display 1180 during a videoconference.

  Similarly, according to some embodiments, local video and remote video are displayed in two side-by-side display areas (eg, left and right display windows or top and bottom display windows) or two diagonally aligned display areas in UI 1105. Can be displayed. As described further below, the manner of PIP display or default display mode may be specified by the user in some embodiments through device preferences or through user-selectable controls during a video conference.

  If the user of device 1100 of FIG. 11 invites a remote user to a video conference, the remote user may accept or reject the invitation. FIG. 12 shows the UI 1205 of the remote user's device 1200 in six different stages 1210, 1215, 1220, 1225, 1230 and 1235 showing a sequence of operations for presenting and accepting a video conference invitation at the remote user's device. The following description of UI 1205 will refer to the user of device 1200 (ie, the device that receives the video conference request) as the invitation recipient, and refer to the user of device 1100 (ie, the device that transmits the video conference request) as the invitation requester. . Also, in this example, the invitation recipient device 1200 is assumed to be a dual camera device, like the invitation requester device. However, in other examples, one or both of these devices are single camera devices.

  The first stage 1210 shows the UI 1205 when the invitation recipient receives an invitation to a video conference from the invitation requester John Smith. As shown in FIG. 12, the UI 1205 at this stage includes a name field 1235, a message field 1240, and two selectable UI items 1245 and 1250. The name field 1235 displays the name of the person requesting the video conference. In some embodiments, the name field 1235 displays the person's telephone number instead of the name of the person requesting the video conference. In the message field 1240, an invitation from the invitation requester to the invitation recipient is displayed. In this example, the field 1240 "Video Invite" indicates that the invitation requester has requested a video conference with the invitation recipient. Selectable UI items 1245 and 1250 (which may be implemented as selectable buttons) provide selectable request reject options 1245 and request accept options 1250 for use by the invitation recipient to reject or accept the invitation. . In different embodiments, these options may be displayed differently and / or other options may be displayed.

  When the invitation recipient sees the "Video Invitation" notation displayed in message field 1240, the invitee rejects or accepts the request by selecting the request rejection option 1245 or the request acceptance option 1250 in the UI, respectively. can do. The second stage 1215 indicates that the user selects the request acceptance option 1250 in the example shown in FIG. In this example, the selection is made by the user's finger tapping on request acceptance option 1250, which is indicated by the highlighting of option 1250. In some embodiments, other techniques (e.g., double-tap, etc.) to select the accept request option 1245 or reject request option 1250 to indicate a selection (e.g., highlight the border or text of a UI item). Provided.

  The third stage 1220 shows the UI 1205 after the invitation recipient has agreed to join the video conference. At this stage, the UI 1205 enters a preview mode that shows a full screen presentation of the video from the front camera of the remote device in the display area 1244. The front camera is in this case aimed at the user of the remote device (ie, Nancy Jones in this example). Thus, in this preview mode, her image is shown. This preview mode allows the invitee to ensure that their video is properly displayed and satisfied with their appearance before the video conference begins (eg, before the actual transmission of the video begins). In some embodiments, a notation, such as a “Preview” notation, may be displayed below display area 1244 to indicate that the invitation recipient is in preview mode.

  As described further below, some embodiments allow an invitee to select a rear camera as a default camera when a video conference is disclosed, or to select a front camera or a rear camera at the beginning of a video conference. . Also, in other embodiments, the preview display of the invitation recipient is displayed differently (eg, on the smaller image located in the corner of display area 1244). In yet another embodiment, this preview mode is not included, and the video conference starts immediately after the invitation recipient accepts the request.

  In the third stage, the UI 1205 shows two selectable UI items 1275 and 1246, one of which overlaps the display area 1244 and the other is below the display area 1244. Selectable UI item 1275 is an accept button 1275 that the user may select to start a video conference. The selectable UI item 1246 is an end button 1246 that can be selected when the invitation recipient decides not to participate in the video conference at this stage.

  The fourth step 1225 shows the UI 1205 after the invitation recipient has selected the Accept button 1275. In this example, the accept button 1275 is highlighted to indicate that the invited recipient is ready to start a video conference. Such a selection may be indicated differently in other embodiments.

  Fifth stage 1230 shows UI 1205, which is the transition state after the invitation recipient has accepted the video conference request. In this transition phase, as indicated by arrow 1260, the size of display area 1244 displaying the video of the invited recipient (in this example, captured by the front camera) is stepwise reduced (ie, stepped). To shrink). The invitation recipient's video shrinks so that the UI 1205 can display a display area 1265 (eg, display window 1265) that includes the video from the invitation requester's camera behind display area 1244. In other words, the shrinking of the invitation recipient's video generates a PIP display 1280 having a foreground insertion display area 1244 of the invitation recipient's video and a background main display 1265 of the invitation requester.

  In this example, the background main display 1265 presents a male video (ie, John Smith who is the user of the local device 1100) whose video has been captured by the front camera of the local device. In another example, the video can be a video of a male whose video is being captured by the rear camera of the local device (eg, a male whose video is being captured by John Smith). In different embodiments, the fifth phase of this transition can be animated differently.

  The UI in the fifth step 1230 includes a selectable UI item 1285 (eg, a mute button 1285) for muting other users' voices during the video conference, and a UI for terminating the video conference, as described in further detail below. A display area 1155 (eg, a toolbar or menu bar) including a selectable UI item 1287 (eg, a conference end button 1287) and a selectable UI item 1289 (eg, a camera switch button 1289) for switching cameras is also displayed. I do. Thus, the invitation recipient can select any of the selectable UI items 1285 to 1289 (e.g., by a single finger tap) to perform the desired action during the video conference. Different embodiments allow the invitation recipient to perform any of the different ways of operation, such as, for example, by flipping a switch on the mobile device, giving a voice command, and so on.

  Although FIG. 12 illustrates an exemplary layout of the display area 1155, some embodiments include the display area 1155 of FIG. A different layout of the display area 1155 such as the layout of the above is provided. Other layouts of display area 1155 may include any number of different selectable UI items for performing different functions. Further, a fifth step 1230 shows a display area 1155 displayed at the bottom of the UI 1205. Different embodiments of the display area 1155 may be displayed at different locations within the UI 1205 and / or may be defined as different shapes.

  FIG. 12 shows display area 1155 as a static display area (ie, display area 1155 is always displayed). However, in some embodiments, display area 1155 is a dynamic display area. In some such embodiments, display area 1155 is not normally displayed. Rather, display area 1155 is displayed when a triggering event (e.g., a user's selection of such a single tap on display area 1280, a voice command, etc.) is received. After a user selection is received (eg, selectable mute UI item 1285) or after a defined amount of time (eg, 3 seconds), display area 1155 is hidden, and this amount of time is It can be specified by the user through the preferences of the mobile device or videoconferencing application. In some such embodiments, display area 1155 is automatically displayed after a video conference has begun and is hidden in the same manner described above.

  The sixth step 1235 shows the UI 1205 after the animation of the fifth transition step has been completed. Specifically, the sixth stage shows a PIP display 1280 presented by the UI 1205 during a video conference. As described above, this PIP display 1280 includes two video displays, a larger background display 1265 from the local camera and a smaller foreground insert display 1244 from the remote camera. This PIP display 1280 is just one way to present a composite view of the video being captured by the remote and local devices. In addition to this composite view, the devices of some embodiments provide other composite views. For example, instead of having a larger background display for the invitation recipient, the larger background display can be for the invitation requester and the smaller foreground insertion display can be for the invitation recipient. As described further below, some embodiments allow a user to control the inset view and main view of the PIP display to switchably display local and remote cameras. Similarly, according to some embodiments, local video and remote video are displayed in two side-by-side display areas (eg, left and right display windows or top and bottom display windows) or two diagonally aligned display areas in UI 1205. Can be displayed. As described further below, the manner of PIP display or default display mode may be specified by the user in some embodiments through device preferences or through user-selectable controls during a video conference.

  Although FIG. 12 illustrates the sequence of operations for presenting and accepting a video conference invitation as six different stages of operation, in some embodiments, the operations may be performed in a smaller number of stages. For example, in some of such embodiments, presenting the third step 1220 and the fourth step 1225 may be omitted, and the second step 1215 to the fifth step 1215 may be performed after the user selects the request acceptance option 1250. Proceed to 1230. In other embodiments that implement the operation (ie, present and accept the videoconference invitation) in fewer steps, the first step 1210 and the second step 1215 may be omitted and the invitation recipient The third step 1220 can be presented to the user when an invitation to a video conference is received from the user.

  FIG. 13 shows fewer stages in FIG. 12 by combining the first and third stages into one stage and combining the second and fourth stages into one stage. An example of performing an operation is shown. Specifically, this figure shows the UI 1205 of the remote user's device 1200 in five different stages 1390, 1392, 1394, 1230 and 1235. The first step 1390 is similar to step 1110, except that the name field 1235 displays the name "John Smith" to indicate the name of the other party in the call. That is, a call has been established between the user of the remote portable device and the user of the local device (ie, John Smith in this example). The second and third stages 1392 and 1394 are similar to the second and third stages of FIG. 12 except that the second and third stages 1394 and 1394 also show a preview of the user of the remote portable device (ie, Nancy Jones in this example). It is the same as the first step 1210 and the second step 1215. The fourth step 1230 and the fifth step 1235 are the same as the fifth step 1230 and the sixth step 1235 of FIG.

  In addition to launching the video conferencing tool through options available during a call, some embodiments allow a user of a dual camera device to start a video conference directly without having to first make a call. FIG. 14 illustrates another such alternative method of initiating a video conference. The figure shows a UI 1405 in seven different stages 1410, 1415, 1420, 1425, 1430, 1435 and 1440, which show an alternative sequence of operations for initiating a video conference.

  In a first step 1410, the user is searching the contact list of this mobile device for the person he wants to be involved in a video conference in a similar manner to how to find a contact to call. In a second step 1415, the user selects the person 1455 that the user wants to have a video conference with (eg, via a single finger tap 1460 to the person's name 1455). This selection triggers a UI 1405 displaying contact information and various user selectable options. In this example, Johnson's name 1455 is highlighted, indicating that the user is the one who wants to have a video conference. Different embodiments may indicate such a choice in different ways. In a second step 1415, the user of the device 1400 can select through the contact list who the user wants to have a video conference with, but according to some embodiments, the user may have recently created a video conference or call with the user of the device 1400. A person can be selected through a "Recent" call history listing the specific number or name of the person who made the call.

  In a third step 1420, the UI 1405 displays information 1462 for the selected person and various selectable UI items 1468, 1472 and 1470 after the person's name 1455 has been selected. In this example, one of the various selectable UI items 1472 (which may be implemented as selectable icons or buttons) provides a video conferencing tool. Using the video conferencing option 1472, the user can invite the person identified by the contact 1466 to the video conference. In different embodiments, information 1462 and selectable UI items 1468, 1472, and 1470 are displayed differently (eg, in different configurations).

  A fourth step 1425 indicates that the user selects videoconferencing option 1472 (eg, through a single finger tap). In this example, video conferencing option 1472 is highlighted to indicate that video conferencing tool 1472 has been activated. Such a selection may be indicated differently in different embodiments (eg, by highlighting the text or border of the selected icon).

  The fifth step 1430, the sixth step 1435 and the seventh step 1440 are similar to the third step 1120, the fourth step 1125 and the fifth step 1130 shown in FIG. 11, and by referring to the description of these steps, Can be understood. In short, the fifth step 1430 represents a transition waiting step to wait for the remote user to respond to the video conference invitation. The sixth step 1435 is that after the remote user has accepted the videoconference request, the size of the viewing area 1480 (which displays the local user's video) is gradually reduced, so that the UI 1405 displays the video from the remote user's camera. Indicates that the containing display area 1492 can be shown behind the display area 1480. In a seventh step 1440, a PIP display 1447 is presented by the UI 1405 during the video conference. In some embodiments, the layout of the display area 1155 in the sixth step 1435 and the seventh step 1440 is similar to the layout of the display area 1155 in FIG. 12 described above.

  FIGS. 10, 11, 12, 13 and 14 show several ways to establish a video conference. In some embodiments, during a call, audio data (eg, voice) is transmitted over one communication channel (via a communication network such as a circuit-switched communication network or a packet-switched communication network), and during a videoconference. The audio data is transmitted over another communication channel. Thus, in such embodiments, audio data (eg, voice) is transmitted over the communication channel before the video conference is established, and when the video conference is established, the audio is transmitted over the communication channel used during the call. (Instead of).

  To provide a seamless transition of audio data from a call to a video conference (eg, handoff), some embodiments do not end the call before establishing the video conference. For example, in some embodiments, a peer-to-peer video conference connection is established (eg, after completing the message sequence shown in FIG. 10) before ending the call and transmitting audio / video data through the peer-to-peer communication session. Begin to. Alternatively, in other embodiments, a peer-to-peer videoconferencing connection is established (eg, after completing the message sequence shown in FIG. 10), and audio / video data is transmitted through the peer-to-peer videoconferencing session before ending the call. And start presenting the received audio / video data.

  The peer-to-peer videoconferencing connection of some embodiments allows mobile devices in a videoconference to communicate directly with each other (eg, instead of communicating through a central server). With some embodiments of peer-to-peer videoconferencing, mobile devices in a videoconference may share resources with each other. For example, as described in more detail below, through a video conference control communication channel, one portable device processes images differently, such as exposure adjustment operations, focus adjustment operations, and / or camera switching operations ( That is, by transmitting instructions from one portable device to another to direct the other portable device to share its image processing resources), the operation of another portable device during a videoconference can be remotely controlled. Can control.

2. Dynamic Bit Rate Setup Typically, mobile devices during video conferencing are connected to various types of communication networks, such as various private and public wireless communication networks (eg, cellular networks such as GSM, UMTS, etc.). Data (eg, audio and video images) communicate with one another over a communication channel. Hereinafter, an example of such a wireless communication network will be described with reference to FIGS.

  Due to the change in the number of mobile devices accessing the communication network at a given time, the available bandwidth of the communication network for conducting video conferences varies from time to time. Available bandwidth can change even during a video conference. Furthermore, flooding the communication network with high bit rates or heavy signaling during a video conference in an attempt to track the optimal video conference bit rate is undesirable.

  For these reasons, some embodiments employ a novel method for identifying an optimal initial bit rate for a video conference. To identify the optimal initial bit rate for video conferencing, the method starts the video conferencing at a specific bit rate and, if a network condition that degrades the video conferencing quality is not detected in these embodiments, a specific interval. To increase the bit rate gradually.

  One example of such an embodiment is shown in FIG. This diagram conceptually illustrates a process 1500 of some embodiments of the invention for setting a bit rate for a video conference. Process 1500 may be performed as part of a video conferencing setup (eg, as shown in FIG. 10) to dynamically determine a bit rate for transmitting data (eg, audio and video images) based on various network conditions. (As part of establishing a video conference). In some embodiments, process 1500 is performed by management layer 935 of video conferencing module 925 described above with respect to FIG. A more detailed version of this video conferencing module is described below with respect to FIG.

  As shown in FIG. 15, the process 1500 begins (at 1505) by setting the bit rate to an initial bit rate. In some embodiments, the initial bit rate is the device's default baseline rate. However, in some embodiments, the user can specify the initial bit rate. At 1505, process 1500 also initiates a video conference transmitting data (eg, audio and video images) at an initial bit rate to a remote device over one or more communication channels.

  Next, the process 1500 identifies (at 1510) a set of network state parameters received from the remote device in the video conference. In some embodiments, the local device receives a set of network state parameters from a remote device through a Real-time Transport Protocol (RTP) communication session established at the start of the video conference. For example, some embodiments provide network state parameters through an extension of RTP. Further, the RTP extension in some embodiments indicates the presence of an extension header in the RTP packet header and defines any type of information (such as a set of network state parameters) by defining an extension header for additional information. Can be used to send.

  In different embodiments, devices in a videoconference relay different sets of network state / congestion parameters. In the embodiments described below, the network state parameter set includes one-way latency and bandwidth estimation bit rate. In another embodiment, the network state parameter set includes packet loss data and round trip time (RTT) delay data. Thus, different embodiments may include any number of different network state parameters in the network state parameter set.

  In some embodiments, the set of network state parameters received from the remote device of the video conference is a local mobile device (ie, a mobile device performing process 1500) during the video conference at the initial bit rate set in operation 1505. (E.g., audio and video) transmitted from a remote device to a remote device. For example, in some embodiments, the remote device may increase the one-way latency by calculating the time it takes for the audio packet to propagate from the local mobile device to the remote device over the network connection using the time stamp of the audio packet. Can decide. Specifically, in some embodiments, each audio packet is time stamped. Without packet delay, the remote device would receive audio packets at set intervals equal to the timestamp difference. However, if there is a one-way latency delay, the remote device receives audio packets at intervals greater than the time stamp difference.

  Also, in some embodiments, the remote device determines the bandwidth estimation bit rate by examining the time at which the video packets were received, the time at which consecutive video packets were received, and the size of the consecutive video packets. I do. That is, the difference between the reception times of two consecutive video packets and the size of the second video packet are used to estimate the available bandwidth of the network connection. In some embodiments, the bandwidth estimation bit rate is determined by examining multiple pairs of consecutive video packets. In the above example, a particular type of data is used (ie, audio data for determining one-way latency and video data for determining a bandwidth estimation bit rate). However, in some embodiments, other types of data communicated over a network connection between a local portable device and a remote device may also be used.

  After identifying the set of network conditions (at 1510), the process 1500 then determines (at 1515) whether the one-way latency has degraded beyond a defined threshold amount. In some embodiments, the threshold amount is defined as a specific amount of latency, and the one-way latency is defined as the threshold amount if the difference between the current one-way latency and the previous one-way latency exceeds the specific amount of latency. It is determined that it has deteriorated beyond that. In other embodiments, the threshold amount is defined as a specific rate of change in one-way latency. Therefore, it is determined that the one-way latency has exceeded the threshold amount when the rate of change of the set of one-way latencies (for example, the current one-way latency and the previous one-way latency) exceeds a specific rate of change.

  The process 1500 ends if the one-way latency is determined to degrade beyond the threshold amount. Otherwise, the process 1500 determines (at 1520) whether the current bit rate has reached the bandwidth estimate bit rate. In some embodiments, the bandwidth estimation bit rate indicates the amount of bandwidth available for the network connection (eg, 15 kilobits per second (kbps)). If the process 1500 determines that the current bit rate has exceeded the bandwidth estimation bit rate, the process 1500 ends. If the process 1500 determines that the current bit rate does not exceed the bandwidth estimate bit rate, the process 1500 proceeds to operation 1525.

  At 1525, process 1500 determines whether the current bit rate has reached a specified maximum bit rate. If the process 1500 determines that the current bit rate has exceeded the specified maximum bit rate, the process 1500 ends. Otherwise, the process 1500 proceeds to operation 1530, where the current bit rate is increased by a specified amount. Different embodiments define the amount by which the bit rate is increased differently. Examples of predefined amounts for increasing the current bit rate include 32 kbps, 64 kbps, among any number of other amounts for increasing the bit rate.

  Next, the process determines (at 1535) whether a specified amount of time has elapsed. This amount of time may be one second, two seconds, five seconds, or any other possible amount of time, as different embodiments define the amount of time differently. The remote device receives data (eg, audio and video images) transmitted from the local mobile device at the new increased bit rate (at operation 1530) and determines a network state parameter based on the new increased bit rate. To do so, the process 1500 waits for a specified amount of time to elapse. If the process 1500 determines that the specified amount of time has not elapsed, the process 1500 returns to operation 1535 until the specified amount of time has elapsed. If the process 1500 determines that the specified amount of time has elapsed, the process 1500 returns to operation 1510. Operation of process 1500 proceeds from 1510 as described above until process 1500 ends.

  When process 1500 ends (ie, after operations 1515, 1520 or 1525), the bit rate setup for the video conference has been completed and the optimal bit rate has been determined. Some embodiments are based on a set of network state parameters (ie, one-way latency and bandwidth estimation bit rate) received from the remote device, as the bandwidth available for the video conference may change during the video conference. Keep adjusting the bit rate. By increasing the bit rate, the bit rate can be adjusted during the video conference. For example, if the one-way latency has degraded beyond a specified threshold amount and the process 1500 ends and the one-way latency has improved during the video conference, in some embodiments the bit rate is increased. Similarly, if the bit rate has exceeded the bandwidth estimation bit rate, the process 1500 ends and during a video conference, if the bandwidth estimation bit rate increases, in some embodiments, the bit rate is increased. .

  In contrast, by reducing the bit rate, the bit rate may be adjusted during a video conference. For example, if one-way latency continues to degrade during a video conference beyond a specified threshold amount, in some embodiments, the bit rate is reduced. Also, if the bit rate continues to exceed the bandwidth estimate bit rate during the video conference (eg, the bandwidth estimate bit rate continues to decrease), in some embodiments, the bit rate is reduced.

  Further, the description of process 1500 uses the one-way latency and bandwidth estimation bit rate to determine whether to increase the bit rate. However, those skilled in the art will appreciate that in different embodiments, any number of network state parameters can be used to determine whether to increase the bit rate. For example, the decision to increase the bit rate can be based solely on RTT delay data or packet loss data.

C. Video Conferencing Architecture As described above, FIG. 16 conceptually illustrates a software architecture for a video conferencing and processing module 1600 of a dual camera portable device of some embodiments. As shown, the video conferencing and processing module 1600 includes a client application 1665, a video conferencing module 1602, a media exchange module 1620, a buffer 1625, a captured image processing unit (CIPU) driver 1630, and an encoder driver 1635. , A decoder driver 1640. In some embodiments, buffer 1625 is a frame buffer that stores images of the video for display on display 1645 of the dual camera portable device.

  In some embodiments, client application 1665 is the same as video conference client 945 of FIG. As described above, the client application 1665 may be integrated into another application or may be implemented as a stand-alone application. The client application 1665 may be an application that uses the video conferencing functionality of the video conferencing module 1602, such as a video conferencing application, a voice over IP (VOIP) application (eg, Skype), or an instant messaging application.

  In some embodiments, the client application 1665 sends instructions to the video conferencing module 1602, receives instructions from the video conferencing module 1602, and sends the instructions to the dual camera portable device, such as instructions to start and end the conference. From the user to the video conferencing module 1602 to generate a user interface that is displayed on the dual camera portable device and allows the user to interact with the application.

D. Video Conference Manager As shown in FIG. 16, the video conference module 1602 includes a video conference manager 1604, an image processing manager 1608, a networking manager 1614, and buffers 1606, 1610, 1612, 1616 and 1618. In some embodiments, video conferencing module 1602 is the same as video conferencing module 925 shown in FIG. 9, and thus performs some or all of the same functions described above with respect to video conferencing module 925.

  In some embodiments, video conferencing manager 1604 may initialize some (eg, image processing manager 1608 and networking manager 1614) or all of the other modules of video conferencing module 1602 when a video conference starts. , Controlling the operation of the video conference module 1602 during the video conference, and suspending the operation of some or all of the other modules of the video conference module 1602 when the video conference ends.

  In some embodiments, the video conference manager 1604 captures images from one or more devices in a video conference and captures by one of both cameras of the dual camera mobile device for display on the dual camera mobile device. The processed image is also processed. For example, the video conferencing manager 1604 of some embodiments reads the decoded image received from another device participating in the video conference from the buffer 1618 and processes the image processed by the CIPU 1650 (ie, the dual camera handset). The image (image captured by the device) is read from the buffer 1606. In some embodiments, video conferencing manager 1604 also scales and composites the images before displaying them on a dual camera portable device. That is, the video conference manager 1604, in some embodiments, generates a PIP view or other composite view for display on the mobile device. In some embodiments, images read from buffers 1606 and 1618 are scaled, while other embodiments only scale images read from one of buffers 1606 and 1618.

  Although FIG. 16 shows the video conference manager 1604 as part of the video conference module 1602, some embodiments of the video conference manager 1604 are implemented as separate components from the video conference module 1602. Accordingly, a single video conference manager 1604 may be used to manage and control several video conference modules 1602. For example, in some embodiments, during a multi-party conference, a separate video conferencing module runs on the local device to interact with each party, and each of these video conferencing modules on the local device has one Managed and controlled by the Video Conference Manager.

  After the image processing manager 1608 of some embodiments processes the image captured by the camera of the dual camera portable device, the image is encoded by the encoder 1655. For example, some embodiments of image processing manager 1608 perform one or more of exposure adjustment, focus adjustment, perspective correction, dynamic range adjustment, and image resizing on images processed by CIPU 1650. In some embodiments, the image processing manager 1608 controls the frame rate of encoded images transmitted to other devices in a video conference.

  Some embodiments of the networking manager 1614 manage one or more connections between the dual camera portable device and other devices participating in the video conference. For example, the networking manager 1614 of some embodiments establishes connections between the dual camera portable device and other devices in the video conference at the start of the video conference and disconnects these connections at the end of the video conference.

  During a videoconference, networking manager 1614 transmits images encoded by encoder 1655 to other devices in the videoconference and routes images received from other devices in the videoconference to decoder 1660 for decoding. I do. In some embodiments, networking manager 1614, rather than image processing manager 1608, controls the frame rate of images transmitted to other devices in the video conference. For example, such an embodiment of some of the networking managers 1614 may be configured to discard (i.e., do not transmit) some of the encoded frames that are assumed to have been transmitted to other devices in the videoconference, and Control the rate.

  As shown, the media exchange module 1620 of some embodiments includes a camera source module 1622, a video compressor module 1624, and a video decompressor module 1626. The media exchange module 1620 is the same as the media exchange module 310 shown in FIG. 3 and will be described in more detail. Camera source module 1622 routes messages and media content between video conference module 1602 and CIPU 1650 through CIPU driver 1630, and video compressor module 1624 routes video conference module 1602 and encoder 1655 through encoder driver 1635. The video decompressor module 1626 routes messages and media content between the video conferencing module 1602 and the decoder 1660 through a decoder driver 1640. In some embodiments, a TNR module 315 (not shown in FIG. 16) included in the media exchange module 310 as part of the camera source module 1622, while in other embodiments, one of the video compressor modules 1624 is implemented. The TNR module 315 is implemented as a unit.

  In some embodiments, CIPU driver 1630 and encoder driver 1635 are the same as CIPU driver 305 and encoder driver 320 shown in FIG. The decoder driver 1640 in some embodiments functions as a communication interface between the video decompressor module 1626 and the decoder 1660. In such an embodiment, decoder 1660 decodes images received from other devices in the video conference through networking manager 1614 and routed through video decompressor module 1626. After the image is decoded, the image is sent back to video conferencing module 1602 through decoder driver 1640 and video decompressor module 1626.

  In addition to performing video processing during a video conference, the video conferencing and processing module 1600 for dual camera portable devices of some embodiments also performs audio processing operations during the video conference. FIG. 17 shows such a software architecture. As shown, the video conference and processing module 1600 includes a video conference module 1602 (including a video conference manager 1604, an image processing manager 1608, and a networking manager 1614), a media exchange module 1620, and a client application 1665. Including. Other components and modules of the video conference and processing module 1600 shown in FIG. 16 are omitted in FIG. 17 for simplicity. Video conferencing and processing module 1600 also includes frame buffers 1705 and 1710, an audio processing manager 1715, and an audio driver 1720. In some embodiments, audio processing manager 1715 is implemented as a separate software module, while in other embodiments, audio processing manager 1715 is implemented as part of media exchange module 1620.

  The audio processing manager 1715 processes audio data captured by the dual camera portable device for transmission to other devices in a video conference. For example, audio processing manager 1715 may receive audio data captured by microphone 1725 through audio driver 1720, encode the audio data, and buffer the encoded audio data for transmission to another device. 1705. The audio processing manager 1715 also processes audio data captured by and received from other devices in a video conference. For example, the audio processing manager 1715 reads audio data from the buffer 1710, decodes the audio data, and then outputs the audio data to the speaker 1730 through the audio driver 1720.

  In some embodiments, video conferencing module 1602 is part of a larger conferencing module, along with audio processing manager 1715 and its associated buffers. This video conferencing and processing module 1600 facilitates the exchange of audio over the Internet Protocol (IP) layer, where a multi-participant audio conference takes place between several devices without exchanging video content. Only the networking manager 1614 and the audio processing manager 1715 are used.

  Here, the operation of the video conference manager 1604 according to some embodiments will be described with reference to FIG. FIG. 18 conceptually illustrates a process 1800 performed by a video conference manager of some embodiments, such as video conference manager 1604 shown in FIG. This may be equivalent to being performed by the management layer 935 of FIG. In some embodiments, when a user of a dual camera mobile device accepts a video conference request (eg, through a user interface displayed on the dual camera mobile device) or when a user of another device The video conferencing manager 1604 performs the process 1800 if it accepts the request sent by the other user.

  Process 1800 begins by receiving (at 1805) an instruction to start a video conference. In some embodiments, the instructions are received from a client application 1665 or from a user through a user interface displayed on a dual camera portable device, and transferred by the client application 1665 to a video conference manager 1604. For example, in some embodiments, if the user of the dual camera mobile device accepts the videoconference request, the instructions are received through the user interface and forwarded by the client application. On the other hand, if the user of the other device accepts the request sent from the local device, in some embodiments (although there may be interaction with the previous user interface to send the initial request) ) Receive instructions from the client application without interacting with the user interface.

  Next, the process 1800 initializes (at 1810) a first module that interacts with the video conference manager 1604. Modules of some embodiments that interact with the videoconference manager 1604 include a CIPU 1650, an image processing manager 1608, an audio processing manager 1715, and a networking manager 1614.

  In some embodiments, initializing CIPU 1650 includes instructing CIPU 1650 to begin processing images captured by one or both cameras of the dual camera portable device. In some embodiments, the image processing manager 1608 is initialized by instructing the image processing manager 1608 to begin reading images from the buffer 1610 and processing and encoding the read images. To initialize the audio processing manager 1715, in some embodiments, audio data captured by the microphone 1725 is stored in a buffer 1710 (received from another device) for encoding and output to a speaker 1730. The audio processing manager 1715 is instructed to start decoding the audio data. Initializing the networking manager 1614 in some embodiments includes instructing the networking manager 1614 to establish a network connection with another device in a video conference.

  The process 1800 then determines (at 1815) whether there are any modules left to initialize. If there are modules left to initialize, the process 1800 returns to operation 1810 to initialize another of the modules. When all required modules have been initialized, process 1800 generates (at 1820) an image composition for display on a dual camera portable device (ie, a local display). These composite images may include the images shown in FIG. 65 (ie, a PIP or other composite display) described below and participate in a video conference with images from the camera of the local dual camera portable device. It may include various combinations of images from cameras of other devices.

  Next, the process 1800 determines (at 1825) whether a change has been made to the video conference. In some embodiments, changes to a video conference are received through interaction with a user using a user interface displayed on a dual camera portable device, while in other embodiments, through a networking manager 1614 (ie, remote control). Receive changes to video conferences from other devices. In some embodiments, changes to the video conferencing settings may be received from the client application 1665 or other modules within the video conferencing module 1602. Video conference settings may change due to changes in network conditions.

  If a change has been made, process 1800 determines (at 1830) whether the change to the video conference is a change to the network settings. In some embodiments, the change is either a change to network settings or a change to image capture settings. If the video conference change is a change to network settings, the process changes the network settings (at 1840) and then proceeds to operation 1845. Changing the network settings in some embodiments includes changing the bit rate at which images are encoded or the frame rate at which images are transmitted to other devices.

  If the change to the video conference is not a change to the network settings, the process 1800 determines that the change is a change to the image capture settings and then proceeds to operation 1835. Process 1800 then performs (at 1835) a change to the image capture settings. In some embodiments, changes to the image capture settings may include, among other settings changes, camera switching (ie, switching which camera on a dual camera handheld device captures video), focus adjustment, This may include adjusting the exposure, displaying or not displaying images from one or both cameras of the dual camera portable device, and zooming in or out of the image displayed on the dual camera portable device.

  At operation 1845, process 1800 determines whether to end the video conference. If the process 1800 determines that the video conference has not ended, the process 1800 returns to operation 1820. If the process 1800 determines that the video conference ends, the process 1800 ends. Some embodiments of the process 1800 may be performed when the process 1800 receives an instruction from the client application 1665 to end the video conference (i.e., received through a user interface of a local dual camera mobile device or participating in a video conference). (Due to instructions received from other devices that are present).

  In some embodiments, video conference manager 1604 performs various operations not shown in process 1800 when the video conference ends. In some embodiments, the CIPU 1650 is instructed to stop generating images, the networking manager 1614 is disconnected to disconnect the network connection with other devices during the video conference, and stops generating and encoding images. To the image processing manager 1608 as follows.

E. FIG. Temporal Noise Reduction Some embodiments include a separate temporal noise reduction module for processing video images to reduce video noise. The temporal noise reduction module of some embodiments compares subsequent images of the video sequence to identify and eliminate unwanted noise from the video.

  FIG. 19 conceptually illustrates a software architecture for such a temporal noise reduction (TNR) module 1900 of some embodiments. In some embodiments, the TNR module 1900 is implemented as part of an application (eg, as part of the media exchange module shown in FIG. 3), but in other embodiments, a stand-alone application used by other applications. The TNR module 1900 is implemented. In yet another embodiment, the TNR module 1900 is implemented as part of an operating system running on a dual camera portable device. In some embodiments, the TNR module 1900 is implemented by a set of APIs that provide some or all of the functions of the TNR module 1900 to other applications.

  As shown in FIG. 19, the TNR module 1900 includes a TNR manager 1905, a difference module 1910, a pixel averaging module 1915, and a motion history module 1920. Although FIG. 19 shows the three modules 1910, 1915, and 1920 as separate modules, some embodiments implement the functions of these modules in a single module, as described below. The TNR module 1900 of some embodiments receives an input image, a reference image, and a motion history as inputs. In some embodiments, the input image is the image currently being processed and the reference image is the previous image in the video sequence, to which the input image is compared. The TNR module 1900 outputs an output image (a noise-reduced version of the input image) and an output motion history.

  The TNR manager 1905 of some embodiments directs the flow of data through the TNR module 1900. As shown, the TNR manager 1905 receives an input image, a reference image, and a motion history. The TNR manager 1905 also outputs an output image and output motion history. The TNR manager 1905 sends the input image and the reference image to the difference module 1910, and receives the difference image from the difference module 1910.

  In some embodiments, difference module 1910 processes data received from TNR manager 1905 and sends the processed data to TNR manager 1905. As shown, difference module 1910 receives an input image and a reference image from TNR manager 1905. The difference module 1910 of some embodiments generates a difference image by subtracting pixel values of one image from pixel values of another image. The difference image is sent to the TNR manager 1905. The difference image of some embodiments indicates a difference between the two images to identify a changed section of the input image and a remaining section of the input image as compared to a previous image.

  TNR manager 1905 also sends the input image and the reference image to pixel averaging module 1915. As shown, in some embodiments, the motion history is also sent to the pixel averaging module 1915. However, in other embodiments, only the input image and the reference image could be sent without sending the motion history. In either embodiment, the TNR manager 1905 receives the processed image from the pixel averaging module 1915.

  The pixel averaging module 1915 of some embodiments uses the motion history to determine whether to average the pixels from the input image and the reference image for a particular location in the image. In some embodiments, the motion history includes a probability value for each pixel of the input image. The particular probability value represents the probability that the corresponding pixel of the input image has changed (ie, is a dynamic pixel) relative to the corresponding pixel of the reference image. For example, if the probability value of a particular pixel of the input image is 20, it indicates that the probability that the particular pixel of the input image has changed with respect to the corresponding pixel of the reference image is 20%. As another example, if the probability value of a particular pixel of the input image is 0, it means that the particular pixel of the input image has not changed relative to the corresponding pixel of the reference image (ie, a static pixel Yes).

  In different embodiments, the probability values of the input image are stored differently. In some embodiments, one data array may store the probability values for each pixel of the input image. In other embodiments, the probability values could be stored in a matrix (eg, an array of arrays) having the same dimensions as the resolution of the video image. For example, if the resolution of a video image is 320 × 240, the matrix is also 320 × 240.

  When the pixel averaging module 1915 receives the motion history from the TNR manager 1905 in addition to the input image and the reference image, the pixel averaging module 1915 reads the probability value of each pixel of the input image. If the probability value for a particular pixel in the input image is below a specified threshold (eg, 5%, 20%), the pixel averaging module 1915 is less likely to have motion for the particular pixel, and therefore Based on the assumption that the difference between images at that pixel may be due to noise, the specific pixel value is averaged with the corresponding pixel value of the reference image.

  If the probability for a particular pixel of the input image does not fall below a specified threshold, the pixel averaging module 1915 does not change the particular pixel of the input image (ie, the pixel value of that pixel is Remains the same as the image). This is because there is a high likelihood of motion at a particular pixel, and therefore the differences between the images are likely not to be the result of noise. In some embodiments, if the motion history is not sent to the pixel averaging module 1915, the pixel averaging module 1915 averages each pixel of the input image with the corresponding pixel of the reference image. The processed image output by the pixel averaging module 1915 and sent to the TNR manager 1905 is the input image pixel value for every pixel that was not averaged and the average for every pixel that was averaged by the pixel averaging module 1915. Contains the converted pixel values.

  In some embodiments, motion history module 1920 processes data received from TNR manager 1905 and returns result data to TNR manager 1905. The motion history module 1920 of some embodiments receives input images and motion history from the TNR manager 1905. In some embodiments, this data is input to a Bayesian estimator to generate a new motion history (ie, a set of probability values) that can be used for pixel averaging for the next input image. In other embodiments, another estimator is used to generate a new motion history.

  Here, the operation of the TNR module 1900 will be described with reference to FIG. This diagram conceptually illustrates a process 2000 of some embodiments for reducing temporal noise in a video image. Process 2000 begins with TNR manager 1905 receiving (at 2005) an input image, a reference image, and a motion history. The input image is the image currently being processed for noise reduction. In some embodiments, the reference image is the image immediately preceding the sequence of images of the video received from the CIPU. However, in other embodiments, the reference image is an output image generated from processing of the immediately preceding input image (ie, the output of TNR module 1900). The motion history is an output motion history generated from the processing of the immediately preceding input image.

  If the input image is the first image of a video, the TNR module 1900 of some embodiments does not process the first image (ie, does not apply a TNR to the first image). In other words, the TNR manager 1905 receives the first image and simply outputs the first image. In another embodiment, if the input image is the first image of a video, the first image is used as the input image and the reference image, and the TNR module 1900 processes the image as described below. Further, if the input image is the first image of a video, the motion history is empty (eg, null, all zeros, etc.), and the TNR manager 1905 simply outputs the empty motion history as the output motion history. is there.

  TNR manager 1905 then determines (at 2010) whether the input image is static. To make this determination, in some embodiments, the input image and the reference image are sent to the difference module 1910, and the difference image is received from the difference module 1910. If the difference between the two images is below a specified threshold (eg, 5% difference, 10% difference, etc.), some embodiments classify the input image as static.

  If the input image is a static image, the TNR manager 1905 may use the input image to average (at 2015) the pixels of the input image with the pixels of the reference image in order to reduce noise from the static image. And the reference image to the pixel averaging module 1915. The process then proceeds to 2040, which is described below.

  If the input image is not a static image, the TNR manager sends the input image, the reference image, and the motion history to the pixel averaging module 1915 for processing. Pixel averaging module 1915 selects (at 2020) pixels of the input image. As described above, the pixel averaging module 1915 uses the motion history to determine (at 2025) whether the probability of pixel motion is below a certain threshold.

  If the probability of the selected pixel is below a certain threshold, the pixel averaging module 1915 averages (at 2030) the pixels of the input image with the corresponding pixels of the reference image. Otherwise, the pixels are not averaged and at that particular pixel, the output image is the same as the input image. Pixel averaging module 1915 then determines (at 2035) whether unselected pixels remain in the input image. If any pixels have not been processed, the process returns to operation 2020 and selects the next pixel. Pixel averaging module 1915 performs operations 2020-2030 until all pixels have been evaluated.

  The process then updates (at 2040) the motion history. As shown in FIG. 19 and described above, the motion history module 1920 updates the motion history based on the input image. A new motion history is output by the TNR manager along with the processed image from the pixel averaging module.

F. Image Processing Manager and Encoder In some embodiments, in addition to the temporal noise reduction and image processing operations performed by the CIPU and / or CIPU driver, various image processing operations are performed in the image processing layer 930 of the videoconferencing module 925. Execute. These image processing operations may include, among other things, exposure adjustment, focus adjustment, perspective correction, dynamic range adjustment, and image resizing.

  FIG. 21 conceptually illustrates a process 2100 for performing such an image processing operation. In some embodiments, some or all of the operations of process 2100 are performed by a combination of image processing manager 1608 and encoder driver 1635 of FIG. In some of such embodiments, image processing manager 1608 performs pixel-based processing (eg, resizing, dynamic range adjustment, perspective correction, etc.). In some embodiments, the process 2100 for images to be transmitted to another device participating in the video conference is performed during the video conference.

  Here, the process 2100 will be described with reference to FIG. The process begins by reading an image from buffer 1606 (at 2105). In some embodiments, the image read is an image of a video (ie, an image in an image sequence). This video may have been captured by the camera of the device on which process 2100 is performed.

  Next, the process 2100 performs (at 2110) an exposure adjustment on the read image. In some embodiments, the exposure adjustment is performed through a user interface displayed on the dual camera portable device. FIG. 22 illustrates an exemplary exposure adjustment operation of such an embodiment.

  This figure illustrates the exposure adjustment operation with respect to three stages 2210, 2215 and 2220 of the UI 2205 of the device 2200. The first stage 2210 shows a UI 2205 that includes a display area 2225 and a display area 1155. As shown, the display area 2225 displays an image 2230 of the sun and a man with darker face and body color. Darker face and body colors indicate that the male is not properly exposed. Image 2230 could be a video image captured by the camera of device 2200. As shown, display area 1155 includes a selectable UI item 2250 to end the video conference. In some embodiments, the layout of display area 1155 is the same as the layout of display area 1155 of FIG. 12 described above.

  The second step 2215 indicates that the user of the device 2200 initiates an exposure adjustment operation by selecting an area of the display area 2225. In this example, the selection is made by placing finger 2235 at any position within display area 2225. In some embodiments, the user selects an exposure adjustment from a menu of possible image setting adjustments.

  The third stage 2220 shows the male image 2240 after the exposure adjustment operation has been completed. As shown, image 2240 is similar to image 2230, except that the men in image 2240 are properly exposed. In some embodiments, a properly exposed image is an image captured after an improperly exposed image. The exposure adjustment operation initiated in the second step 2215 adjusts the exposure of subsequent images captured by the camera of device 2200.

  Returning to FIG. 21, process 2100 then performs (at 2115) a focus adjustment on the image. In some embodiments, the focus adjustment is performed through a user interface displayed on the dual camera portable device. FIG. 23 conceptually shows an example of such a focus adjustment operation.

  FIG. 23 illustrates the focus adjustment operation with respect to three different stages 2310, 2315 and 2320 of the UI 2305 of the device 2300. The first stage 2310 shows a UI 2305 that includes a display area 2325 and a display area 1155. Display area 2325 presents a blurred image 2330 of the male captured by the camera of device 2300. Blurring indicates that the male image 2330 is out of focus. That is, when the male image 2330 was captured by the camera, the camera lens was not focused on the male. Image 2330 may also be a video image captured by the camera of device 2300. As shown, display area 1155 includes a selectable UI item 2350 to end the video conference. In some embodiments, the layout of display area 1155 is the same as the layout of display area 1155 of FIG. 12 described above.

  The second stage 2315 shows that the user of the device 2300 starts the focus adjustment operation by selecting an area of the display area 2325. In this example, the selection is made by placing finger 2335 at an arbitrary position in display area 2325. In some embodiments, the user selects focus adjustment from a menu of possible image setting adjustments.

  The third stage 2320 shows the male image 2340 after the focus adjustment operation has been completed. As shown, image 2340 is the same as image 2330, but the men in image 2340 are displayed more clearly. This indicates that the camera lens is properly focused on the male. In some embodiments, a properly focused image is an image captured after an improperly focused image. The focus adjustment operation, initiated in the second step 2315, adjusts the focus of subsequent images captured by the camera of device 2300.

  Returning to FIG. 21, process 2100 performs (at 2120) an image resize on the image. In some embodiments, image resizing is performed on the image in order to reduce the number of bits used to encode the image (ie, reduce the bit rate). In some embodiments, process 2100 performs image resizing, as described below with respect to FIG.

  Process 2100 then performs a perspective correction (at 2125) on the image. In some embodiments, process 2100 performs perspective correction as described below in FIG. Such perspective corrections include using data acquired by one or more accelerometers and / or gyroscope sensors that identify the orientation and movement of the dual camera portable device. This data is then used to modify the image in order to correct for the misaligned perspective.

  After perspective correction has been performed on the image, process 2100 adjusts (at 2130) the dynamic range of the image. In some embodiments, the dynamic range of an image is the range of possible values that each pixel of the image can have. For example, an image having a dynamic range of 0 to 255 may be adjusted to a value in the range of 0 to 128 or any other range. By adjusting the dynamic range of the image, the amount of bits used to encode the image may be reduced (ie, the bit rate may be reduced), thereby smoothing the image.

  Adjusting the dynamic range of an image can also be used for various other purposes. One purpose is to reduce image noise (eg, the image was captured by a noisy camera sensor). To reduce noise, the dynamic range of the image is adjusted so that the black level is redefined to include lighter black (ie, crush blacks). In this way, image noise is reduced. Another purpose of the dynamic range adjustment is to adjust one or more colors or color ranges to improve image quality. For example, in some embodiments, the image captured by the front camera may be assumed to be an image of a human face. Therefore, the dynamic range of the image can be adjusted to increase the red and pink colors to make the human cheek appear rosy / more rosy. The dynamic range adjustment operation can be used for other purposes.

  Finally, the process 2100 determines (at 2135) one or more rate controller parameters used to encode the image. Such rate controller parameters may include, in some embodiments, quantization parameters and frame types (eg, P (predictive), B (bi-directional), I (intra-coded)). Thereafter, the process ends.

  Although various operations of process 2100 are shown to be performed in a particular order, many of these operations (exposure adjustment, focus adjustment, perspective correction, etc.) can be performed in any order and are dependent on each other. It will be understood by those skilled in the art that this is not the case. That is, the process of some embodiments may perform focus adjustment prior to exposure adjustment, or may perform similar changes as the process illustrated in FIG.

1. Perspective Correction As described above, some embodiments perform a perspective correction on an image before displaying or transmitting the image. In some cases, one or more of the cameras on the dual camera portable device are not properly oriented to the subject, and the subject appears distorted in the uncorrected image. Perspective correction can be used to process an image so that the image closely reflects how objects in the image are visible to humans.

  FIG. 24 conceptually illustrates a perspective correction process 2400 performed by the image processing manager of some embodiments, such as the process shown in FIG. Process 2400 of some embodiments is performed by image processing layer 930 (which may include image processing manager 1608) shown in FIG. In some embodiments, process 2400 is performed in operation 2125 of process 2100 to correct the perspective of a recently captured video image before displaying or transmitting the image.

  Process 2400 begins by receiving data (at 2405) from an accelerometer sensor, which in some embodiments is part of a dual camera portable device. In some embodiments, the accelerometer sensor measures the rate of change of the speed of the device (ie, the acceleration of the device) along one or more axes. The process also receives (at 2410) data from the gyroscope sensor, which may also be part of a dual camera portable device in some embodiments. The gyroscope sensor and accelerometer sensor of some embodiments may be used separately or in combination to identify the orientation of a dual camera portable device.

  Next, the process 2400 determines (at 2415) the amount of perspective correction to make based on the data obtained from the accelerometer and gyroscope sensors. Generally, when the orientation is further off-axis, more perspective correction is needed to generate an optimal image. In some embodiments, a warp parameter representing a perspective correction amount is calculated based on a device orientation.

  After determining the amount of perspective correction to perform, process 2400 receives (at 2420) an image captured by a camera of the dual camera portable device. This process may be performed for each image of the video sequence captured by the camera. In some embodiments, separate calculations may be performed for images arriving from each of the two cameras on the dual camera portable device.

  The process then modifies (at 2425) the image based on the determined perspective correction amount. In some embodiments, in addition to the distortion parameter or other notation of the perspective correction amount, a baseline image or other information (eg, a point entered by a user as to which correction is to be performed) is also used. After changing the image, process 2400 ends.

  FIG. 25 conceptually illustrates an exemplary image processing operation of some embodiments. This figure illustrates a first image processing operation 2505 performed by a first image processing module 2520 that does not use perspective correction, and a second image processing operation 2550 performed by a second image processing module 2565 that uses perspective correction. Show.

  As shown, a first image processing operation 2505 is performed on the first image 2510 of block 2515 from atmospheric perspective looking down at an angle toward the block. From that perspective, the top of block 2515 is closer than the bottom of the block. Thus, block 2515 appears to tilt toward the camera that captured the first image 2510. FIG. 25 also shows the processed first image 2525 after processing by the first image processing module 2520. As shown, block 2515 of processed first image 2525 appears to be the same post-processing because first image processing module 2520 did not perform a perspective correction.

  The second image processing operation 2550 is performed on the second image 2555 of block 2560. Block 2560 is the same as block 2515 of first image 2510. FIG. 25 also shows a processed second image 2575 after processing of the second image 2555 by the perspective collector 2570 of the second image processing module 2565. Perspective collector 2570 may use process 2400 to correct the perspective of second image 2555. Based on data from the accelerometer and gyroscope indicating that the camera that captured the second image 2555 is tilted at a downward angle (and possibly based on other data), the perspective collector 2570 may The second image can be corrected so that the blocks appear to appear straight on the processed second image 2575.

2. Resizing and Bitstream Operations Among the functions performed by the image processing layer 930 of some embodiments and described above with respect to FIG. 21 are image resizing and bitstream operations. In some embodiments, the image resizing (performed in operation 2130) includes enlarging or reducing the image (ie, changing the number of pixels used to represent the image). In some embodiments, the bitstream operation includes inserting data indicating the size of the resized image into the bitstream. This resizing and bitstream manipulation is performed in some embodiments by an encoder driver (eg, driver 1635).

  FIG. 26 conceptually illustrates a software architecture for such an encoder driver 2600 of some embodiments, where an exemplary resizing and bitstream performed by the encoder driver 2600 on an exemplary image 2605. The operation is shown. In some embodiments, image 2605 is an image of a video captured by a camera of a dual camera portable device for transmission to another device (s) in a video conference. Referring to FIG. 16, in some embodiments, a video image will go from CIPU 1650 through CIPU driver 1630 and camera source module 1622 to buffer 1606, where it will be read by image processing manager 1608. After being subjected to image processing (eg, focus adjustment, exposure adjustment, perspective correction) by the image processing manager 1608, the image is sent to the encoder driver 1635 through the buffer 1610 and the video compressor module 1624.

  As shown, the encoder driver 2600 includes a processing layer 2610 and a rate controller 2645. An example of a rate controller of some embodiments is shown in FIG. 30, described below. The processing layer 2610 includes an image resizer 2615 and a bit stream manager 2625. In some embodiments, these modules perform various operations on the image both before and after the image is encoded. In this example, the image resizer is shown as part of the processing layer 2610 of the encoder driver 2600, however, in some embodiments, the image resizer is implemented as part of the image processing manager 1608 instead of the encoder driver 2600. (That is, the image resizing occurs before sending the image and size data to the encoder driver.)

  As shown, image resizer 2615 resizes the image before it is sent to encoder 2650 through rate controller 2645. Image 2605 is sent through resizer 2615 and reduced to image 2630. In some embodiments, an image can be enlarged in addition to being reduced.

  As shown in FIG. 26, in some embodiments, an incoming image (eg, image 2605) is reduced, and then the reduced image (eg, image 2630) is reduced to the same size (in pixels) as the incoming image. (I.e., the number of rows and columns of pixels in image 2605 is the same as the number of rows and columns of pixels in spatially redundant image 2635) and is superimposed on a spatially redundant image (e.g., image 2635). I do. In some embodiments, the reduced image 2630 is superimposed on the upper left corner of the spatially redundant image (to generate a composite image 2640, as shown), while in other embodiments, the reduced image is Are superimposed on different sections of the spatially redundant image (eg, center, upper right, upper center, lower center, lower right, etc.).

  In some embodiments, the spatially redundant image has an image that is substantially all one color (eg, black, blue, red, white, etc.) or a repeating pattern (eg, checks, stripes, etc.). It is an image. For example, the spatially redundant image 2635 shown in FIG. 26 has a repeating cross pattern. The spatially redundant portions of the composite image 2640 can be easily compressed by the encoder into small amounts of data because of the repeatability. Moreover, if the sequence of images is all reduced and the spatially redundant images used are the same for each image in the sequence, the amount of data needed to represent the encoded image is further reduced Temporal compression can be used to achieve this.

  Some embodiments of the image resizer 2615 also generate size data 2620 indicating the size of the resized image (eg, the size of the reduced image 2630) and send the generated size data 2620 to the bitstream manager 2625. . The size data 2620 of some embodiments indicates the size of the resized image 2630 in terms of the number of rows of pixels and the number of columns of pixels (ie, height and width) of the resized image 2630. In some embodiments, size data 2620 also indicates the location of resized image 2630 within composite image 2640.

  After the image has been resized, the composite image 2640 is sent to the encoder 2650 through the rate controller 2645. Rate controller 2645, in some embodiments, controls the bit rate (ie, data size) of the image output by encoder 2650, as described in further detail below. The encoder 2650 of some embodiments compresses and encodes an image. Encoder 2650 is a H.264 encoder. H.264 encoding or another encoding may be used.

  In some embodiments, the bitstream manager 2625 receives a bitstream of one or more encoded images from the encoder 2650 and inserts size data into the bitstream. For example, in some embodiments, bitstream manager 2625 receives size data 2620 from image resizer 2615 and inserts size data 2620 into encoded composite image 2640 bitstream 2655 received from encoder 2650. The output of bitstream manager 2625 is in this case a modified bitstream 2660 that includes size data 2620. In different embodiments, the size data 2620 is inserted at different locations in the bitstream 2655. For example, the bit stream 2660 indicates the size data 2620 inserted at the head of the bit stream 2660. However, in other embodiments, the size data 2620 is inserted at the end of the bitstream 2655, at the center of the bitstream 2655, or at any other location within the bitstream 2655.

  In some embodiments, bitstream 2655 is a bitstream of a sequence of one or more encoded images including composite image 2640. In some of such embodiments, the images in the sequence are all resized to the same size, and the size data 2620 indicates the size of those resized images. After the image has been transmitted to the device at the other end of the video conference, the receiving device can extract the size information from the bitstream and use the size information to properly decode the received image.

  FIG. 27 conceptually illustrates an image resizing process 2700 performed by an encoder driver of a dual camera portable device, such as driver 2600. Process 2700 begins by receiving (at 2705) an image (eg, image 2605) captured by a camera of a dual camera portable device. If the dual camera device is capturing images with both cameras, in some embodiments, the process 2700 is performed on images from both cameras.

  Next, the process 2700 resizes (at 2710) the received image. As described above, different embodiments resize image 2605 differently. For example, image 2605 of FIG. 26 is scaled down and superimposed on spatially redundant image 2635 to generate composite image 2640.

  Process 2700 then sends (at 2715) an encoder 2650 to encode the resized image (eg, a composite image 2640 including resized image 2630). In some embodiments of the process 2700, the resized image 2630 (included in the composite image 2640) is sent to the encoder 2650 through a rate controller that determines the bit rate at which the encoder encodes the image. The encoder 2650 of some embodiments compresses and encodes an image (eg, using a discrete cosine transform, quantization, entropy encoding, etc.) and converts a bitstream having the encoded image to an encoder driver 2600. Return to

  Next, the process 2700 sends data (at 2720) to the bitstream manager indicating the size of the resized image (eg, size data 2620). As shown in FIG. 26, this operation is performed within encoder driver 2600 in some embodiments (ie, one module of encoder driver 2600 sends size data to another module of encoder driver 2600).

  After the resized image has been encoded by the encoder 2650, the process 2700 receives (at 2725) a bitstream from the encoder. As shown, in some embodiments, the bitstream is received at a bitstream manager, which also receives the size data. The received bitstream includes the encoded composite image, and may also include one or more additional images in the video sequence.

  Process 2700 then ends (at 2730) with data indicating the size of the resized image (eg, size data 2620) into the bitstream and ends. As shown in FIG. 26, this operation is also performed by the bitstream manager in some embodiments. As mentioned above, different embodiments insert the size data into different parts of the bitstream. In the illustrated example, size data 2620 is inserted at the beginning of bit stream 2655, as shown in the resulting bit stream 2660. This bitstream can now be transmitted to another device participating in the video conference and decoded and viewed on another device.

  In some embodiments, a decoder driver (eg, driver 1640) performs the opposite function of an encoder driver. That is, the decoder driver extracts size data from the received bit stream, passes the bit stream to the decoder, and resizes the decoded image using the size data. FIG. 28 conceptually illustrates a software architecture for such a decoder driver 2800 of some embodiments, with exemplary bitstream operations performed by the decoder driver 2800 on an exemplary bitstream 2825 and Indicates a resize operation.

  In some embodiments, the bitstream 2825 is a bitstream (eg, a bitstream from an encoder driver such as the driver 2600) that includes an encoded image of video captured by the device camera in a video conference. It is transmitted to the device on which the decoder driver 2800 operates. Referring to FIG. 16, in some embodiments, the bitstream is received by the networking manager 1614, sent to the buffer 1616, read from the buffer 1616 by the video decompressor module 1626, and sent to the decoder driver 1640.

  As shown, the decoder driver 2800 includes a processing layer 2805. The processing layer 2805 includes an image resizer 2810 and a bit stream manager 2820. In some embodiments, these modules 2810 and 2820 perform various operations on the received image both before and after the image is encoded. In this example, the image resizer 2810 is shown as part of the processing layer 2805 of the decoder driver 2800, however, some embodiments implement the image resizer as part of the image processing manager 1608 rather than a decoder driver. (That is, the image size change is performed after the image is transmitted from the decoder driver 2800).

  As shown, the bitstream manager 2820 of some embodiments receives a bitstream of one or more encoded images (ie, images in a video sequence) and decodes the bitstream for decoding by a decoder 2835. Extract size data from the bitstream before sending it to. For example, as shown in FIG. 28, a bitstream manager 2820 receives a bitstream 2825 of an encoded image, extracts size data 2815 from the bitstream 2825, and generates the resulting data for decoding ( The bit stream 2830 (without size data 2815) is sent to the decoder 2835. As shown, the bitstream manager 2820 sends the extracted size data 2815 to the image resizer 2810 in some embodiments.

  The size data 2815 in some embodiments is the same as the size data 2620 inserted into the bitstream by the encoder driver 2600. As described above with reference to FIG. 26, the size data 2815 of some embodiments indicates the size of the sub-image 2845 as the number of pixel rows and the number of pixel columns of the sub-image 2845. Size data 2815 may also indicate the position of sub-image 2845 within larger spatially redundant image 2840. In this example, the bit stream 2825 indicates the size data 2815 inserted at the head of the bit stream 2825. However, as described above, different embodiments insert the size data 2815 at different locations in the bitstream 2825.

  The image resizer 2810 of some embodiments uses the size data received from the bitstream manager 2820 to extract sub-images from the image. For example, FIG. 28 illustrates that image resizer 2810 receives image 2840 including sub-image 2845 from decoder 2835. As shown, the image resizer 2810 of some embodiments extracts a sub-image 2845 from the image 2840. This extracted image can then be displayed on a dual camera portable device.

  FIG. 29 conceptually illustrates an image extraction process 2900 of some embodiments performed by a decoder driver of a device participating in a video conference, such as driver 2800. The process begins by receiving (at 2905) a bitstream of an encoded image (eg, bitstream 2825). The bitstream may be transmitted from a device participating in a video conference with a device on which the decoder driver is operating, or may be stored in a storage device of the device. If the device is receiving images from multiple sources, some embodiments perform process 2900 on the images from each source.

  Next, the process 2900 extracts (at 2910) size data from the bitstream. As described above, this size data may be found at various locations in the bitstream. While some embodiments know where to look for the size data, other embodiments look for a particular signature that indicates where the size data is in the received bitstream. In some embodiments, the size data indicates the size (eg, the number of pixels in each row and the number of pixels in each column) and position of the sub-images in the encoded image.

  Process 2900 then sends (at 2915) the extracted size data to the image resizer. As shown in FIG. 28, this operation is performed within the decoder driver in some embodiments (ie, one module of the decoder driver sends size data to another module of the decoder driver).

  Process 2900 also sends (at 2920) the bitstream to the decoder for decoding. The decoder, in some embodiments, decompresses and decodes the bitstream (eg, using an inverse discrete cosine transform, inverse quantization, etc.) and returns the reconstructed image to a decoder driver.

  After the bitstream is decoded by the decoder, process 2900 receives (at 2925) the decoded image from the decoder. As shown, some embodiments receive an image at an image resizer, which also receives size data from a bitstream manager. The process then extracts (at 2930) a sub-image from the decoded image using the received size data. As shown, the sub-image 2845 is extracted from the upper left of the decoded image 2840, as indicated by the size data 2815. The extracted sub-image may now be displayed on a display device (eg, a screen of a dual camera portable device).

3. Rate Controller In some embodiments, the two cameras of the device have different sets of characteristics. For example, in some embodiments, the front camera is a lower resolution camera that is optimized for capturing motion video images, and the rear camera is a higher resolution camera that is optimized for capturing still images. Camera. Other embodiments may use different combinations of cameras with different characteristics, for reasons such as device cost, function and / or shape.

  Cameras with different characteristics can result in different artifacts. For example, a higher resolution camera may reveal more noise than a lower resolution camera. Images captured by higher resolution cameras may exhibit a higher level of spatial or temporal complexity than images captured by lower resolution cameras. Similarly, different cameras with different optical properties may result in different gamma values in the captured image. Different light sensing mechanisms used by different cameras to capture images can also result in different artifacts.

  Some of these camera-specific artifacts obscure artifacts generated from other sources. For example, in images captured by high-resolution cameras with high levels of noise, artifacts that are a by-product of the video encoding process are less visible. When encoding noise to hide behind camera-specific artifacts (such as quantization distortion), the video encoding process can use a larger quantization step size to achieve a reduced bit rate. On the other hand, when the camera introduces fewer artifacts (as in lower resolution cameras), the video encoding process may use a finer quantization step size to avoid unacceptable levels of visual distortion due to quantization. Can be used. Thus, a video coding process optimized to exploit or compensate for these camera-specific characteristics achieves a better rate-distortion trade-off than a video coding process that does not consider these camera-specific characteristics. it can.

  To take advantage of these camera-specific properties for the purpose of performing a rate-distortion trade-off, some embodiments implement two video encoding processes, each of which is optimized for each of the two cameras Be transformed into FIG. 30 shows an example of a system having two video encoding processes for two cameras 3060 and 3070. As shown in FIG. 30, the system 3000 includes an encoder driver 3010, rate controllers 3020 and 3040, and a video encoder 3030. Encoder 3030 encodes video images captured from video cameras 3060 and 3070 into bit streams 3080 and 3090.

  In some embodiments, video encoder driver 3010 is a software module running on one or more processing units. Video encoder driver 3010 provides an interface between video encoder 3030 and other components of the system, such as a video camera, an image processing module, a network management module, and storage buffers. The encoder driver 3010 controls the flow of captured video images from the camera and image processing modules to the video encoder 3030, and routes to the storage buffer and network management module for the encoded bit streams 3080 and 3090. (Conduit1) is also provided.

  As shown in FIG. 30, encoder driver 3010 includes two different instances 3020 and 3040 of a rate controller. These multiple instances may be two different rate controllers for two different cameras, or one rate controller configured in two different ways for two different cameras. . Specifically, in some embodiments, the two rate controllers 3020 and 3040 represent two separate rate controllers. Alternatively, in other embodiments, the two rate controllers 3020 and 3040 are two different configurations of a single rate controller.

  FIG. 30 also shows an encoder driver 3010 that includes a status buffer 3015 that stores encoded status information for rate control operations used during a video conference. Specifically, in some embodiments, two different rate controllers or two different configurations of the same rate controller share the same encoded state information stored in state buffer 3015 during a video conference. Such sharing of state information allows for uniform rate controller operation in dual video capture video conferencing. This sharing also allows for optimal video encoding during camera switching operations during a single video capture video conference (i.e., the rate control operation for encoding video captured by the current camera is To use the encoding state information maintained by the rate control operation for encoding the video captured by this camera). FIG. 30 shows the state buffer 3015 which is a part of the encoder driver 3010, but in another embodiment, the state buffer 3015 is mounted outside the encoder driver 3010.

  In state buffer 3015, different embodiments store different types of data (eg, different types of encoding parameters) that represent encoding state information. One example of such encoding state information is the current target bit rate for a video conference. One scheme for identifying the target bit rate is described in Section III. B. Other examples of such encoding state information include, among other encoding state information, one of buffer fullness, maximum buffer occupancy, and bit rate of recently encoded frames. Including one or more.

  Thereafter, the rate controller can use the target bit rate (or another encoded state parameter stored in a state buffer) to calculate one or more parameters used in the rate control operation. For example, as described further below, the rate controller of some embodiments uses a current target bit to calculate a quantization parameter QP for a macroblock or frame. By way of example, in some embodiments, the current target bit rate is used to calculate a quantization adjustment parameter that derives a macroblock and / or frame quantization parameter QP. Thus, by sharing the target bit rate between the two rate control operations (of two rate controllers or of two different configurations of one rate controller) during the video conference camera switching operation, The rate control operation for encoding the captured video will benefit from the encoding state data from the previous rate control operation for encoding the video captured by the immediately prior camera.

  FIG. 30 shows encoder driver 3010 to include two different rate-controller instances 3020 and 3040. However, in other embodiments, these rate controller instances 3020 and 3040 are incorporated into video encoder 3030. Video encoder 3030 encodes video images captured from cameras 3060 and 3070 into digital bit streams 3080 and 3090. In some embodiments, the video encoder generates a bitstream that conforms to a conventional video coding standard (eg, H.264 MPEG-4). In some of these embodiments, the video encoder performs coding operations including motion estimation, discrete cosine transform ("DCT"), quantization and entropy coding. The video encoder also performs a decoding operation which is the reverse function of the encoding operation.

  In some embodiments, encoder 3030 includes a quantizer module 3032 for performing quantization. The quantizer module is controlled by a quantization parameter 3022 or 3042 from the rate controller 3020 or 3040. In some embodiments, each quantization parameter is set by a corresponding rate controller and is a function of one or more attributes of the camera associated with the rate controller, as described further below. The rate controller can reduce the number of bits used for encoding according to a coarser quantization step size setting, or can increase the number of bits used by setting a finer quantization step size. By controlling the quantization step size, the rate controller also determines how much distortion is introduced into the encoded video image. Thus, the rate controller can perform a tradeoff between bit rate and image quality. In performing the rate-distortion trade-off, the rate controller monitors the bit rate to avoid overflowing the memory buffer, underflowing the memory buffer, or exceeding the transmission channel capacity. The rate controller must also control the bit rate to provide the highest image quality possible and avoid unacceptable distortion of the image quality due to quantization. In some embodiments, each rate controller stores the monitored data in state buffer 3015 as a set of state data values. In some embodiments, rate controllers 3020 and 3040 use camera-specific attributes to optimize the rate-distortion trade-off.

  In some embodiments, each rate controller optimizes the rate-distortion trade-off by directly applying a change factor to its quantization parameter. In some of these embodiments, the modification factor is predetermined and is integrated into the device along with the camera. The device does not need to dynamically calculate these change factors. In another embodiment, the system uses the incoming images captured by the camera to dynamically determine the appropriate change factor specific to the camera. In some of these embodiments, the system analyzes the sequence of incoming video images captured by the camera in multiple encoding passes to gather certain statistics about the camera. The system then uses these statistics to derive change factors for the camera optimized quantization parameters.

In some embodiments, these camera-specific modification factors are applied to quantization parameters via visual masking attributes of the video image. The visual masking attribute of an image or part of an image is an indication of how many coding artifacts are acceptable in the image or image part. Some embodiments calculate a visual masking attribute that quantifies the luminance energy of the image or image portion, while other embodiments quantify the activity energy or complexity of the image or image portion. Calculate visual masking attributes. Regardless of how the visual masking attribute is calculated, some embodiments use the visual masking attribute to calculate a modified or masked quantization parameter for a video frame. . Some of these embodiments calculate the masked quantization parameter as a function of the frame-level visual masking attribute φ _frame and the reference visual masking attribute φ _R. In some embodiments, the quantization parameters modified by the visual masking attributes φ _frame and φ _R are expressed as follows:

MQP _frame = QP _nom + β _frame * (φ _frame -φ _R ) / φ _R (1)
Where MQP _frame is the masked or modified quantization parameter of the frame, QP _nom is the initial or nominal quantization value, and β _frame is a constant that fits the local statistics It is. In some embodiments, the reference visual masking attribute φ _R and the nominal quantization parameter QP _nom are predetermined from an initial or periodic assessment of network conditions.

In some embodiments, the visual masking attribute φ _frame in equation (1) is calculated as follows.

φ _frame = C · (E · avgFrameLuma) ^β · (D · avgFrameSAD) ^α (2)
In the above equation, avgFrameLuma is the average luminance value of the frame, and avgFrameSAD is the average absolute value difference sum of the frame. The constants α, β, C, D and E fit the local statistics. These constants are, in some embodiments, compatible with camera-specific properties.

Some embodiments also calculate masked quantization parameters for a portion of the video image, such as a macroblock. In those examples, the masked quantization parameter is calculated as a function of the macroblock visual masking attribute φ _MB as follows:

MQP _MB = MQP _frame + β _MB * (φ _MB- φ _frame ) / φ _frame (3)
Where β _MB is a constant that fits the local statistics, and the MQP _frame is calculated in some embodiments using equations (1) and (2). In some embodiments, the visual masking attribute φ _MB in equation (3) is calculated as follows.

φ _MB = A · (C · avgMBLuma) ^β · (B · MBSAD) ^α (4)
In the above equation, avgMBLuma is the average luminance value of the macroblock, and avgMBSAD is the average absolute value difference sum of the macroblock. The constants α, β, A, B and C fit the local statistics. These constants are, in some embodiments, compatible with camera-specific properties.

Some embodiments, instead of using multiple camera-specific constants to calculate the modified quantization parameters as described above, use only a single camera-specific coefficient to calculate the quantization parameters. By performing the calculation, the rate control specific to the camera is executed. For example, given the visual masking attributes φ _frame and φ _MB and the quantization parameter QP _frame , some embodiments use a single camera-specific coefficient μ to calculate the macroblock quantization parameters as follows: Is calculated.

QP _MB = μ * (φ _frame −φ _MB ) + QP _frame (5)
To calculate Equation (5), some embodiments use the frame and macroblock complexity measures as the visual masking attributes φ _frame and φ _MB , respectively.

Some embodiments apply different camera-specific coefficients to the calculation of QP _MB . For example, in some embodiments, QP _MB is calculated as follows.

QP _MB = ρ · (1−φ _MB / φ _frame ) · QP _frame + QP _frame (6)
In the above equation, ρ is a coefficient adapted to camera-specific characteristics.

As described above, state buffer 3015 stores encoding state information that two different rate controller instances 3020 and 3040 may share during a video conference to obtain better encoding results from rate control operations. Target bit rate R _T is, in some embodiments, an example of such shared state information. This rate is the desired bit rate for encoding a sequence of frames. Typically, this bit rate is expressed in units of bits per second, see Section III. It is determined based on a process such as the process described above in B.

  As described above, the rate controller of some embodiments uses the target bit rate to calculate the frame and / or macroblock quantization parameter (s) QP for output to video encoder 3030. For example, some embodiments use the current target bit rate to calculate a quantization adjustment parameter that derives a quantization parameter QP for a macroblock and / or frame. In some embodiments, the quantization adjustment parameter is expressed as a ratio calculated by dividing either the previous frame bit rate or the moving average of the previous frame bit rate by the current target bit rate. You. In other embodiments, this tuning parameter is not calculated exactly as such, but rather is more generally (1) proportional to the bit rate of the previous frame or a running average of the bit rate of the previous frame. And (2) inversely proportional to the current target bit rate.

  After calculating such a quantization adjustment parameter, the rate controller of some embodiments uses this parameter to calculate and adjust the macroblock and / or frame quantization parameter (s). One way to make such an adjustment is to multiply the calculated macroblock and / or frame quantization parameter (s) by a quantization adjustment parameter. Another way to make this adjustment is to calculate a quantization parameter offset value from the quantization adjustment parameters, and then apply this parameter offset to the calculated macroblock and / or frame quantization parameter (s) (eg, , Subtraction). Thereafter, the rate controller of these embodiments outputs the adjusted macroblock and / or frame quantization parameter (s) to video encoder 3030.

  In other embodiments, the rate controller uses the target bit rate to calculate other parameters used in the rate control operation. For example, in some embodiments, the rate controller uses this bit rate to change the visual masking strength of a macroblock or frame.

G. FIG. Networking Manager FIG. 31 conceptually illustrates a software architecture for the networking manager 3100 of some embodiments, such as the networking manager 1614 shown in FIG. As described above, the networking manager 3100 may provide a network connection between the dual camera portable device on which the networking manager 3100 operates and a remote device (eg, establishing a connection, monitoring a connection, adjusting a connection, disconnecting a connection in a video conference). Etc.). During a video conference, the networking manager 3100 of some embodiments also processes data for transmission to the remote device and processes data received from the remote device.

  As shown in FIG. 31, the networking manager 3100 includes a session negotiation manager 3105, a transmitter module 3115, a general-purpose transmission buffer 3120, a general-purpose transmission buffer manager 3122, a virtual transport protocol (VTP) manager 3125, and a receiver. It includes a module 3130 and a media transport manager 3135.

  The session negotiation manager 3105 includes a protocol manager 3110. The protocol manager 3110 ensures that the transmitter module 3115 transmits data to the remote device during the video conference using the appropriate communication protocol and enforces the rules of the communication protocol used. Some embodiments of the protocol manager 3110 include several communication protocols, such as Real-time Transport Protocol (RTP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Hypertext Transfer Protocol (HTTP), among others. Support.

  The session negotiation manager 3105 is responsible for establishing connections between the dual camera portable device and one or more remote devices participating in the video conference and disconnecting those connections after the conference. In some embodiments, the session negotiation manager 3105 determines whether the dual camera portable device and the remote device (eg, video stream and / or audio) are in a video conference (eg, using Session Initiation Protocol (SIP)). It is also responsible for establishing multimedia communication sessions (for sending and receiving streams).

  The session negotiation manager 3105 also receives feedback data from the media transport manager 3135 and, based on the feedback data, operates the universal transmission buffer 3120 through the general transmission buffer manager 3122 (eg, transmits or discards packets / frames). Or do). This feedback may include, in some embodiments, one-way latency and bandwidth estimation bit rate. In other embodiments, the feedback includes packet loss information (e.g., determined based on receiving packets transmitted to and acknowledgments from the remote device in a video conference) and round trip delay time. The session negotiation manager 3105 determines, based on information from the media transport manager 3135, whether too many packets are being transmitted and causes the general-purpose transmission buffer manager 3122 to send fewer packets to the general-purpose transmission buffer 3120. (Ie, adjusting the bit rate as described in FIG. 15).

  The transmitter module 3115 reads the encoded image from a video buffer (e.g., as a bitstream) (e.g., buffer 1612 in FIG. 16) and transmits the video in a remote teleconference via the generic transmission buffer 3120 and the virtual transport protocol manager 3125. Packetize the image for transmission to the device. The manner in which the encoded image is created and sent to the transmitter module 3115 can be based on instructions or data received from the media transport manager 3135 and / or the session negotiation manager 3105. In some embodiments, the packetization of the image comprises decomposing the received bitstream into groups of packets each having a particular size (ie, a size specified by the session negotiation manager 3105 according to a particular protocol); And adding any necessary headers (eg, address headers, protocol specification headers, etc.).

  The general-purpose transmission buffer manager 3122 controls the operation of the general-purpose transmission buffer 3120 based on data and / or instructions received from the session negotiation manager 3105. For example, general purpose transmission buffer manager 3122 can be instructed to instruct general purpose transmission buffer 3120 to send data, stop sending data, discard data, etc. As described above, in some embodiments, the remote device participating in the conference appears to have discarded the packet, but this will be recognized based on the acknowledgment received from the remote device. . To reduce packet discard, the generic transmission buffer manager 3122 may be instructed to transmit packets to the remote device at a slower rate.

  The general-purpose transmission buffer 3120 stores data received from the transmitter module 3115 and transmits the data to the remote device via the VTP manager 3125. As described above, general purpose transmission buffer 3120 may discard data (eg, video images) based on instructions received from general purpose transmission buffer manager 3122.

  In some embodiments, RTP is used to communicate data packets (eg, audio and video packets) over UDP during a video conference. Other embodiments use RTP to communicate data packets over TCP during a video conference. In different embodiments, other transport layer protocols can be used.

  Some embodiments define a particular communication channel between two portable devices by a pair of port numbers (ie, a source port number and a destination port number). For example, one communication channel between portable devices may be defined by a port number pair (eg, source port 50 and destination port 100), and another different communication channel between portable devices may be defined by another different port number pair (eg, , Source port 75 and destination port 150). Some embodiments also use a pair of Internet Protocol (IP) addresses in defining a communication channel. In some embodiments, different communication channels are used to transmit different types of data packets. For example, video data packets, audio data packets, and control signal data packets may be transmitted on separate communication channels. Thus, the video communication channel carries video data packets and the audio communication channel carries audio data packets.

  In some embodiments, the control communication channel is for transmitting messages between a local portable device and a remote device during a video conference. Examples of such message transmissions include sending and receiving requests, notifications, and acknowledgments to such requests and notifications. Another example of message transmission involves sending a remote control command message from one device to another. For example, a remote control operation described below (e.g., instructing a device to send only images from a particular camera, or to capture only images with a particular camera) requires that the local device It may be performed by sending instructions from a local device to a remote device over a control communication channel to remotely control operation of the device. Different embodiments implement the control communication using various protocols such as Real-time Transport Control Protocol (RTCP), RTP extensions, SIP, etc. For example, some embodiments use the RTP extension to relay one set of control messages between two portable devices in a video conference and relay another set of control messages between the mobile devices during the video conference. To use SIP packets.

  The VTP manager 3125 of some embodiments allows different types of data packets designated to be transmitted over different communication channels (eg, using different port number pairs) over a single communication channel. (Eg, using the same port number pair). One technique for doing this is to identify the data packet type, identify the designated communication channel over which the data packet is to be transmitted by extracting the specified port number pair of the data packet, And through changing a port number pair of a data packet to be a single communication channel port number pair (ie, all data packets are transmitted over the same port number pair) Including specifying the data packets to be transmitted.

  To keep track of the original port number pair for each type of data packet, some embodiments store a mapping of the original port number pair to the data packet type. Thereafter, some of these embodiments use the packet type field of the protocol to distinguish different packets that are multiplexed on one communication channel. For example, some embodiments that allow the VTP manager to multiplex audio, video, and control packets into one RTP stream may be used to transmit audio, video, and audio packets transmitted over a single RTP channel to other devices in a video conference. The RTP packet type field is used to distinguish control packets. In some of these embodiments, the VTP manager also routes control message transmissions in SIP packets to other devices.

  In some embodiments, to distinguish between different packets communicated using different protocols (eg, to distinguish between packets carried using RTP and packets carried using SIP). And identifying and examining the data packet signature (ie, packet header format). In such an embodiment, after data packets of different protocols are determined, the fields of data packets using the same protocol (eg, audio and video data using RTP) are examined as described above and different Identify the data type. In this way, the VTP manager 3125 transmits different data packets intended to be transmitted over different communication channels over a single communication channel.

  Although one means of combining different types of data over a single communication channel has been described above, other embodiments utilize other techniques to multiplex different packet types into a single communication stream. For example, one technique of some embodiments includes tracking the original port number pair of a data packet, and storing the original port number pair in the data packet itself for later extraction. There are still other ways to combine different types of data between two video conference participants into one port pair channel.

  When the VTP manager 3125 receives a data packet from a remote device over a virtual communication channel, the VTP manager 3125 examines the signature of the data packet to identify different packets sent using different protocols. Such a signature can be used to distinguish SIP packets from RTP packets. In some embodiments, the VTP manager may also be able to demultiplex multiple different types of packets (eg, audio packets, video packets, and control packets) that are multiplexed into a single virtual channel. Use some or all packet type fields. After identifying these different types of packets, the VTP manager associates each different type of packet with its corresponding port pair number based on the mapping between the port pair numbers it manages and the packet type. After that, the VTP manager 3125 changes the port number pair of the data packet with the identified port number pair, and transfers the data packet to be depacketized. In other embodiments that use different techniques to multiplex different packet types onto a single channel, the VTP manager uses different techniques to analyze the packets.

  The VTP manager 3125 creates a single virtual communication channel (eg, a single port number pair) by using such techniques to multiplex and demultiplex different packets, and Video data, audio data, and control signal data are transmitted through one virtual communication channel, and audio packets, video packets, and control packets are received from a remote device through the single virtual communication channel. Thus, from a network perspective, data is transmitted over this single virtual communication channel, but from a session negotiation manager 3105 and protocol manager 3110 perspective, video data, audio data and control signals are transmitted over different communication channels. Data is transmitted.

  Like the image transmitted to the remote device in the video conference, the image transmitted from the remote device in the video conference is received in a packet format. Receiver module 3130 receives the packet, depackets the packet, reconstructs the image, and then stores the image to be decoded in a video buffer (eg, buffer 1616 in FIG. 16). In some embodiments, depacketizing the image includes removing any headers and reconstructing a bitstream having only image data (and possibly size data) from the packet.

  The media transport manager 3135 may provide feedback data (eg, one-way latency, bandwidth estimation bit rate, packet loss data) received from the network to dynamically and adaptively adjust the rate (ie, bit rate) of data transmission. , Round trip delay time data, etc.). The media transport manager 3135 also controls error resilience based on the processed feedback data in some other embodiments and coordinates other operations of the video conferencing module 1602 such as scaling, resizing, and encoding. Feedback data may be sent to the video conference manager 1604 to do so. In addition to having the general purpose transmission buffer discard packets if the remote device cannot handle all of the packets in the conference, the video conferencing module and the encoder also allow the video conferencing module to transmit fewer packets per image. Can be used for encoding.

  In some embodiments, the media transport manager 3135 may also control other aspects of the device, such as power consumption and thermal levels, that may affect how the operating power mode of the camera is configured, as described above. May be monitored. This data may be used as an additional input to the feedback data (eg, if the device is too hot, media transport manager 3135 may attempt to slow down processing).

  Some example operations of the networking manager 3100 will now be described with respect to FIG. Transmission of an image captured by a camera of a dual camera portable device to a remote device in a video conference will be described first, followed by reception of an image from the remote device. Transmitter module 3115 reads the encoded image from buffer 1612, which can be transmitted to a remote device in a video conference.

  The protocol manager 3110 determines the appropriate protocol to use (eg, RTP for transmitting audio and video), and the session negotiation manager 3105 notifies such a protocol to the transmitter module 3115. Next, the transmitter module 3115 packetizes the image and sends the packetized image to the general-purpose transmission buffer 3120. The general purpose transmission buffer manager 3122 receives the command from the session negotiation manager 3105 and instructs the general purpose transmission buffer 3120 to transmit or discard the image. VTP manager 3125 receives packets from universal transmission buffer 3120 and processes the packets to transmit the packets to a remote device over a single communication channel.

  Upon receiving an image from a remote device, the VTP manager 3125 receives the packetized image from the remote device through a single virtualized communication channel, and the communication channel assigned to receive the image (eg, , Video communication channel) to direct the image to the receiver module 3130.

  Receiver module 3130 depackets the packets to reconstruct the image and sends the image to buffer 1616 for decoding by decoder 1660. Receiver module 3130 also forwards control signal messages (eg, acknowledgments of packets received from remote devices in a video conference) to media transport manager 3135.

  Some exemplary operations of networking manager 3100 have been described above. These are only illustrative examples. The reason is that various other embodiments perform these or different operations using different modules or with functions spread differently between the modules. In addition, additional operations such as dynamic bit rate adjustment may be performed by modules of networking manager 3100 or other modules.

IV. Coordination and control operations during a meeting Change of picture-in-picture Corner Snaps According to some embodiments of the present invention, a user of a dual camera portable device changes a composite view displayed on a device by moving one or more viewports that form the composite view. One such example is to move the insertion display area of the PIP display. FIG. 32 illustrates such an example performed during a video conference. In a video conference, the user may want to move the foreground insert display area for various reasons, such as when this area blocks the area of interest in the background display area.

  FIG. 32 illustrates the movement of the insertion display area 3240 in the device UI 3205 as five different stages 3210, 3215, 3220, 3225 and 3230 of this UI. The first stage 3210 shows the UI 3205 during a video conference between a local user of the device and a remote user of a remote device. The UI 3205 in FIG. 32 shows the PIP display, which is the same PIP display shown in the fifth stage in FIG. 11 after the video conference has started. In this example, video captured by the local user's device is displayed in the insert display area 3240, and video captured by the remote user's device is displayed in the background display area 3235. As shown, display area 1155 includes a selectable UI item 3245 to end the video conference. In some embodiments, the layout of display area 1155 is the same as the layout of display area 1155 of FIG. 12 described above.

  The second stage 3215 indicates that the user initiates the corner snap operation by selecting the insertion display area 3240. In this example, the selection is made by placing finger 3255 at an arbitrary position in insertion display area 3240. As shown, this selection is displayed as a thick border 3260 of the insertion display area 3240. Different embodiments may indicate such selections in different ways, such as by highlighting display area 3240, oscillating display area 3240, and so on.

  The third step 3220 shows the UI 3205 after the user has begun to move the insertion display area 3240 of the PIP display 3250 from one area in the PIP display 3250 to another area in this display. In this example, insertion display area 3240 has begun to move from the lower left corner of PIP display 3250 to the lower right corner of the display, as indicated by arrow 3265. In this example, the insertion display area 3240 is moved by the user dragging his / her finger 3255 toward the lower right corner of the PIP display 3250 after selecting the insertion display in the second step 3215. Some embodiments provide other techniques for moving the insertion display area 3240 in the PIP display 3250.

  The fourth step 3225 shows the UI 3205 in a state after the user has released his / her finger 3255 from the screen of the device 3200. In this state, the insertion display area 3240 is still moving toward the lower right corner of the PIP display 3250 identified based on the movement of the user's finger in the third stage 3220. In other words, after finger 3255 begins to move insertion display area 3240 toward the lower right corner of PIP display 3250, UI 3205 maintains this movement even after finger 3255 is released. To maintain this movement, the UI 3205 of some embodiments may indicate that the user's drag motion is greater than a certain threshold amount (e.g., a certain distance or a certain distance) before the user releases his or her finger 3255. Greater than the length of time). Otherwise, these embodiments keep the display area 3240 in its original left corner position after slightly or not moving the insertion display area 3240.

  However, while some embodiments allow the insertion display area to be moved even after the user stops dragging before the insertion display area reaches its new position, other embodiments use the insertion display area. Ask the user to maintain his drag action until the zone reaches its new location. Some embodiments provide yet another technique for moving the insertion display area. For example, some embodiments may require the user to specify where the insertion display area 3240 should be directed before the insertion display area 3240 actually moves. In some embodiments, simply tilting the mobile device to a different angle may allow the display area to be slidable and allow for cornering snap operations.

  Fifth stage 3230 shows UI 3205 after insertion display area 3240 has reached its new position at the lower right corner of PIP display 3250. The removal of the thick border 3260 in the fifth step 3230 indicates that the corner snapping operation has been completed.

  To facilitate the movement shown in the third stage 3220, the fourth stage 3225, and the fifth stage 3230 described above, the UI 3205 of some embodiments may allow the user to move the insertion display area toward a corner of the PIP display 3250. Use a snap rule that allows the insertion display area to snap quickly to that corner when moved. For example, if the user drags the insertion display area 3240 beyond a threshold amount toward a particular corner, the UI 3205 of some embodiments may specify the direction of movement of the insertion display area 3240, and It is determined that the threshold amount has been exceeded, and thereafter, the insertion display area 3240 is automatically moved without further input of the next grid point in the UI 3205 where the insertion display area 3240 can be snapped. In some embodiments, the only grid points provided to snap inset display area 3240 are grid points at the four corners of PIP display 3250. Other embodiments provide other grid points in the UI 3205 (eg, in the PIP display 3250) where the inset viewport 3240 can snap (ie, be placed or aligned on a side or vertex of the area 3240). I do.

  Still other embodiments may not employ grid points so that the insertion display area can be located at any point of the PIP display 3250. Still other embodiments provide the ability for a user to turn on or off snapping to a grid point feature in the UI. Further, different embodiments may allow a user to snap to corners using various items, such as icons, in addition to the video captured from the device.

  FIG. 33 illustrates two other examples 3330 and 3335 of a corner snap action in the UI 3205. These other corner snapping actions indicate that the insertion display area 3240 is moved vertically or diagonally within the PIP display 3250 based on the user's vertical or diagonal drag action.

  While FIGS. 32 and 33 show the movement of the insertion display area within the PIP display, other embodiments utilize similar techniques to allow for other types of PIP display or other types of composite display. It will be appreciated by those skilled in the art to move the viewing area. For example, as described further below, some embodiments of the PIP display have more than one foreground insert display, and these insert displays use techniques similar to those described above with respect to FIGS. Can be used to move in the PIP display. Also, some embodiments use a similar technique to move the display area in a composite display (eg, moving a display area from the left side of the screen to the right side of the screen through a user's drag movement). In addition, moving the display area (s) of the composite display causes changes in the image processing operation of the dual camera portable device, such as causing the video conference manager 1604 to re-synthesize the display area in the composite display in response to user input. be able to. As described further below, some embodiments include snap and push in which pressing the first display area from the first position moves the second display area from the third position to the first position. Adopt technology.

2. Rotation Some embodiments rotate the PIP display presented during a video conference when the user of the mobile device used for the video conference rotates the device during the conference. FIG. 34 shows the rotation of the UI 1105 of the device 3400 when the device is rotated from a vertical position to a horizontal position. When the long side of the screen is vertical, the device 3400 is held vertically, and when the long side of the screen is horizontal, the device 3400 is held horizontally. In the example shown in FIG. 34, the UI 1105 rotates from a vertical view optimized for vertical holding of the device to a horizontal view optimized for horizontal holding of the device 3400. This rotation feature allows the user to see the UI 1105 displayed in an upright position whether the portable device 3400 is held vertically or horizontally.

  FIG. 34 shows the rotation of the UI 1105 as six different operating phases 3410, 3415, 3420, 3425, 3430 and 3435. The first stage 3410 shows the UI 1105 during a video conference between a local user of the device and a remote user of a remote device. The UI 1105 of FIG. 34 shows a PIP display 1180 that is the same PIP display as shown in the fifth stage of FIG. 11 after the video conference has been established. In this example, the video captured by the local user's device is displayed in the insert display area 1160, and the video captured by the remote user's device is displayed in the background display area 1170. The display area 1155 below the PIP display 1180 includes a selectable UI item 3485 (e.g., end conference button 3485), and the user can use this UI item to end the video conference (e.g., single finger tap). Through).

  The second stage 3415 shows the UI 1105 after the user has begun to tilt the device 3400 sideways. In this example, the user has begun to tilt device 3400 from being held vertically to being held horizontally, as indicated by arrow 3460. The appearance of the UI 1105 does not change. In other situations, the user may wish to tilt the device 3400 from a horizontally held state to a vertically held state, and in these situations, the UI 1105 may have a horizontally optimized state. Switch from a view to a vertically optimized view.

  The third step 3420 shows the UI 1105 after the device 3400 has been tilted from being held vertically to being held horizontally. In this state, the appearance of the UI 1105 has not yet changed. In some embodiments, the device 3400 is tilted past a threshold amount, beyond which a rotation operation is triggered after being held for a period of time. In the example shown in FIG. 34, it is assumed that the UI 1105 does not rotate due to the threshold amount and the rotation speed until a certain short interval elapses after the device is placed in the horizontal position. Different embodiments have different threshold amounts and waiting periods for triggering the rotation operation. For example, some embodiments may have a lower threshold for triggering a rotation operation such that the UI 1105 always appears to be displayed in an upright position regardless of the orientation of the device 3400. In other embodiments, a user of device 3400 may specify (eg, through a menu preference) when a rotation operation can be triggered. Also, some embodiments may not delay rotation after the device is tilted if the device exceeds a threshold amount. Furthermore, different embodiments may be able to trigger the rotation in different ways, such as by selecting a menu, by switching a switch on the mobile device, and giving a voice command.

  The fourth step 3425 shows the UI 1105 after the rotation operation has started. Some embodiments animate the rotating display area to provide feedback to the user regarding the rotating motion. FIG. 34 shows an example of such an animation. Specifically, FIG. 34 shows that, at its fourth stage 3425, display areas 1180 and 1155 begin to rotate together. Display areas 1180 and 1155 rotate about axis 3465 (ie, the z-axis) passing through the center of UI 1105. Viewing areas 1180 and 1155 rotate the same amount in the opposite direction of rotation of device 3400 (eg, through tilting of device 3400). In this example, the device 3400 has been rotated 90 degrees in the clockwise direction (by moving from a state held vertically to a state held horizontally), and this rotation causes the display areas 1180 and 1155 to rotate counterclockwise. Rotated 90 degrees in the direction. As the display areas 1180 and 1155 rotate, the display areas 1180 and 1155 shrink proportionately to fit the UI 1105 so that the display areas 1180 and 1155 are still entirely visible on the UI 1105. Some embodiments may provide a message indicating the status of the device 3400 (eg, by displaying the word “rotating”).

  The fifth step 3430 shows the UI 1105 after the display areas 1180 and 1155 have been rotated 90 degrees counterclockwise from the vertical view to the horizontal view. At this stage, the display areas 1180 and 1155 have been rotated but not yet expanded to the full width of the UI 1105. The arrow 3475 indicates that at the end of the fifth phase, the display areas 1180 and 1155 begin to expand sideways to fit the entire width of the UI 1105. Different embodiments may not include this step, since the expansion can be performed simultaneously with the rotation in the fourth step 3425.

  A sixth step 3435 shows the UI 1105 after the display areas 1180 and 1155 have been expanded to occupy the entire display of the UI 1105. As mentioned above, other embodiments may implement this rotation differently. In some embodiments, simply rotating the screen of the device beyond a threshold amount may trigger a rotation operation regardless of the orientation of the device 3400.

  Also, other embodiments could provide different animations to show the rotation action. The rotation operation performed in FIG. 34 includes the display areas 1180 and 1155 rotating around the center of the UI 1105. Alternatively, the display areas may be individually rotated about the central axis of these individual display areas. One such approach is shown in FIG. FIG. 35 illustrates an alternative method of animating the rotation of the display areas 1170 and 1160 of the PIP display 1180 of the UI 1105. The PIP display 1180 shown in FIG. 35 is the same PIP display 1180 as shown in FIG.

  FIG. 35 shows the rotation of the PIP display 1180 as six different phases of operation 3410, 3415, 3420, 3525, 3530 and 3535. The first three stages of the operation of the UI 1105 are the same as the first three stages of the operation described with reference to the UI 1105 of FIG. In the third stage for both FIGS. 34 and 35, device 3500 has transitioned from a vertically held state to a horizontally held state, and rotation of UI 1105 has not yet started.

  The fourth step 3525 shows an alternative way to animate the rotation. At this stage, the rotation operation has started. Specifically, in a fourth step 3525, it is shown that the display areas 1170 and 1160 begin to rotate. Viewports 1170 and 1160 respectively rotate about axes 3567 and 3565 (ie, the z-axis) passing through the respective centers of the displayportions. Display areas 1170 and 1160 rotate the same amount in the opposite direction of rotation of device 3500 (eg, due to tilt of device 3500). Since device 3500 has been rotated 90 degrees clockwise (by moving from a vertically held state to a horizontally held state), similar to that shown in fourth stage 3425 of FIG. 34 above. , The display areas 1170 and 1160 are rotated 90 degrees counterclockwise. As the display areas 1170 and 1160 rotate, the display areas 1170 and 1160 shrink proportionally to fit the UI 1105 so that the display areas 1170 and 1160 are still fully visible on the UI 1105.

  Fifth stage 3530 shows UI 1105 after each of display areas 1170 and 1160 has been rotated 90 degrees counterclockwise from portrait view to landscape view. At this stage, the display areas 1170 and 1160 have rotated, but have not yet expanded to the full width of the UI 1105. Further, display area 1160 has not moved to its final position. The final position of the insertion display area 1160 in the PIP display 1180 is determined by the position of the insertion display area 1160 in the PIP display 1180 shown in the first step 3410 (for example, the insertion display area 1160 at the lower left corner of the PIP display 1180). You. At this stage, the insertion display area 1160 is still in the upper right corner of the UI 1105.

  The arrow 3580 indicates that at the end of the fifth stage 3530, the display areas 1170 and 1160 begin to expand sideways until the main display area 1170 fits the entire width of the UI 1105 of the horizontally held device. Further, arrow 3575 indicates that insertion display area 1160 slides to the lower left corner of PIP display 1180.

  Different embodiments may implement this differently. In some embodiments, the movement of the insertion display area 1160 may be performed simultaneously or sequentially with the expansion of the main display area 1170. Further, some embodiments may resize the insertion display area 1160 before, during, or after expansion of the main display area 1170 to generate a new PIP display 1180. In this example, display area 1155 is hidden while display areas 1160 and 1170 are being rotated. However, in some embodiments, display area 1155 may remain in UI 1105 during rotation and rotate with display areas 1160 and 1170.

  The sixth step 3535 shows the UI 1105 after the insertion display area 1160 has reached its new position and the display areas 1160 and 1170 have been appropriately expanded to fit the full width of the UI 1105. In this example, the insertion display area 1160 is now at the lower left corner of the PIP display 1180 and overlaps the main display area 1170. The PIP display 1180 now has the same display configuration as the PIP display 1180 in the first stage 3410. The appearance of the display area 1155 below the PIP display 1180 at the sixth stage indicates that the rotation operation has been completed. As described above, by simply rotating the screen of the device beyond the threshold amount, the rotation operation can be triggered regardless of the orientation of the device 3500.

  In the example described above with reference to FIGS. 34 and 35, the orientation of the display area 1170 also changes (ie, from vertical to horizontal). That is, after the display area 1170 is rotated in the third step 3420, the orientation of the display area 1170 is changed from vertical to horizontal by expanding the PIP display 1180 horizontally so as to fill the entire UI 1105. In some embodiments, when the device 3500 is rotated, the video captured by the remote device rotates, but the orientation of the viewing area displaying the video captured by the remote device remains unchanged. One such example is shown in FIG. This figure is similar to FIG. 35, except that the video displayed in display area 1170 rotates, but display area 1170 remains displayed vertically.

  FIG. 36 also shows an example of a rotation operation where the display area 1155 remains in the same position (rather than rotating and expanding horizontally to fill the PIP display 1180 as shown in FIG. 35). Further, this figure includes a layout of the display area 1155 which is the same as the layout of the display area 1155 of FIG. 12 described above. As shown, when the device 3500 is rotated in steps 3640, 3645, 3650, 3655, 3865, and 3690, the display area 1155 remains in the same position.

  In some embodiments, the orientation of the viewport displaying the video captured by the local device may be different from that of the local device after a rotation operation has been performed on the local device (rather than remaining the same orientation as shown in FIG. 35). Provide a rotating motion that changes to reflect the orientation of the FIG. 36 shows an example of such a rotating operation of the UI 1105 as six different stages 3640, 3645, 3650, 3655, 3885 and 3690. In this figure, the first stage 3640 shows an inset display area 1160 that displays video captured by the camera of device 3500 in portrait orientation. The second stage 3645 and the third stage 3650 are similar to the second stage 3415 and the third stage 3420 of FIG. 35 because they show the tilt of the device 3500 at various stages of the rotation operation. At this point, the camera of device 3500 has captured the image in landscape orientation. To illustrate this transition, some embodiments provide the animations shown in fourth stage 3655 and fifth stage 3685, while other embodiments do not provide any animations.

  In the fourth step 3655, the image displayed in the insertion display area 1160 is rotated because the insertion display area 1160 is rotated sideways due to the inclination of the device 3500 in the second step 3445 and the third step 3650, but the insertion is performed. Display area 1160 itself is not rotated. In a fifth step 3685, the rotated image in the insertion display area 1160 is horizontally expanded to fill the insertion display area 1160, and the insertion display area 1160 is the same as the insertion display area 1160 in the PIP display of the first step 3640. In order to place the insertion display area 1160 at a relative position, it starts moving toward the lower left area of the PIP display 1180.

  In some embodiments, the orientation of the viewing area displaying the video captured by the remote device also changes to reflect the orientation of the remote device after the rotation operation has been performed on the remote device. FIG. 37 illustrates four different stages of the UI 1105 of the device 3500, in which (1) the orientation of the display area (in this example, the display area 1160) that displays the video captured by the local device is rotated. Changes to reflect the orientation of the local device after it has been executed on the local device, and (2) the orientation of the display area (in this example, display area 1170) that displays the video captured by the remote device is Changes to reflect the orientation of the remote device after the rotation operation is performed.

  In the first step 3705, the UI 1105 is the same as the UI 1105 in FIG. Specifically, since device 3500 is shown in portrait orientation and the remote device is in portrait orientation (not shown), first stage 3705 shows display areas 1160 and 1170 in portrait orientation. From a first step 3705 to a second step 3710, a rotation operation is performed on the local device by rotating the device 3500 from an upright position to a sideways position by 90 degrees. The second step 3710 shows the UI 1105 after the rotation operation of the device 3500 has been completed. At this stage, the video displayed in display areas 1170 and 1160 has been rotated to the upright position. However, because the rotation operation was performed only on the local device (ie, device 3500), only the display area 1160 of the locally captured video rotated from portrait to landscape. The display area 1170 remains vertical.

  From a second stage 3710 to a third stage 3715, a rotation operation is performed on the remote device by rotating the remote device from an upright position to a sideways position (not shown). The third step 3715 shows the UI 1105 after the rotation operation of the remote device has been completed. At this stage, since the rotation operation is executed only by the remote device, the display area 1170 of the video displayed on the display area 1170 and the video captured remotely is rotated from portrait to landscape. Thus, this stage of the UI 1105 displays both the locally captured video display area 1160 and the remotely captured video display area 1170 in landscape orientation.

  From the third step 3715 to the fourth step 3720, a rotation operation is performed on the local device by rotating the device 3500 from the sideways position to the upright position by 90 degrees. The fourth step 3720 shows the UI 1105 after the completion of this rotation operation. In this fourth stage 3720, the video displayed in display areas 1160 and 1170 has been rotated to an upright position. However, since the rotation operation was performed only on the local device (ie, device 3500), only the display area 1160 of the locally captured video is rotated from landscape to portrait. Display area 1170 remains horizontal.

  From a fourth step 3720 to a first step 3705, a rotation operation is performed on the remote device by rotating the remote device 90 degrees from the horizontal position to the upright position (not shown). In this case, the first stage 3705 shows the display area 1170 after the completion of this rotation operation. Therefore, the UI 1105 at this stage shows the display areas 1160 and 1170 in portrait orientation. While FIG. 37 shows a sequence of different rotation operations, other embodiments can perform any number of rotation operations in any number of different sequences.

  FIGS. 34, 35, 36, and 37 illustrate rotation operations performed on local and remote devices during a video conference. If the rotation operation is performed on the local mobile device, some embodiments may also allow the remote device to make any changes to the local device's video (such as rotating the viewing area displaying the local device's video). Notify the remote device of the rotation action to execute. Similarly, if a rotation operation is performed on the remote device, the remote device notifies the local device of the operation so that the local device can make any changes to the remote device's video. Some embodiments provide a control communication channel for communicating notifications of rotational movement between a local device and a remote device during a video conference.

  While FIGS. 34, 35, 36 and 37 illustrate different ways in which the animation of rotation can be performed, it will be appreciated by those skilled in the art that other embodiments may display the animation of rotation in other different ways. Will be understood. In addition, the animation of the rotation action causes the video conferencing manager 1604 to re-synthesize the display area (s) at different angles in the UI 1105 and scale the image displayed in the display area (s), such as an image on the local mobile device. Changes to processing operations can be triggered.

3. Adjusting Window Size Some embodiments allow a user of a mobile device to adjust the size of the inset view area of a PIP display presented during a video conference. Different embodiments provide different techniques for resizing the inset display area. FIG. 38 shows one technique for resizing the insertion display area. In this approach, the user of the mobile device adjusts the size of the insertion display area by selecting a corner of the insertion display area and then expanding or contracting the insertion display area.

  In FIG. 38, the UI 3800 of the mobile device 3825 presents a PIP display 3865 during a video conference with a remote user of another mobile device. This PIP display 3865 includes two video displays, a background main display area 3830 and a foreground insert display area 3835. While the background main display area 3830 occupies most of the PIP display 3865, the foreground insertion display area 3835 is smaller than the background main display area 3830 and overlaps it. In this example, the background main display area 3830 presents a video of a person with a guitar, the person being video captured by the front camera of the remote device or the video captured by the rear camera of the remote device. It is assumed that there are people. The foreground insert viewport 3835 presents a video of the person wearing the hat, which in this example is the person whose video is being captured by the front camera of the local device or the video is being captured by the rear camera of the local device. You are assumed to be a person. Below the PIP display 3865 is a display area 1155 that includes a selectable UI item 3860 (eg, button 3860) labeled “End Conference” that allows the user to end the video conference by selecting.

  This PIP display 3865 is just one way to present a composite view of the video being captured by the remote and local devices. Some embodiments may provide other composite views. For example, instead of having a larger background display for the video from the remote device, the larger background display 1170 is for the video from the local device and the smaller foreground insert display is for the video from the remote device. It can be. Similarly, according to some embodiments, local video and remote video are displayed in two side-by-side display areas (eg, left and right display windows or top and bottom display windows) or two diagonally aligned display areas in UI 3800. Can be done. In some embodiments, the type of PIP display or default display mode may be specified by the user. In other embodiments, the PIP display may also include a larger background display and two smaller foreground inserts.

  FIG. 38 shows the size change operation as four operation stages of the UI 3800. In the first stage 3805, the foreground insert display 3835 is significantly smaller than the background main display area 3830. Also, in this example, the foreground insertion display area 3835 is at the lower right corner of the PIP display 3865. In other examples, foreground insert display area 3835 may be of a different size or may be in a different area within PIP display 3865.

  In a second step 3810, a resize operation is started. In this example, the operation is initiated by the user selecting a corner of the insertion display area 3835 whose resizing is desired (eg, by pressing the upper left corner of the insertion display area 3835 with a finger 3840). The second stage 3810 of the UI 3800 shows this selection as a thick border 3845 of the insertion display area 3835. At this stage, the user may enlarge or shrink the insertion display area 3835 (eg, by dragging his finger 3840 away from the insertion display area 3835 on the PIP display 3865 or toward the insertion display area 3835). be able to.

  The third step 3815 is by moving his finger 3840 away from the insertion display area 3835, as shown by arrow 3850 (ie, in this example, diagonally toward his upper left corner of UI 3800). 21 shows the UI 3800 after the user has begun to expand the inset display area 3835 (by moving. Also, as shown by arrow 3855, movement of finger 3840 expands insertion display area 3835 in a manner that is proportional in both height and width. In another example, the user can shrink the insert viewport 3835 using the same technique (ie, by dragging a finger toward the insert viewport 3835).

  The fourth step 3820 displays the UI 3800 after the resizing of the insertion display area 3835 is completed. In this example, when the insertion display area 3835 reaches the desired size, the user stops dragging his / her finger 3840 and removes his / her finger from the PIP display 3865, thereby completing the size change of the insertion display area 3835. As a result of this procedure, the resized insertion display area 3835 is larger than its original size in the first stage 3805. Removal of the thick border 3845 indicates that the insertion display area resizing operation has been completed at this point.

  Some embodiments provide other techniques for allowing a user to resize the insert viewport 3835 in the PIP display 3865 during a video conference. FIG. 39 illustrates one such other technique. This figure shows that the end of the insertion display area 3835 is selected (ie, one of the sides of the display area 3835), and then the insertion display area 3835 is resized by expanding or contracting the insertion display area 3835. Here are the techniques for doing so.

  FIG. 39 shows this size changing operation as four operation stages of the UI 3800 of FIG. The first step 3805 in FIG. 39 is the same as the first step 3805 in FIG. Specifically, at this stage, the UI 3800 of the device 3925 shows a PIP display 3865 having a larger background main display area 3830 and a smaller foreground insertion display area 3835 at the lower right corner of the PIP display 3865. Although FIGS. 38 and 39 show two different techniques for resizing the insert viewport 3835 with the same UI 3800, some embodiments do not provide both of these techniques with the same UI. Will be understood by those skilled in the art.

  The second step 3910 indicates the start of the resize operation. In this example, the user initiates an action by selecting a side of the insertion display area 3835 for which the user wishes to resize (eg, by placing a finger 3840 on the top or side edge of the insertion display area 3835). In this example, the user places his or her finger 3840 at the top of insertion display area 3835 to make this selection. The second stage 3910 shows this selection as a thick border 3845 of the insertion display area 3835.

  The third step 3915 is when the user moves his finger 3840 away from the insertion display area 3835 (ie, vertically toward the top of the PIP display 3865), as indicated by arrow 3950, and 3C shows the UI 3800 after starting to expand 3835. Also, as shown by arrow 3955, movement of finger 3840 expands insertion display area 3835 in a manner that is proportional in both height and width. In another example, the user can shrink the insertion display area 3835 using the same technique (eg, by dragging a finger 3840 toward the insertion display area 3835).

  The fourth step 3920 displays the UI 3800 after the resizing of the insertion display area 3835 is completed. In this example, when the insertion display area 3835 reaches the desired size, the user stops dragging his / her finger 3840 and removes his / her finger 3840 from the display screen of the device to complete the resizing of the insertion display area 3835. I do. As a result of this procedure, the resized insertion display area 3835 is larger than its original size in the first stage 3805. The deletion of the thick border 3845 indicates that the insertion display area size change operation has been completed at this point.

  As shown by FIGS. 38 and 39, some embodiments adjust the size of the insertion display area 3835 proportional to the height and width in response to a drag operation. In other embodiments, the user may be able to adjust the height and / or width of the insertion display area 3835 without affecting other attributes. FIG. 40 shows an example of such a size changing method.

  Specifically, FIG. 40 illustrates that when one of the edges of the insertion display area 3835 is selected and moved horizontally or vertically, the UI 3800 of FIG. 40 causes the insertion display area 3835 to be moved horizontally and / or vertically. 38 shows a UI 3800 for a portable device 4025 that is similar to the UI 3800 in FIG. To simplify the description of UI 3800, FIG. 40 shows a PIP display 3865 on UI 3800 that is similar to PIP display 3865 of FIG. 38, except that at this point insert display area 3835 is at the upper right corner of PIP display 3865. . PIP display 3865 includes two video displays, a background main display area 3830 and a foreground insert display area 3835. In this example, the background main display area 3830 presents the video being captured by the front or rear camera of the remote device. The foreground insert display area 3835 presents video captured by the front or rear camera of the local device.

  Like FIG. 38, FIG. 40 illustrates the resize operation as four operating stages of the UI 3800. The first step 4005 is similar to the first step 3805 in FIG. 38, except that at this point the insertion display area 3835 is at the upper right corner. The other three steps 4010, 4015 and 4020 indicate that the selection and movement of the lower end of the insertion display area 3835 has expanded the insertion display area 3835 only vertically, without affecting the width of the insertion display area 3835. Except for three stages 3910, 3915 and 3920.

  38, 39 and 40 provide an exemplary embodiment in which the user can resize the insertion display area 3835 of the PIP display 3865 by selecting a corner or side of the insertion display area 3835. Some embodiments provide other techniques for resizing the insertion window 3835. For example, FIG. 41 illustrates that insertion display area 3835 can be resized by selecting inside insertion display area 3835 according to some embodiments. In this technique, the user adjusts the size of the insertion display area 3835 by placing two fingers 4155 and 4156 on the screen and dragging the fingers either away from or closer to each other.

  In FIG. 41, the UI 3800 of the mobile device 4140 provides a PIP display 3865 during a video conference with a remote user of another mobile device. To simplify the description of UI 3800, FIG. 41 shows a PIP display 3865 on this UI 3800 that is similar to PIP display 3865 in FIG.

  FIG. 41 shows the size changing operation as seven operation stages of the UI 3800. The first four stages 3805, 4110, 4115 and 4120 show expansion of the insertion display area 3835, while the last three stages show contraction of the insertion display area 3835. The first step 3805 in FIG. 41 is the same as the first step 3805 in FIG. Specifically, at this stage, UI 3800 shows a PIP display 3865 having a larger background main display area 3830 and a smaller foreground insertion display area 3835. In this example, the background main display area 3830 presents the video being captured by the front or rear camera of the remote device. The foreground insert display area 3835 presents video captured by the front or rear camera of the local device.

  The second step 4110 shows the UI 3800 after the resize operation has begun. In this example, the user initiates an action by selecting the insertion display area 3835 for which the user wishes to resize (eg, by placing two fingers 4155 and 4156 within the insertion display area 3835). The second stage 4110 of the UI 3800 shows this selection as a thick border 4190 of the insertion display area 3835.

  Third stage 4115 moves fingers 4155 and 4156 away from each other, as indicated by arrow 4160 (ie, moves finger 4155 toward the upper left corner of PIP display 3865 and moves finger 4156 to PIP display 3865). Of the UI 3800 after the user has begun to expand the inset display area 3835 (by moving toward the lower right corner of the UI). As indicated by arrow 4165, movement of fingers 4155 and 4156 expands insertion display area 3835 proportionally in both height and width.

  In a fourth step 4120, the UI 3800 after the resizing of the insertion display area 3835 is completed is displayed. In this example, the user completes the size change of the insertion display area 3835 by stopping the drag of the fingers 4155 and 4156 and releasing the finger 4155 and 4156 from the display screen of the device. As a result of this procedure, the resized insertion display area 3835 is larger than its original size in the first stage 3805. Deletion of the thick border 4190 indicates that the insertion display area size changing operation has been completed at this stage.

  In a fifth step 4125, the user reselects the insertion display area 3835 by placing two fingers 4155 and 4156 on the insertion display area 3835. The sixth step 4130 shows the UI 3800 after the user has begun to shrink the insertion display area 3835 by moving the fingers 4155 and 4156 closer together, as indicated by the arrow 4170. As indicated by arrow 4175, movement of fingers 4155 and 4156 causes insertion display area 3835 to shrink in proportion to both height and width.

  The seventh step 4135 is the same as the fourth step 4120 in FIG. 41 except that the size of the insertion display area 3835 is reduced as a result of the operation. Removal of the thick border 4190 indicates that the insertion display area resizing operation has been completed at this stage.

  The above description of FIGS. 38-41 shows some exemplary user interfaces that allow the user to resize the inset display area of the PIP display. In some embodiments, resizing the inset view area causes the video conferencing manager 1604 to change the scale and composition of the inset view area in the PIP display in response to user input, such that Changes to image processing operations occur. Further, in some embodiments, the layout of the display area 1155 in FIGS. 38 to 41 is the same as the layout of the display area 1155 in FIG. 12 described above.

4. Identifying Regions of Interest According to some embodiments, the user may be able to perform image processing (e.g., image processing manager 1608 of FIG. 16), encoding (e.g., encoder 1655 of FIG. 16), mobile device and its camera during a videoconference. Regions of interest (ROIs) in the video displayed during the video conference can be identified to change the behavior, or a combination thereof. Different embodiments provide different techniques for identifying such regions of interest in a video. FIG. 42 illustrates a user interface of some embodiments for identifying regions of interest in a video to enhance the image quality of the video.

  In FIG. 42, UI 4200 of mobile device 4225 presents PIP display 4265 during a videoconference with a remote user of another mobile device. The PIP display of FIG. 42 is substantially similar to that of FIG. Specifically, the PIP display 42 of FIG. 42 includes two video displays, a background main display 4230 and a foreground insert display 4235. In this example, the background main display area 4230 presents a video of the person wearing the tree and hat, which is captured by the tree and the person or the rear camera of the person or remote device whose video is being captured by the front camera of the remote device. It is assumed that the trees and people whose videos are being captured. The foreground insert display 4235 presents a male video, which in this example is a male whose video is being captured by the front camera of the local device or a male whose video is being captured by the rear camera of the local device. is assumed. Below the PIP display is a display area 1155 that includes a selectable UI item 4260 (eg, button 4260) labeled “End Conference” that allows the user to end the video conference by selecting.

  This PIP display is just one way to present a composite view of the video being captured by the remote and local devices. Some embodiments may provide other composite views. For example, instead of having a larger background display for the video from the remote device, the larger background display may be that of the video from the local device and the smaller foreground insert display may be from the remote device. Video. Similarly, according to some embodiments, local video and remote video are displayed in two side-by-side display areas (eg, left or right display windows or top and bottom display windows) or two diagonally aligned display areas in a UI. Can be done. In other embodiments, the PIP display may also include a larger background display and two smaller foreground inserts. In some embodiments, the type of PIP display or default display mode may be specified by the user.

  FIG. 42 illustrates the ROI identification operation as four stages of operation of the UI 4200. As shown in the first stage 4205, the video presented on the background display 4230 has very low quality (ie, the video image is blurred). In this example, the user of the mobile device 4225 desires to identify the area of the background display 4230 where the human face 4270 is displayed as a region of interest.

  In a second step 4210, an operation of identifying a region of interest is started. In this example, by selecting an area in the video presented in the background display 4230 that the user wishes to identify as a region of interest (eg, around a person's face 4270 displayed in the background display 4230) The action is started (by tapping the screen of the device with the finger 4250 at the location).

  As shown in the third step 4215, upon selection of the user's area, the UI 4200 draws an enclosure 4275 (eg, dashed square 4275) surrounding the area of the user's selection. The fourth step 4220 displays the UI 4200 after the region of interest has been identified. As a result of this process, the quality of the video in the region of interest is substantially improved from the quality in the first stage 4205. Removal of box 4275 indicates that the ROI selection operation has been completed at this point. In some embodiments, the ROI identification process also causes the same change of the same video displayed on the remote device as it does for the local device 4225. In this example, for example, the image quality in the region of interest of the same video displayed on the remote device is also substantially improved.

  In some embodiments, the user may move the finger 4250 toward the upper right corner of the screen or shrink the shroud 4275 in a third step 4215 (eg, press the display with the finger 4250 and stretch the shroud 4275). Enclosure 4275 may be stretched or shrunk (by moving finger 4250 toward the lower left corner of the screen). Also, according to some embodiments, the user may reposition the enclosure 4275 in a third step 4215 (eg, by pressing the display with a finger 4250 and moving the finger 4250 horizontally or vertically on the display). it can. In some other embodiments, the selection of the area may prevent the UI 4200 from drawing the enclosure 4275 in the third step 4215.

  Other embodiments provide different techniques for allowing a user to identify a region of interest in a video. FIG. 43 illustrates one such other technique. In FIG. 43, the user identifies a region of interest by drawing a shape that is a boundary of the region. In this example, the shape is rectangular, but can be any other shape (eg, any other polygon, circle, ellipse, etc.). Some embodiments provide this alternative technique of FIG. 43 in a device UI that also provides the technique shown in FIG. However, other embodiments do not provide both of these techniques in the same UI.

  FIG. 43 shows this ROI identification operation as five operation stages of the UI 4200. The first step 4205 in FIG. 43 is the same as the first step 4205 in FIG. Specifically, in this first step 4205, the UI 4200 shows a PIP display 4265 having a larger background main display area 4230 and a smaller foreground insertion display area 4235 at the lower right corner of the PIP display 4265.

  In a second step 4310, an operation of identifying a region of interest is started. In this example, by selecting a first position for a period of time to define a region of interest in the video presented in the background display area 4230 (eg, the human face 4270 displayed in the background display 4230 on the screen of the device) Movement is initiated by pressing the surrounding position with the finger 4350 for a period of time). In a third step 4315, the UI 4200 indicates that the first position 4370 has been selected as a point 4355 next to the selected first position on the background display area 4230.

  The fourth step 4320 shows the UI 4200 after the user has selected the second position 4375 for defining the region of interest. In this example, as indicated by arrow 4360, the user drags finger 4350 across the device screen from the first position after point 4355 is displayed, and displays the displayed hat in background display area 4230. This second position 4375 is selected by stopping at a position between the trees displayed as. As shown in the fourth stage, this drag causes the UI 4200 to draw a square border 4365 for the region of interest area having a first location 4370 and a second location 4375 at its opposing vertices.

  The fifth step 4325 shows the UI 4200 after the region of interest has been identified. In this example, the user completes identification of the region of interest by stopping dragging finger 4350 and releasing finger 4350 from the display screen of the device once the desired region of interest is identified. A fifth step 4325 indicates that the quality of the video in the region of interest has substantially improved from the quality in the first step 4205 as a result of the drawing process. In some embodiments, the drawing process also causes the same changes made to the local device 4225 to be displayed on the remote device. In this example, for example, the image quality in the region of interest of the same video displayed on the remote device is substantially improved.

  The above description of FIGS. 42 and 43 shows different ways of identifying regions of interest in a video to enhance the quality of the identified regions. In some embodiments, by enhancing the quality of the identified region of interest, the encoding operation of the dual camera portable device, such as allocating more bits to the identified region when encoding the video. Changes occur.

  According to some embodiments, a user may identify a region of interest in a video to make various changes to a mobile device or its camera. For example, FIG. 44 shows an example of identifying a region of interest in a video to extend or shrink the region of interest on a display. In this approach, the user selects an area on the display as the center of the region of interest, and then identifies the region of interest in the video by expanding or contracting the region of interest area.

  In FIG. 44, UI 4400 of mobile device 4425 presents PIP display 4265 during a video conference with a remote user of another mobile device. The PIP display 4265 of FIG. 44 is substantially similar to the PIP display 4265 of FIG. 42, except that the foreground insert display 4235 of FIG. 44 is at the lower left corner of the PIP display 4265.

  FIG. 44 shows the ROI selection operation as four operation stages of the UI 4400. As shown in the first step 4405, the background display 4430 presents a video with a man to the left of the display 4430 and a tree 4440 to the right. Further, the tree 4440 is relatively small and occupies only the right side of the background display area 4430. In this example, the user of the mobile device 4425 wants to identify the area where the tree 4440 is displayed in the display 4430 as a region of interest.

  In a second step 4410, an operation of identifying a region of interest is started. In this example, by selecting an area 4440 in the video presented on the background display 4430 that the user wishes to identify as a region of interest (e.g., creating a background display area 4430 where trees 4440 are displayed by two lines) The movement is started (by pressing with fingers 4445 and 4446). At this stage 4410, the user can expand the region of interest area 4440 by dragging the fingers 4445 and 4446 further away from each other to obtain a larger portion of the background display area 4430. By dragging his or her fingers 4445 and 4446 closer together, the user can also shrink the region of interest 4440 and obtain a smaller portion of the background display area 4430.

  The third stage 4415 moves its fingers 4445 and 4446 away from each other, as indicated by arrow 4450 (ie, moves finger 4445 toward the upper left corner of background display area 4430 and moves finger 4446 Moving toward the lower right corner of display 4430) shows UI 4400 after the user has expanded region of interest 4440 and has begun to occupy a larger portion of background display area 4430. In some embodiments, movement of the finger also causes the same change in the display of the remote device as it did for the local device. In this example, for example, the region of interest of the same video expands and occupies a larger portion of the background display area 4430 of the remote device. In some embodiments, as described further below, expansion of the region of interest in the local and / or remote displays may cause one or both of the mobile device or its camera to perform one or more of the other operations. change.

  The fourth step 4420 displays the UI 4400 after the region of interest has been identified. In this example, the user identifies the region of interest by stopping dragging his finger 4445 and 4446 and releasing the finger 4445 and 4446 from the device display screen when the region of interest reaches the desired ratio in the background display area 4430. Complete. As a result of this process, the region of interest occupies most of the background display 4430. Identification of the region of interest motion has been completed at this point.

  Some of the above examples are provided by the user to improve image quality in the selected region of interest of the video (e.g., by increasing the bit rate to encode the region of interest of the video). Show how you can identify regions of interest in a video. In some embodiments, identifying regions of interest in the video causes changes to the image processing operations of the portable device, such as exposure, scaling, focus, and the like. For example, identifying a region of interest in a video (eg, by identifying a region of interest to zoom) may cause videoconferencing manager 1604 to scale and composite images of the video differently.

  In other embodiments, identifying a region of interest in the video causes a change to the operation of the camera (s) of the mobile device (eg, frame rate, zoom, exposure, scaling, focus, etc.). In still other embodiments, identifying a region of interest in the video causes a change to the encoding behavior of the portable device, such as allocating more bits to the identified region, scaling, and the like. Further, while the exemplary ROI identification operation described above may cause only one of the above-described changes to the mobile device or its camera, in some other embodiments, the ROI identification operation is Multiple of the changes to the operation of the device or its camera may be triggered. Further, in some embodiments, the layout of the display area 1155 of FIGS. 42 to 44 is the same as the layout of the display area 1155 of FIG. 12 described above.

B. Switching Cameras Some embodiments provide a procedure for switching cameras during a video conference (ie, changing the camera from which images are captured). Different embodiments provide different procedures for performing the camera switching operation. Some embodiments provide a procedure performed by a dual camera portable device to switch the camera of the device (ie, local switch), while other embodiments provide that the dual camera portable device For instructing a dual camera portable device to switch cameras of the other device (ie, remote switching). Still other embodiments provide procedures for both. Section IV. B. 1 describes a process for performing a local camera switching operation on a dual camera portable device. Section IV. B. Section 2 describes a process for performing a remote camera switching operation on a dual camera portable device.

1. Local Camera Switching FIG. 45 shows a process 4500 that some embodiments perform on a local dual camera mobile device to switch between two cameras of the device during a video conference with a remote mobile device that includes at least one camera. . In some embodiments, process 4500 is performed by video conference manager 1604 shown in FIG. For purposes of explanation, in this description one camera of the local dual camera portable device will be referred to as camera 1 and the other camera of the local dual camera portable device will be referred to as camera 2.

  Process 4500 begins by initiating (at 4505) a video conference between a local dual camera mobile device and a remote mobile device. Next, the process 4500 transmits (at 4510) a video image from the currently selected camera (eg, camera 1) of the local dual camera mobile device to the remote mobile device for display on the remote mobile device. At 4510, the process also generates and displays a composite display based on the video image and the video image received from the remote portable device.

  Process 4500 then determines (at 4515) whether a video conference termination request has been received. As discussed above, in some embodiments, the video conference may be at the request of a user of the local dual camera mobile device (eg, through a local dual camera mobile device user interface) or (eg, through a remote mobile device user interface). Also, it can be terminated at the request of the user of the remote portable device. If the process 4500 receives a video conference end request, the process 4500 ends.

  If process 4500 does not receive an end video conference request, process 4500 then determines (at 4520) whether the user of the local dual camera portable device has instructed the device to switch cameras for video conference. If process 4500 determines that the device has not been instructed to switch cameras (at 4520), process 4500 returns to operation 4510. However, if process 4500 determines (at 4520) that the device has been instructed to switch cameras, process 4500 proceeds to 4525.

  The process 4500 sends, at 4525, a notification to the remote portable device indicating that the local dual camera portable device is switching cameras. In some embodiments, process 4500 sends a notification over a video conferencing control channel multiplexed with audio and video channels by VTP manager 3125 as described above.

  After sending the notification, process 4500 performs (at 4530) a camera switch operation. In some embodiments, performing the camera switching operation (at 4530) includes instructing the CIPU to stop capturing video images by Camera 1 and start capturing video images by Camera 2. These instructions can simply instruct the CIPU to switch to capturing images from the pixel array associated with camera 2 and begin processing these images. Alternatively, in some embodiments, the instructions to the CIPU capture (1) operate the camera 2 based on a specific set of settings, and (2) capture the video generated by the camera 2 at a specific frame rate. And / or (3) with a set of initialization parameters instructing the CIPU to process video images from camera 2 based on a particular set of settings (eg, resolution, etc.).

  In some embodiments, the camera switch command (at 4530) also includes a command to switch an unused camera to the fourth operating power mode described above. In this example, the camera switching command includes a command for camera 2 to switch to its fourth operating power mode. In addition, the camera switch command may be used to quickly switch to the fourth operating power mode when requested to capture an image and to allow the camera 1 to save power from its fourth operating power mode so that capture can begin. It also includes an instruction to switch to another operating power mode, such as an operating power mode, or a third operating power mode. A camera switch operation 4530 combines the image captured by camera 2 of the local dual camera mobile device (rather than the image captured by camera 1) with the image received from the remote mobile device for display on the local dual camera mobile device. Including doing.

  After instructing a camera switch at 4530, the process 4500 includes a camera switch animation on the local dual camera handheld device to display the transition between displaying images from camera 1 and camera 2 ((4). (At 4535). Following the camera switch animation on the local dual camera mobile device, process 4500 loops back through operations 4510-4520 until a video conference end request or a new camera switch request is received.

  FIG. 46 illustrates an example of how a camera switch operation can be requested via the dual camera device UI 1105 according to some embodiments, and how these embodiments animate the camera switch operation. This figure shows the camera switching operation as eight different operating phases 4610, 4615, 4620, 4625, 4630, 4635, 4640 and 4645 of the UI 1105 of the device. The first four stages 4610, 4615, 4620 and 4625 of the UI 1105 illustrate an example of receiving a user request to switch cameras. In some embodiments of the invention, the user of the device has another mechanism for making such a request.

  The first step 4610 is the same as the fifth step 1130 of the UI 1105 in FIG. 11, and shows the UI 1105 after the video conference has been set up. At this stage, the UI 1105 displays a PIP display that includes two video displays, a larger background display from the remote camera and a smaller foreground insertion display from the local camera. In this example, the background main display area 1170 presents a video of a woman, which in this example is assumed to be a woman whose video is being captured by the remote device, and the foreground insert display area 1160 is the front camera of the local device in this example. Presents a video of a man assumed to be a man whose video has been captured.

  Thereafter, a second step 4615 indicates the beginning of the camera switching operation through selection of the PIP display area 1180 of the UI 1105. As shown, the selection is made by placing the user's finger 4670 on the PIP display 1180. The third step 4620 shows the UI 1105 including a selectable UI item 4675 (eg, a camera switch button 4675) to request switching between cameras of the local device 4600 during a video conference. A fourth step 4625 shows the UI 1105 after the user of the local device 4600 has selected the selectable UI item 4675 (eg, by a single finger tap) and this selection is indicated through highlighting of the selectable UI item 4675. By selecting this selectable UI item 4675, the user has instructed device 4600 to switch from the front camera of device 4600 to the rear camera of device 4600 during a video conference. In another example, where the rear camera of device 4600 is capturing video, the user's selection of selectable UI item 4675 instructs device 4600 to switch from the rear camera of device 4600 to the front camera of device 4600. After the fourth phase, the video conference manager sends a command to the CIPU and the remote device to initiate a camera switch operation.

  The last four stages 4630, 4635, 4640 and 4645 of the UI 1105 show an example of a camera switching animation on the local device. In this animation, the video captured from the front and rear cameras of the local device is currently displayed on two opposite sides of the viewing pane, only one of which is arbitrary. The aim is to provide an impression that is visible at the time by the user. If a camera switch is requested in the middle of a video conference, this view pane will be made to appear to have rotated around the vertical axis, so that the view pane previously showed the video of one camera to the user. The presentation of one camera's video on one side of the camera rotates away from the user until it is replaced by the other side of the view pane, with the other side of the view pane showing the video of the other camera. This animation and appearance of the perceived view pane rotation is such that (1) the video image from that camera in one camera viewport is progressively smaller and applies a perspective correction operation to it, followed by ( 2) Achieved by progressively expanding the video image from the other camera in the viewing area and reducing the perspective correction operation on it.

  Thus, the fifth step 4630 indicates the start of “rotation of the view pane” about the vertical axis 4682. To give the appearance of a rotation of the view pane, the UI 1105 reduces the size of the video image of the front camera in the video display area 1160, and applies a perspective operation so that the right side of the video image is compared to the left side of the video image. So that it can be seen from far away.

  The sixth step 4635 indicates that the view pane has been rotated 90 degrees so that the user can see only the edges of this pane, as indicated by the thin line 4686 displayed in the center of the display area 1160. A seventh step 4640 indicates that the view pane is continuing to rotate so that the back side of the view pane 4688 is now visible to the user in steps to show the video captured from the user's rear camera. Again, in some embodiments, this display of the rotating animation reduces the size of the rear camera video image in the video display area 4688, and applies a perspective operation to the video image relative to the right side of the video image. This is achieved by making the left side of the image appear far from the user.

  The eighth step 4645 indicates the completion of the animation indicating the camera switching operation. Specifically, this stage displays a video image of the car captured by the rear camera of device 4600 in display area 1160.

  The example described above with reference to FIG. 46 activates the camera switching operation through the camera switching user interface. Other embodiments invoke the camera switching operation differently. For example, some embodiments initiate a camera switch operation by having a camera switch selectable UI item that is permanently displayed on the UI during a video conference, such as UI 1105 in FIG. In FIG. 47, the camera switching button 1289 is shown in the display area 1155 together with the mute button 1285 and the conference end button 1287. In some embodiments, the layout of display area 1155 is the same as the layout of display area 1155 described above with respect to FIG.

  FIG. 47 shows the camera switching operation of the UI 1105 as six stages: 4610, 4790, 4630, 4635, 4640 and 4645. 47, except that the layout of the display area 1155 shows a mute button 1285, a conference end button 1287, and a camera switch button 1289 instead of a single conference end button. Similar to step 4610. The second step 4790 shows the UI 1105 after the user of the local device 4600 has selected the camera switchable selectable UI item 1289 (eg, through a single finger tap using the finger 4670). In this example, by selecting this selectable UI item 1289, the user instructs device 4600 to switch from the front camera of device 4600 to the rear camera of device 4600 during a video conference. 47 are similar to the latter four stages of FIG. 46, except that the layout of the display area 1155 is the same as the above-described layout in the first stage 4610, and thus is unnecessary. In order not to obscure the description of the present invention with great detail, no further description is given.

  In some embodiments, if the remote mobile device receives an image from a different camera of the local dual camera mobile device (i.e., the local dual camera mobile device has switched cameras), the remote mobile device will have a local dual camera mobile device. A camera switching animation is also performed to indicate the transition between displaying images from one camera of the device and the other camera of the local dual camera portable device. FIG. 48 illustrates one example of such a camera switching animation as five operating phases 4810, 4815, 4820, 4825, and 4830 of the UI 4805. This figure shows an exemplary camera switching animation on a remote portable device 4800. The operating steps are the same as the exemplary animation of FIG. 46, except that the animation is performed on the image displayed in display area 4835, where display area 4835 is where the image from the local dual camera portable device is displayed. The position displayed on the remote portable device 4800. Accordingly, the male image displayed in display area 4835 is positioned on a vertical axis 4855 at the center of display area 4850 to indicate the transition between displaying the male image in display area 4835 and displaying the image of car 4870. It is animated so that it appears to rotate 180 degrees. The implementation of the camera switching animation in some embodiments is the same as the implementation of the animation described above.

  The above example shows a camera switching animation on a remote device with a particular user interface layout. Other embodiments could perform this camera switching animation on remote devices with different user interface layouts. For example, FIG. 49 shows one such example of a remote device 4800 having a different user interface layout 4805. In particular, the UI 4805 in FIG. 49 includes a mute button 1285, a conference end button 1287, and a camera switching button 1289 included in the display area 1155. Will be displayed permanently. The layout of the three buttons was described above with respect to FIG. Other than the different user interface layouts, the five steps 4810, 4815, 4820, 4825, and 4830 of FIG. 49 are identical to the five steps 4810, 4815, 4820, 4825, and 4830 of FIG.

2. Remote Camera Switching FIG. 50 shows a process 5000 for switching between two cameras of a remote dual camera device during a video conference. This process 5000 is performed by a video conference manager of a device that includes at least one camera. In the following description, a device for which the user instructs to switch a remote camera is called a local device, and a device for switching between two cameras is called a remote device. Similarly, in the following description, the remote device is said to switch between its front camera (ie, camera 1) and its rear camera (ie, camera 2).

  The process 5000 of FIG. 50 is described with respect to FIGS. 51, 52, 53, and 54. FIG. 51 shows a UI 5105 of a local device 5100 used by a user to request that a remote device switch between its two cameras during a video conference. The figure shows eight different stages of operation 5110, 5115, 5120, 5125, 5130, 5135, 5140 and 5145 of this UI 5105. FIG. 54 shows a UI 5405 of the remote device 5400 that receives a camera switching request from the local device 5100. FIG. 54 illustrates six different operating stages 5410, 5415, 5420, 5425, 5430, and 5435 of the UI 5405.

  As shown in FIG. 50, process 5000 begins by initiating (at 5005) a video conference between a local device and a remote device. Thereafter, process 5000 receives (at 5010) images from one camera of each device (eg, from the front camera of each device) and generates a composite view for the video conference based on these images. At 5010, process 5000 also sends a video image from the local device to the remote device.

  Thereafter, process 5000 determines (at 5015) whether a video conference termination request has been received. As mentioned above, the video conference may be terminated in some embodiments at the request of the user of the local or remote device. If the process 5000 receives the videoconference end request, the process 5000 ends.

  If process 5000 does not receive a video conference termination request, then the user of the device performing process 5000 (ie, the user of the local device) requests that the remote device switch its camera for the video conference. Process 5000 determines (at 5020) whether the device has been instructed to do so. If the process 5000 determines that it has not been instructed to initiate a remote camera switch (at 5020), the process 5000 returns to operation 5010. If the process 5000 determines (at 5020) that the device has been instructed to initiate a remote camera switch, the process 5000 proceeds to 5025, which is described further below.

  The first four stages 5110, 5115, 5120 and 5125 of the UI 5105 of FIG. 51 show an example of receiving a user request to switch the camera of a remote device. The first step 5110 and the second step 5115 are the same as the first step 4610 and the second step 4615 of FIG. The third step 5120 is selectable for a request to the remote device 5100 to switch cameras, in addition to a selectable UI item 5175 for the third step 5120 to request the local device 5100 to switch cameras. Same as the third step 4620 except that it includes a UI item 5180. A fourth step 5125 indicates that the user of the local device 5100 selects a UI item 5180 (eg, via a single finger tap 5170 to the selectable UI item 5180) to request the remote device to switch cameras. The selection is indicated by highlighting the selectable UI item 5180. While FIG. 51 illustrates one example of performing this operation, other embodiments may perform the operation to request the remote device to switch cameras differently.

  The example described above with reference to FIG. 51 activates a remote camera switching operation through a remote camera switching user interface. Other embodiments invoke the remote camera switching operation differently. For example, some embodiments activate the camera switching operation by causing the UI to permanently display a camera switching selectable UI item during a video conference, such as UI 5105 in FIG. In FIG. 52, the remote camera switching button 5288 is shown in the display area 1155 together with the mute button 5282, the conference end button 5284, and the local camera switching button 5286.

  FIG. 52 illustrates the remote camera switching operation of the UI 5105 of the device 5100 as six different stages 5110, 5290, 5130, 5135, 5140 and 5145. The first step 5110 in FIG. 52 is similar to the first step 5110 in FIG. 51 except that the layout of the display area 1155 shows a mute button 5282, a local camera switch button 5286, a remote camera switch button 5288, and a conference end button 5284. The same is true. The second step 5290 shows the UI 1105 after the user of the local device 5100 has selected the selectable remote camera switching UI item 5288 (eg, through a single finger tap 5170). 52 are similar to the latter four stages in FIG. 51, except that the layout of the display area 1155 is the same as the above-described layout in the first stage 5110, and thus is unnecessary. In order not to obscure the description of the present invention with great detail, no further description is given.

  Some embodiments provide a layout similar to the layout shown in FIG. 52, except that the selectable remote camera switch UI item is displayed in the PIP display 5165 instead of in the display area 1155. FIG. 53 shows such a layout 5105. In particular, this figure shows a PIP display having a selectable remote camera switch UI item 5180 and a display area 1155 having a mute button 5282, a local camera switch button 5286, and a conference end button 5284.

  As described above, if the user requests a remote camera switch, the process 5000 moves to 5025. At 5025, process 5000 sends a camera switch request to the remote device. In some embodiments, the request is sent over a video conferencing control channel multiplexed with audio and video channels by VTP manager 3125 as described above.

  After the camera switch request is received, the process 5000 determines (at 5030) whether the remote device has responded to the camera switch request. In some embodiments, the remote device automatically sends an acknowledgment (ie, sends an acknowledgment) to the local device over the video conference control channel. However, in other embodiments, the user of the remote device must accept this request through the user interface of the remote device.

  The first two stages 5410 and 5415 of the UI 5405 in FIG. 54 show an example in which the remote user accepts the camera switching request of the remote device 5400. The first step 5410 includes (1) a display area 5440 for displaying text notifying the request to the remote user, and (2) a selectable UI item 5465 for accepting the camera switching request of the remote device (eg, an enable button). 5465), and (3) a selectable UI item 5470 (for example, a reject button 5470) for rejecting the camera switching request of the remote device. Thereafter, a second step 5415 includes selecting a UI item 5465 by the remote device user (eg, via a single finger tap 5480) to accept the camera switch request, as indicated by highlighting of the selectable UI item 5465. A later UI 5405 is shown.

  If the process 5000 determines that a response has not yet been received from the remote device (at 5030), the process 5000 determines (at 5035) whether a video conference termination request has been received. If a video conference termination request has been received, process 5000 ends. Otherwise, the process receives (at 5040) images from the currently used cameras of the remote device and the local device, generates a composite view for the video conference based on these images, and The video image is transmitted to the remote device, and then the process goes to 5030.

  If the process 5000 determines that a response is still received from the remote device (at 5030), the process 5000 determines (at 5045) whether the remote device has accepted the camera switch request. If not, the process 5000 returns to operation 5010 and continues to receive images from the other device's camera. If not, the process receives an image (at 5050) from another camera on the remote device, then performs a camera switch animation on the local device (at 5055), and displays the previously used remote camera video and current Indicate the transition between the video of the remote camera being used (ie, the image received in operation 5050). After 5055, the process moves to 5010 described above.

  The latter four stages of operation 5130, 5135, 5140 and 5145 shown in the UI 5105 of FIG. 51 illustrate an example of such a remote camera switching animation on the local device 5100. The example animation is shown in FIG. 51 except that FIG. Similar to the exemplary animation shown at 48 steps 4815, 4820, 4825 and 4830. 52 and 53 are the same as the animation of FIG. 51 except that the display area 1155 of FIGS. 52 and 53 includes a different selectable UI item than the display area 1155 of FIG. Show.

  In some embodiments, when the remote device switches cameras, the UI of the remote device also performs a camera switch animation to indicate the transition between the two cameras. The latter four operation stages 5420, 5425, 5430, and 5435 shown in the UI 5405 of FIG. 54 show an example of a camera switching animation displayed on the remote device 5400 when the remote device 5400 switches the camera. This animation is similar to step 4630 of FIG. 46, except that the animation of display area 5445 replaces the video of the woman captured by the front camera of remote device 5400 with the video of the tree captured by the rear camera of remote device 5400. , 4635, 4640 and 4645.

  As described above, FIGS. 46, 47, 48, 49, 51, 52, 53 and 54 show various examples of camera switching animations performed on the user interface. In some embodiments, the camera switching animation causes a change to the image processing behavior of the respective dual camera portable device, such as scaling, compositing, and perspective distortion, which can be performed by, for example, the video conferencing manager 1604 and the image processing manager 1608. .

C. Exposure Adjustment During a video conference between a dual camera mobile device and another mobile device, different embodiments provide different techniques for adjusting the exposure of images captured by the camera of either mobile device. While some embodiments provide techniques for a user of a dual camera portable device to adjust the exposure of images captured by the camera of another device, other embodiments provide the user with a dual camera portable device. Techniques for adjusting the exposure of images captured by a camera are provided. Some exemplary techniques are described in detail below.

  FIG. 55 shows a process 5500 for performing a remote exposure adjustment operation on a dual camera portable device of some embodiments during a video conference. In the following description, a device for which a user instructs a remote device to adjust its exposure level is called a local device. In some embodiments, process 5500 is performed by a videoconferencing manager on the local device. Further, the process 5500 will be described with reference to FIGS. 56, 57 and 58. These figures illustrate various ways for a user of a local device to request a remote device to perform an exposure adjustment operation.

  As shown in FIG. 55, process 5500 begins by initiating (at 5505) a video conference between a local device and a remote device. Thereafter, process 5500 receives (at 5510) video from the remote device for display on the display screen of the local device. Next, the process 5500 determines (at 5515) whether a video conference termination request has been received. As described above, some embodiments can receive a video conference termination request from a user of a local or remote device. If the process 5500 receives the video conference end request, the process 5500 ends.

  However, if process 5500 does not receive a video conference termination request, then process 5500 determines (at 5520) whether a request to adjust the exposure of the remote device's camera has been received. If the process 5500 determines that a request to adjust the camera exposure of the remote device has been received, the process 5500 returns to operation 5510 to receive additional video captured from the remote device. FIGS. 56, 57 and 58 show three different examples that provide a way for a user to make such a request. In FIGS. 56, 57 and 58, the first stages 5610, 5710 and 5810 are all local video displaying two videos, one captured by the camera of the local device and the video captured by the camera of the remote device. Shown are PIP displays 5625, 5750 and 5835 for devices 5600, 5700 and 5800. In the first stage 5610, 5710 and 5810, the men in the background displays 5635, 5760 and 5845 are dark, indicating that the men are not properly exposed.

  The second step 5615 in FIG. 56 is for the user of the local device 5600 to request the remote device to perform an exposure adjustment by selecting the video of the remote device (eg, through a single tap on the background display 5635). Show one way. In this manner, the UI 5605 instructs the remote device to perform an exposure adjustment on the user's region of interest, as defined by box 5645, and thus performs the exposure adjustment operation on the remote device. Automatically associated with the user's desire to instruct the videoconferencing manager of the local device to contact. The defined region of interest is used by the remote device to calculate the exposure adjustment.

  Like the second step 5615 of FIG. 56, the second step 5715 of FIG. 57 is similar to the second step 5715 of FIG. 3 shows the local user's selection of the video. A fourth step 5725 illustrates that the user of the local device selects the selectable UI item 5770 to instruct the remote device to perform the above-described exposure adjustment operation.

  The second step 5815 of FIG. 58 is similar to the second step 5715 of FIG. 57, except that the user's selection of the video on the remote device does not instruct the UI to display a single selectable UI item. As shown in the third step 5820, the user's selection instructs the UI 5805 to display a menu of selectable UI items 5855, 5860, 5865 and 5870. The selectable UI items include an auto focus item 5855, an automatic exposure item 5860, a camera switching item 5865, and a cancel item 5870. In some embodiments, camera switch selectable UI item 5865 is used to request a local camera switch operation, while in other embodiments, camera switch selectable UI item 5865 is used for remote camera switch operation. Used to request. The fourth step 5825 illustrates that the user selects the automatic exposure item 5860 to instruct the remote device to perform the above-described exposure adjustment operation.

  If the process 5500 determines (at 5520) that the local user has instructed the local device to request an exposure adjustment operation, the process 5500 may be captured by a camera that is currently capturing video and transmitting to the local device. A command is sent (at 5525) to the remote device over the video conferencing control channel to adjust the exposure of the video. After operation 5525, process 5500 moves to operation 5510, described above.

  In some embodiments, the user of the remote device is required to provide approval before the remote device performs an exposure adjustment operation, while in other embodiments, the remote device receives a request from the local device Then, the exposure adjustment operation is automatically performed. Further, in some embodiments, some of the video conferencing functions are performed by video conferencing manager 1604. In some of these embodiments, video conferencing manager 1604 performs an exposure adjustment operation by instructing CIPU 1650 to adjust the exposure settings of the sensors of the remote device camera being used.

  The last step 5620, 5730 and 5830 of FIGS. 56, 57 and 58 shows the video of the remote device brighter, indicating that the male is properly exposed. Although FIGS. 56, 57 and 58 provide examples of receiving an exposure adjustment request to correct the exposure of a remote device, some embodiments require that the local device adjust the exposure of the camera of the local device. Is provided for the user of the local device to request. Such a request may be made in a manner similar to that shown in FIGS. 56, 57 and 58 for requesting the remote device to adjust its camera exposure.

  FIGS. 56-58 described above illustrate some user interfaces for performing exposure adjustment operations. In some embodiments, the exposure adjustment operation can cause a change to the image processing operation of the dual camera portable device, such as invoking the exposure adjustment process 5900 described in further detail below. The exposure adjustment operation can also cause a change in the operation of the camera of a dual camera portable device that is capturing video, for example, a change in the exposure level setting of the camera.

1. Exposure Adjustment Method FIG. 59 conceptually illustrates an exposure adjustment process 5900 performed by the image processing manager of some embodiments, such as the process shown in FIG. In some embodiments, process 5900 is part of the exposure adjustment operation described above with respect to FIGS. 55, 56, 57, and 58. In some of such embodiments, the image processing manager 1608 performs the process 5900 and adjusts the camera's exposure settings by sending instructions to the video conferencing manager 1604, where , Command the CIPU 1650 to adjust the camera sensor 405a or 405b.

  In some embodiments, process 5900 is performed by image processing layer 930 shown in FIG. 9, while in other embodiments, process 5900 is performed by statistics engine 465 shown in FIG. While some embodiments perform process 5900 on images captured by the camera of the device (local or remote) during a video conference, other embodiments perform process 2100 (eg, , Perform process 5900 as part of operation 2110). Some embodiments perform an exposure adjustment operation to expose images captured by the camera of the dual camera portable device that are neither too bright nor too dark. In other words, process 5900 is performed to capture an image in a manner that maximizes the amount of detail as much as possible.

  Process 5900 begins by receiving (at 5905) an image captured by a camera of a dual camera portable device. In some embodiments, if the received image is the first image captured by the device's camera in a video conference, process 5900 is not performed on the first image (ie, the first image is less than the first image). Before, there is no image to determine the exposure value). Thereafter, process 5900 reads (at 5910) the pixel value of the defined region of the received image. Different embodiments define the regions differently. Some of such embodiments define regions of different shapes, such as squares, rectangles, triangles, circles, etc., while others of such embodiments include centers, upper centers, lower centers, etc. Regions are defined at different positions in the image.

  Next, the process 5900 calculates (at 5915) the average of the pixel values in the defined region of the image. Process 5900 determines (at 5920) whether the calculated average of the pixel values is equal to a particular prescribed value. Different embodiments define different specific values. For example, some embodiments define a particular value as the median pixel value of the dynamic range of the image. In some embodiments, a range of values is defined rather than a single value. In such an embodiment, process 5900 determines (at 5920) whether the calculated average of the pixel values is within a specified value range.

  If the calculated average of the pixel values is not equal to the specified value, the process 5900 adjusts (at 5925) the exposure value based on the calculated average. If the calculated average of the pixel values is equal to the specified value, the process 5900 ends. In some embodiments, the exposure value represents an amount of time that the camera sensor is exposed to light. In some embodiments, the adjusted exposure value is used to expose the next image to be captured by the camera that captured the received image. After the exposure values have been adjusted based on the calculated average, process 5900 ends.

  In some embodiments, process 5900 is performed iteratively until the calculated average of the pixel values is equal to (or within a specified range of values) a specified value. Some embodiments always perform process 5900 during a video conference, while other embodiments perform process 5900 during a video conference at regular intervals (eg, 5 seconds, 10 seconds, 30 seconds, etc.). I do. Moreover, during a video conference, the process 5900 of some embodiments dynamically redefines certain pixel values before performing the process 5900.

  FIG. 60 conceptually shows an example of the exposure adjustment operation of some embodiments. Examples 6000, 6010 and 6015 each show an image 6020 captured by the camera of a dual camera portable device on the left. Specifically, image 6020 shows a darker person in front of the sun. A dark person indicates that the exposure level of the image is not high enough to expose the person's face or body. The right side of each example 6000, 6010 and 6015 shows images 6025, 6030 and 6035 captured after image 6020, respectively. In some embodiments, the image 6020 and the image on the right are images of a video captured by a camera of a dual camera portable device. In other embodiments, the image 6020 and the right image are still images captured by the cameras of the dual camera portable device at different times.

  The first example 6000 shows an operation without exposure adjustment. Thus, image 6025 looks the same as image 6020. Because no exposure adjustment was performed, the person in image 6025 remains dark, like the person in image 6020.

  In the second example 6010, an exposure adjustment operation is performed on the image 6020. In some embodiments, an exposure adjustment operation is performed by the process 5900 using the defined area 6040. Based on the exposure adjustment operation, the exposure level of the camera is adjusted, and the camera captures an image 6030 using the adjusted exposure level. As shown in FIG. 60, the person in image 6030 is not as dark as image 6025. However, the person's face and body in image 6030 are still not clear.

  The third example 6015 shows an exposure adjustment operation performed on the image 6020. As with the second example 6010, the exposure adjustment operation of the example 6015 of some embodiments is performed by the process 5900 using the defined area 6045. Based on the exposure adjustment operation, the exposure level of the camera is adjusted, and the camera captures an image 6035 using the adjusted exposure level. As can be seen in FIG. 60, the person in image 6035 is completely exposed because the person's face and body can be recognized.

  In some embodiments, the selection of the predefined area may be made by a user of the dual camera portable device. The device itself can also automatically adjust its defined region for exposure adjustment operations through the exposure adjustment feedback loop described above at CIPU 400. The statistics engine 465 of FIG. 4 determines whether the exposure level is appropriate for the captured image and adjusts the data to adjust the camera sensor accordingly (eg, through a direct connection to the sensor module 415). May be collected.

D. Focus Adjustment FIG. 61 shows a process 6100 for adjusting the focus of a dual camera portable device during a video conference. In the following description, a device that a user instructs a remote device to adjust the focus of its camera is called a local device. In some embodiments, process 6100 of FIG. 61 is performed by videoconferencing manager 1604 of the local device. This process is described below with reference to FIGS. 62 and 63. These figures provide two exemplary ways for a user of a local device to request that a focusing operation be performed by a remote device.

  As shown in FIG. 61, process 6100 begins by initiating (at 6105) a video conference between a local device and a remote device. Thereafter, process 6100 receives (at 6110) video from the remote device for display on the display screen of the local device. Next, at 6115, the process 6100 determines whether a video conference termination request has been received. As mentioned above, the video conference may be terminated in some embodiments at the request of the user of the local or remote device. If the process 6100 receives a video conference end request, the process 6100 ends.

  Otherwise, the process determines (at 6120) whether a request to adjust the focus of the remote device's remote camera has been received. If the process 6100 determines that it has not received a request to adjust the focus of the remote camera of the remote device, the process 6100 returns to operation 6110 to receive additional video from the remote device. FIGS. 62, 63 and 64 show three different ways that different embodiments provide a user to make such a request. In FIGS. 62, 63 and 64, the first stages 6210, 6310 and 6472 are all local devices 6200, 6300 that display two videos: one captured by the local device and one captured by the remote device. 6471 shows PIP displays 6225, 6335 and 6482. Display areas 1155 and 1155 in FIGS. 62 and 63 indicate a conference end button. However, in FIG. 64, the layout of the display area 1155 is the same as the layout of the display area 1155 of FIG. 12 described above. Further, the camera switch button 6488 shown in the display area 1155 may be selected to initiate a local camera switch operation in some embodiments, or a remote camera switch operation in other embodiments. As shown in the first stage 6210, 6310 and 6472, the video of the remote device displayed in the background displays 6235, 6345 and 6480 is blurry.

  The second stage 6215 of FIG. 62 illustrates one approach for the user of the local device to request focus adjustment by simply selecting the video of the remote device (eg, through a single tap 6240 on the video of the remote device). In this approach, UI 6205 instructs the remote device to perform an operation (such as focus) on the region of interest of the user, as defined by box 6245, and thus (such as a focus adjustment operation). ) Automatically associate with the user's desire to instruct the videoconferencing manager 1604 of the local device 6200 to contact the remote device to perform a coordination operation. The defined region of interest is used by the remote device to calculate the focus adjustment.

  The second step 6315 of FIG. 63 also illustrates the selection of the local user's remote video (eg, through a tap on the user's remote device video). However, unlike the example shown in FIG. 62, this selection in FIG. 63 is based on the menu of selectable UI items 6355, 6360, 6365 and 6370 (which can be implemented as selectable buttons), as shown in the third step 6320. Is displayed to the UI 6305. These selectable UI items include an autofocus item 6360, an automatic exposure item 6365, a camera switching item 6370, and a cancel item 6355. In some embodiments, the camera switch selectable UI item 6370 is used to request a local camera switch operation, while in other embodiments, the camera switch selectable UI item 6370 includes a remote camera switch Used to request action. Thereafter, a fourth step 6325 indicates that the local user selects the autofocus item 6360.

  The second stage 6474 of FIG. 64 again illustrates the selection of the local user's remote video (eg, via a user's remote device video tap). However, unlike the example shown in FIG. 63, this selection in FIG. 64 instructs UI 6478 to request a focus adjustment operation (ie, in a second step 6474). After the focus adjustment operation is completed, UI 6478 displays a menu of selectable UI items 6484 and 6486 that can be implemented as selectable buttons (ie, in a third step 6476). These selectable UI items include an automatic exposure item 6486 and a cancel item 6484.

  If the process determines (at 6120) that the local user has instructed the local device to request a focus adjustment operation, the process 6100 adjusts the focus of the camera at which the remote device is currently capturing and transmitting video. To do so, it sends a command (at 6140) to the remote device over the video conference control channel. After 6140, the process moves to 6110, described above.

  In some embodiments, the user of the remote device must provide authorization before the remote device performs this operation, while in other embodiments, the remote device receives a request for the local device. Perform this operation automatically. Also, in some embodiments, the focus adjustment operation adjusts a focus setting of a camera of the remote device being used during the video conference. In some of such embodiments, some of the video conferencing functions are performed by video conferencing module 1602 described above. In these embodiments, video conference manager 1604 instructs CIPU 1650 to adjust the sensors of the remote device camera being used.

  The last steps 6220, 6330 and 6476 of FIGS. 62, 63 and 64 show the video of the properly focused remote device. 62, 63 and 64 provide an example of receiving a focus adjustment request to correct the focus of a remote device, but according to some embodiments, the user of the local device may have a local device You can request to adjust the focus of the camera. Such a request can be made in a manner similar to that shown in FIGS. 62, 63 and 64 for requesting the remote device to adjust the focus of the camera.

  FIGS. 62, 63 and 64 show three exemplary user interfaces that allow a user to perform a focus adjustment operation. In some embodiments, the focus adjustment operation causes a change to the operation of the camera of the dual camera portable device capturing the video displayed on the UI, such as changing the focus of the camera.

  As described above in FIGS. 56 and 62, the defined regions of interest were used by the remote portable device to calculate video exposure and focus adjustments, respectively. However, in some other embodiments, the selection of the region of interest by the user may be used to instruct the remote device to perform one or more operations. For example, in some embodiments, both exposure adjustment and focus adjustment may be performed based on a defined region of interest, thereby instructing the remote device to perform both operations.

E. FIG. Frame Rate Control During videoconferencing, some embodiments provide for adjusting or maintaining the rate at which images of video captured by the camera of a dual camera portable device are transmitted to other devices in the videoconference (ie, frame rate). You may wish. For example, assuming a fixed bandwidth, some of such embodiments reduce the frame rate of the video to improve the quality of the image of the video, while others of such embodiments reduce the frame rate of the video. Increase the frame rate of the video to smooth the video (ie, reduce the jitter).

  Different embodiments provide different techniques for controlling the frame rate of a video image during a video conference. One example described previously adjusts the VBI of the camera's sensor module 415 to control the rate at which images captured by the camera are processed. As another example, some embodiments of the management layer 935 of the video conferencing module 925 shown in FIG. 9 control the frame rate by discarding images. Similarly, some embodiments of the image processing layer 930 control the frame rate by discarding the image. Some embodiments provide still other techniques for controlling the frame rate, such as discarding frames in the universal transmission buffer 3120.

V. Dual camera A. Combining views Picture-in-Picture: Displaying Two Remote Cameras According to some embodiments, a dual-camera mobile device may include a video captured from a mobile device during a video conference and a video captured from another dual-camera mobile device. It can be displayed in any of the display configurations. FIG. 65 shows examples of different display configurations for video captured from one or more dual camera portable devices. In FIG. 65, a user of the dual camera portable device 6500 (device A) and a second user of the second dual camera portable device 6505 (device B) are having a video conference with each other.

  FIG. 65 shows four examples of display configurations for device A on the left. The four display configurations for device A are a first view 6510, a second view 6515, a third view 6520, and a fourth view 6525. Further, FIG. 65 also shows four examples of display configurations for device B on the right. The four display configurations for device B are a first view 6565, a second view 6570, a third view 6575, and a fourth view 6580. In this example, device A displays only the two videos captured from device A's camera, and device B captures not only the two videos captured from device A's camera, but also the video captured from device B's camera. Display one or both of the videos.

  In the first view 6510, the UI 6585 of the device A provides a composite display 6512. The composite display 6512 has two display areas: a display area 6530 for displaying video captured from the rear camera of the device A, and a display area 6535 for displaying video captured from the front camera of the device A. including. In this example, display area 6530 is in the upper half of composite display 6512, and display area 6535 is in the lower half of composite display 6512. The two display areas have the same size in the first view 6510. The upper display area 6530 displays a video of the mountain, which is assumed to be the mountain being captured by the rear camera of device A. The display area 6535 displays a tree and a man wearing a hat, which is assumed to be a tree and a man wearing a hat captured by the front camera of the device A.

  The UI 6585 of the second view 6515 shows that the display area 6535 (displaying the video captured from the front camera of the device A) is now in the upper half of the composite display 6517, and the display area 6535 (captured from the rear camera of the device A). It provides a composite view 6517 that includes the same two display areas as the first view 6510, except that the display area 6530 (displaying video) is in the lower half of the composite view 6517.

  In the third view 6520, the UI 6585 provides a PIP display 6595. The PIP display 6595 displays two display areas: a display area 6535 for displaying a video captured from the front camera of the device A as a background display area, and a video captured from the rear camera of the device A as a foreground insertion display area. And a display area 6530. In this view, the background display area 6535 occupies most of the PIP display 6595, but the insertion display area 6530 is smaller than the background display area 6535 and partially overlaps the background display area 6535.

  The UI 6585 of the fourth view 6525 also presents a PIP display 6598 including the display areas 6530 and 6535 shown in the third view 6520. Unlike the PIP display 6595, the PIP display 6598 includes a display area 6530 (captured from the rear camera of device A) as a background main display and a display area 6535 (captured from the front camera of device A) with foreground insertion. Include as display. Further, PIP display 6598 is presented in a landscape view (ie, the width of PIP display 6598 is greater than its height).

  The above example shows four possible composite views for the UI of device A, two of which are vertically aligned with two display areas 6530 and 6535 for displaying the two cameras of the first device. And two are PIP views. Other views are also possible on the device A UI. For example, the two display areas can be arranged horizontally or diagonally, or different PIP views can be used.

  The various views shown for device B indicate that different views for device B's UI are possible. These views include video captured from both cameras on device A as well as one or more cameras on device B. In the first view 6565 of device B, the UI 6590 of device B provides a PIP display 6568. PIP display 6568 includes an insertion display area 6550 that displays the video captured by one of the cameras of device B (eg, the front camera), as well as a composite display area 6569 that is identical to the composite display 6512 displayed on device A. including. The composite display area 6569 includes a display area 6531 for displaying video captured from the rear camera of the device A, and a display area 6536 for displaying video captured from the front camera of the device A. The composite display 6569 that displays the video from device A occupies most of the PIP display 6568, but the insertion display area 6550 is smaller than the composite display 6569 and overlaps it. Display area 6550 displays a video of a smile, which is assumed to be a smile whose video has been captured by the front camera of device B.

  The device B UI 6590 in the second view 6570 provides a PIP display 6572. PIP display 6572 includes a display area 6550 (displaying video captured from the front camera of device B) and a composite display 6573 having display areas 6531 and 6536 displaying video captured from the camera of device A. . The composite display 6573 is the same as the composite display 6517 of the second view 6515 for the device A, and occupies most of the PIP display 6572. Similar to the PIP display 6568 of the first view 6565, the display area 6550 is smaller than the composite display 6573 and overlaps the display. Specifically, in both views, the display area overlaps a portion of the display area 6531 displaying video captured from the rear camera of device A.

  In the third view 6575, the UI 6590 provides a PIP view 6577 that is similar to the PIP view 6595 of the third view 6520 for device A. PIP display 6577 also includes an additional display area 6550 as a second insertion display area that overlaps background display area 6536. The two insertion display areas 6531 and 6550 are tiled horizontally below the main background display area 6536.

  The UI 6590 of the fourth view 6580 provides a composite display 6852. The composite display 6582 includes three displays: a PIP display 6585, a display area 6550, and a display area 6540 (eg, for displaying video captured by the rear camera of device B). PIP display 6581 is the same as PIP display 6598 in fourth view 6525 for device A, and occupies most of composite display area 6582. The displays 6540 and 6550 are smaller than the PIP display area 6583 and are tiled horizontally below.

  FIG. 65 shows four possible views for device B, but many other views are possible. The background composite display of the video from device A can be tiled horizontally rather than vertically, the insertion can overlap the front camera display area of device A instead of the rear camera display area, and the larger display area is device A , The camera (s) of device B can be displayed, and the insertion can be positioned differently.

  Each set of arrows 6560 resulting from each view of device A indicates that there is no need for a correlation between the display shown on device A and the display shown on device B. For example, even if device A is displaying its video in the configuration of view 6510 (eg, according to the selection of that configuration by the user of device A), device B is still displayed (eg, by the user of device B). The video can be displayed in any of the four illustrated configurations (or depending on the configuration choices) or any of a number of other configurations not shown in FIG. In other words, the display configuration for device A is independent of the display configuration for device B. Some embodiments do not transmit the display area from one device to another, but simply transmit the video (eg, in encoded form) and are displayed by the device in its corresponding display area.

2. Special PIP
According to some embodiments, a user of a dual camera portable device can superimpose a video foreground on another video in a PIP display during a video conference. In some embodiments, the foreground of a video mixes with other videos in such a way that it appears as a representation of a single video captured by a single camera. FIG. 66 shows such an example of superimposition of an insertion video on a background video on a foreground in a PIP display.

  FIG. 66 illustrates this video superimposition operation as seven operating stages 6620, 6625, 6630, 6635, 6640, 6660, and 6665 of UI 6670. The first stage 6620 shows a UI 6670 of a dual camera portable device 6600 with a PIP display 6682 during a video conference with a remote device. As shown in the first step 6620, the PIP display 6682 includes two video displays, a background main display 6610 and a foreground insert display 6605. The background main display 6610 occupies most of the UI 6670, but the foreground insertion display 6605 is smaller than the background main display 6610 and overlaps it.

  In this example, the background display area 6610 is displaying a video of the mountain, which is assumed to be a mountain being captured by one of the remote device's cameras. Foreground insert display area 6605 displays a video of the person wearing the hat, which in this example is assumed to be the person whose video is captured by one of the local device's cameras. Below the PIP display 6682 is a selectable UI item 6665 (e.g., a button 6665), labeled as end of conference, which allows the user to end the video conference by selection (e.g., by single or double tapping a button). .

  The second step 6625 shows the activation of the selectable menu 6675. In some embodiments, a menu of selectable UI items 6675 may be activated by selecting (eg, by touching) PIP display area 6682. In lieu of or in conjunction with such a wake-up operation, in some embodiments, the user may have access to other touch-screen operations, such as through different touch screen operations or using one or more other physical inputs of the device. A menu of selectable UI items 6675 can also be activated through operation.

  The third step 6630 displays a UI 6670 with an activated set of selectable UI items for selecting a video overlay operation. In this example, a pop-up menu 6675 with several selectable UI items is displayed on PIP display 6682. The menu of the selectable UI item 6675 includes a “PIP flip” selectable UI item 6640 (eg, button 6640), a “special PIP” selectable UI item 6645 (eg, button 6645), and a “cancel” selectable UI item 6690 ( For example, button 6690). In this example, by selecting the "PIP Flip" button 6640, the UI 6670 replaces the background display 6610 with the insert display 6605 (as described in more detail in the next section) and selects the "Special PIP" button 6645. Accordingly, the UI 6670 starts the video superimposing operation, and deletes the pop-up menu 6675 from the PIP display 6682 by selecting the “cancel” button 6690. Other embodiments include different or more items in the PIP pop-up menu 6675.

  The fourth step 6635 shows the UI 6670 after the user has selected the “special PIP” button 6645 (eg, by tapping the button 6645 with his finger 6695). This selection is indicated by highlighting button 6645 on UI display 6670. Some embodiments use different indications of indication (e.g., highlighting the boundaries of a selected item or text of a selected item).

  The fifth step 6640 shows the UI 6670 after the video overlay operation has started. At this stage, the UI 6670 allows the user to select which video to extract as the foreground and which video to use as the background in the video to be superimposed. The UI 6670 provides options through a pop-up menu 6680 with several selectable UI items displayed on the PIP display 6682. The selectable UI item pop-up menu 6680 includes a “select insert” selectable UI item 6655 (eg, button 6655), a “select main” selectable UI item 6650 (eg, button 6650), and a “cancel” selectable UI. Item 6692 (for example, button 6692).

  Selection of the “Select Insert” button 6655 causes the UI 6670 to superimpose the foreground of the inserted video 6605 from the camera of the local device (ie, the man wearing the hat) on the background main video 6610 from the camera of the remote device. On the other hand, selection of the “select main” button 6650 causes the UI 6670 to superimpose the foreground (ie, mountains) of the background main video 6610 from the camera of the remote device on the insertion video 6605 from the camera of the local device. In some embodiments, this causes the video currently in the inset display area 6605 to occupy most of the UI 6670, and the video currently in the main display area 6610 to be superimposed on the main video at this point. Video feed switching occurs. Selection of the “cancel” button 6692 cancels the video superimposing operation and deletes the pop-up menu 6680 from the PIP display area 6682.

  The sixth step 6660 shows the UI 6670 after the user has selected the “select insert” button 6655 (eg, by tapping the button 6655 with his finger 6695). This selection is indicated by highlighting button 6655 on UI display 6670. Some embodiments use different signage (e.g., the border of the selected item or the text highlighting of the selected item).

  The seventh step 6665 shows the UI 6670 after the video overlay operation has been completed. As shown in UI 6670, the foreground of insert display area 6605 (ie, a man wearing a hat) is extracted from display area 6605. The window frame and background of insert display 6605 (ie, everything but the foreground) are also deleted from the screen. Finally, the foreground (ie, the man wearing the hat) is mixed with the background video 6610 in a manner that appears to be a single video. A variety of different techniques can be used to remove the background of the inserted video. Some embodiments identify pixels that have not moved relative to other pixels, look for patterns or colors that are constant, use a baseline image that is compared to the image containing the foreground, and subtract the differences Or use a different technique.

  Although the example of FIG. 66 shows that the foreground of the insertion display area 6605 stays at the same place in the UI 6670 when superimposed on the background display area 6610, this is only one example of how the superimposition can function. Some embodiments move the foreground video to a particular location within the UI 6670 (eg, center, one of the corners, etc.). Section IV. A. 1 and IV. A. Similar to the functionality shown in FIG. 3, some embodiments allow the user of the local device to drag the superimposed foreground video or change the size of the superimposed foreground video in the UI.

  Different techniques may be used to determine which portion (s) of the video image is "foreground" for the video overlay operation described above. One such method of some embodiments determines which part, if any, of the video image is dynamic. Since the background of a video image is generally static (i.e., no motion), the dynamic parts are considered "foreground". In such an embodiment, the video image is analyzed over a specific time period. A particular pixel is considered to be a static pixel if the difference between the values of the particular pixel for a particular period is not greater than a specified threshold (eg, 5%, 10%, 15%). . After each pixel of the video image has been analyzed, the dynamic (ie, non-static) pixels of the video image are considered to be the "foreground" of the video image.

  FIG. 67 illustrates one example of such a technique for determining a foreground of a video image, which may be performed by, for example, video conferencing manager 1604 or image processing manager 1608. Specifically, FIG. 67 shows a sequence of six images 6705-6730 of a video showing a person wearing a hat and a tree. In this example, it is assumed that the person is not standing still at all, and may speak. As described above, each pixel of the video image is analyzed to determine whether the pixel is dynamic or static. For example, it is determined whether or not the difference between the values of the pixels 6735 of the images 6705 to 6730 is larger than a specified threshold. In this case, pixel 6735 is considered static because pixel 6735 represents a part of the ground, not a person. After all of the pixels in images 6705-6730 have been analyzed, it is determined that the people in the image are dynamic and the rest of the image is static. Therefore, the person is the “foreground” extracted by the operation described above with reference to FIG.

3. Exchanging Video in Picture-in-Picture Display According to some embodiments, a user of a dual camera portable device can exchange two display areas in a PIP display during a video conference (ie, the inset display area is a background in the PIP display). Display area, and the background display area becomes the insertion display area). FIG. 68 shows an example of exchanging the insertion display area 6605 of the PIP display 6682 with the background display area 6610 during a video conference.

  FIG. 68 illustrates the PIP exchange operation as the eight stages of operation of the UI 6670 of the device 6800 of FIG. The first three stages in FIG. 68 are the same as the first three stages in FIG. At these stages, the user has expanded the menu 6675 in the UI 6670 through selection using the touch screen of the local device.

  The fourth step 6840 in FIG. 68 shows the UI 6670 after the user has selected the “PIP Flip” button 6640 (eg, by tapping the button 6640 with his finger 6695). This selection is indicated by highlighting button 6640 on UI display 6670. Some embodiments use different signage (e.g., the border of the selected item or the text highlighting of the selected item).

  The fifth stage 6845 shows the UI 6670 after the PIP exchange operation has started. Some embodiments animate the exchange of an insert view 6605 and a background view 6610 through a flip operation. FIG. 68 shows an example of such an animation. In this example, the animation is a flip of the view pane with the PIP display 6682 (before the exchange operation is performed) on one side and the new PIP display 6684 (after the exchange operation is performed) on the other side. Can be explained through The view pane rotates 180 degrees about a vertical axis 6686 in the center of the PIP display 6682. In this fifth step 6845, the view pane begins to rotate around the vertical axis 6686.

  In a sixth step 6850, the view pane is shown as rotated about 90 degrees. This is indicated by the thin line 6688 (ie, the edge of the view pane) displayed in the center of the screen. The seventh step 6855 shows the rotation of the view pane near completion. The new PIP display 6684 begins to be visible from the other side of the view pane and expands horizontally to fill the device screen. PIP display 6684 includes two display areas 6605 and 6610 after the exchange operation has been performed. The display area 6605 presenting the video of the man wearing the hat (from the camera of the local device) is now in the background of the PIP display 6684, and the display 6610 presenting the video of the mountain (from the camera of the remote device) is At this point, it is in the foreground of the PIP display 6684 overlapping the display 6605. The eighth step 6860 indicates the completion of the exchange display animation.

  Those skilled in the art will appreciate that the animation shown in FIG. 68 is only one of many possible animations of the PIP insertion / background exchange operation. For example, different embodiments may rotate the view pane along a horizontal axis, swap two views instantly, expand one view while shrinking the other view, and so on. While some embodiments provide one animation that is always used for the swap operation, other embodiments allow the user to select from several animations or to select a different animation (eg, through random selection). ) Can be used. Moreover, the swap operation may cause a change to the image processing operation of the dual camera portable device, such as causing the video conferencing manager 1604 to change the video scaling and composition in response to user input.

4. Corner Snaps Some embodiments of the present invention allow a user of a dual camera portable device to change a composite display by moving one or more display areas that form the composite display. An example of such a movement is described in Section IV. A. 1 described above. Such a movement of the insertion display is also possible when the PIP display includes a plurality of insertion display areas.

  FIG. 69 shows such an example performed during a video conference. This example, shown in FIG. 69, shows that FIG. 69 moves not only one such insertion display area, but the insertion display area 6910 of the PIP display 6965, which includes two insertion display areas 6905 and 6910. Except for this, it is the same as the example shown in FIG.

  In FIG. 69, the UI 6960 of the mobile device 6900 presents a PIP display 6965 during a video conference with a remote user of another device. The PIP display 6965 of FIG. 69 includes three video displays, a background main display 6915, and two foreground inserts 6905 and 6910. In this example, the background main display 6915 presents a video of the person singing and playing the guitar, which is assumed to be the video captured by the rear camera of the remote device. The front inset display 6905 presents a video of the person holding the racket, which in this example is assumed to be the video captured by the rear camera of the local device. The other foreground insert display area 6910 presents a video of the person wearing the hat, which in this example is assumed to be the person wearing the hat whose video has been captured by the front camera of the local device. . Below the PIP display 6965 is a selectable UI item 6970 (eg, button 6970) labeled “End Conference” that allows the user to end the video conference by selecting an item.

  This PIP display 6965 is just one way to present a composite view of the video being captured by the remote and local devices. Some embodiments may provide other composite views. For example, instead of having a larger background display 6915 for video from a remote device, the larger background display 6915 is for video from the local device and the smaller foreground inserts 6905 and 6910 are Can be of video from the device. Also, according to some embodiments, the local video and the remote video have the insert displays 6905 and 6910 on one side and the background display 6915 on another side, or have all three adjacent. It can be displayed on the UI 6960. In other embodiments, the PIP display 6965 may also include a larger background display 6915 and / or a smaller foreground insert display. In some embodiments, the manner of PIP display 6965 or the default display mode may be specified by the user.

  FIG. 69 illustrates the movement of one of the two inset display areas of the UI 6960 of the device 6900 as five different operating phases 6920, 6925, 6930, 6935 and 6940. The first stage 6920 shows a UI 6960 during a video conference between a local user of the device 6900 and a remote user of a remote device.

  The second step 6925 indicates that the user initiates the corner snap operation by selecting the insertion display area 6910. In this example, the selection is made by placing finger 6950 at an arbitrary position in insertion display area 6910. As shown, this selection is displayed as the thick border 6962 of the insert display 6910. Different embodiments may indicate such selections in different ways, such as by highlighting insert display 6910, by vibrating insert display 6910, and so on.

  The third step 6930 shows the UI 6960 after the user has begun to move the insertion display area 6910 of the PIP display 6965 from one area in the PIP display 6965 to another in the PIP display 6965. In this example, insertion display area 6910 has begun to move from the lower right corner of PIP display 6965 to the upper left corner of the display, as indicated by arrow 6955. Insert display 6910 is moved by the user dragging his or her finger 6950 toward the upper right corner of PIP display 6965 after selecting insert display 691. Some embodiments provide other techniques for moving the insert view 6910 in the PIP view 6965.

  The fourth step 6935 shows the UI 6960 in a state after the user has released his / her finger 6950 from the screen of the device 6900. In this state, the insertion display area 6910 is still moving toward the upper right corner of the PIP display 6965 identified based on the movement of the user's finger in the third stage. In other words, after finger 6950 has begun to move insert display 6910 toward the upper right corner of PIP display 6965, UI 6960 maintains this movement even after finger 6950 separates. To maintain this movement, the UI 6960 of some embodiments may indicate that the user's drag motion is greater than a certain threshold amount (e.g., a certain distance or a certain amount of time before the user releases the finger). Bigger than that). Otherwise, these embodiments keep the inset display area in its original lower right corner position after slightly or no movement of the inset display area.

  However, in some embodiments, the insertion display area can move even after the user stops his or her drag operation before the insertion display area reaches its new position, while in other embodiments, the insertion display area can be moved. Ask the user to maintain his or her drag action until the display area reaches its new position. Some embodiments provide yet another technique for moving the insertion display area. For example, in some embodiments, the user may be required to specify where the display area 6910 should be directed before the display area 6910 actually moves. In some embodiments, the viewing area can be slid simply by tilting the mobile device at different angles, and can be snapped for corners.

  The fifth step 6940 shows the UI 6960 after the insertion display area 6910 has reached its new position at the upper right corner of the PIP display area 6965. Removal of the thick border line 6962 in the fifth stage indicates that the corner snapping operation has been completed.

  To facilitate the movement shown in the third step 6930, the fourth step 6935, and the fifth step 6940 described above, the UI 6960 of some embodiments allows the user to place the insertion display area 6910 at the corner of the PIP display 6965. A snap rule is adopted in which the insertion display area 6910 can be quickly snapped to its corner when moved toward. For example, if the user drags the insertion display area 6910 beyond a threshold amount toward a particular corner, the UI 6960 of some embodiments identifies the direction of movement of the insertion display 6910, and It is determined that the threshold amount has been exceeded, and thereafter, the insertion display area 6910 is automatically moved without further input of the next grid point in the UI 6960 where the insertion display 6910 can be snapped. In some embodiments, the only grid points provided to snap the inset display 6910 are the grid points at the four corners of the PIP display 6965. Other embodiments provide other grid points in the UI 6960 (eg, in the PIP display 6965) where the inset display 6910 can be snapped.

  Still other embodiments may not employ grid points so that the insertion display area 6910 can be located at any point in the PIP display. Still other embodiments provide the ability for a user to turn on or off snapping to a grid point feature in the UI. Further, different embodiments allow a user to perform corner-oriented snap operations on various items, such as icons, in addition to video captured from the device. As described above, by moving the display area (group) of the composite display, the image processing operation of the dual camera portable device, such as causing the video conference manager 1604 to re-synthesize the display area in the composite display in response to the user's input. Can cause changes.

5. Push and Snap The example of FIG. 69 is a corner snap that allows a user of a dual camera portable device to move one of the two inset display areas from one corner of the PIP display to another corner not occupied by the inset display. The operation is shown. Some embodiments allow a push function to move the first insert to the position of the second insert and to push the second insert to a new position. FIG. 70 illustrates one such example performed during a video conference.

  FIG. 70 illustrates the movement of the insert display from one corner of the PIP display to another corner occupied by another insert display of the PIP display in six different stages 7020, 7025, 7030, 7035, 7040 and 7045 of this UI 6960. Shown as The first stage 7020 shows a UI 6960 during a video conference between a local user of the device and a remote user of a remote device. 70 shows PIP display 6965, which is the same PIP display shown in the first stage of FIG. 69 after the video conference has begun. In this example, video captured by the local user's device is displayed in the insert display areas 6905 and 6910, and video captured by the remote user's device is displayed in the background display area 6915.

  The second step 7025 indicates that the user starts the corner snap operation by selecting the insertion display area 6905. In this example, the selection is made by placing the finger 7055 at an arbitrary position in the insertion display area 6905. As shown, this selection is displayed as a thick border 7065 in the insert display 6905. Different embodiments may indicate such selections in different ways, such as by highlighting display area 6905, by vibrating display area 6905, and so on.

  The third step 7030 is for the user to select the insert view 6905 and then drag his finger 7055 toward the lower right corner of the PIP view 6965, as indicated by the arrow 7050 (as shown by the arrow 7050). 19 shows the UI 6960 after the start of moving the insertion display area 6905 from the lower left corner of the PIP display 6965 to the lower right corner of the PIP display 6965. Some embodiments provide other techniques for moving the insertion display area 6905 in the PIP display 6965.

  The fourth step 7035 shows the UI 6960 after the insertion display area 6905 has contacted the insertion display area 6910. Upon contact, the insertion display area 6910 moves toward the next closest corner. In this example, the insertion display area 6910 begins to move toward the upper right corner of the PIP display 6965 (indicated by arrow 7075). The activation of this push operation is displayed as a thick border 7070 of the insertion display area 6910. Different embodiments may indicate such activation in different ways, such as by highlighting display 6910.

  The fifth step 7040 shows the UI after the insertion display area 6905 has snapped to the lower right corner previously occupied by the insertion display area 6910. In this state, the insertion display area is still moving toward the upper right corner of the PIP display 6965. Also, the thick border 7065 is no longer displayed. As long as the user's drag action from the third step 7030 is greater than the threshold that causes the insertion display area 6905 to snap to the right corner, the insertion display area 6910 is removed from that corner and bother snapping to the next closest corner.

  Some embodiments include a set of rules for determining in which direction to push the second insertion display area 6910. In the case shown in FIG. 70, some embodiments attempt to continue rotating the insertion display area. That is, since display area 6905 is moved counterclockwise, display area 6910 is also moved counterclockwise. Some embodiments provide a hierarchy of possible positions where the pushed insertion display area 6910 can be moved to select the first unoccupied position on the list. For example, if the lower right insert display area is pushed by the lower left insert display area, the upper right corner may be the first position in such a list. However, if a third insert view already exists in the upper right corner, some embodiments will move to the next option on the list (eg, upper left corner, center, or lower left corner). Other embodiments push the third insert display area with the second insert display area, so that the device needs to determine a new position for the third insert display area.

  The sixth step 7045 shows the UI 6960 after the insertion display area 6910 has reached its new position at the upper right corner of the PIP display area 6965. Removal of the thick border 7070 at this stage indicates that the corner snap and push operation has been completed. Similar to the corner push operation described with reference to FIG. 68, moving the display area (group) of the composite display causes the video conference manager 1604 to re-combine the display area in the composite display in response to user input. Changes to the image processing behavior of the camera portable device can be caused.

6. Rotation Some embodiments rotate the PIP display presented during a video conference when the user of the mobile device used for the video conference rotates the device during the conference. FIG. 71 shows the rotation of the UI display 7175 of the device 7100 when the device is rotated from a vertical position to a horizontal position. When the long side of the screen is vertical, the device 7100 is held vertically, but when the long side of the screen is horizontal, the device 7100 is held horizontally. In the example illustrated in FIG. 71, the UI display 7175 rotates from a vertical view optimized for vertical holding of the device to a horizontal view optimized for horizontal holding of the device 7100. This rotation feature allows the user to see the UI 7175 displayed in an upright position when the portable device 7100 is held either vertically or horizontally. This example shown in FIG. 71 is similar to the example shown in FIG. 34, except that FIG. 71 shows rotating a PIP display that includes two insertion display areas rather than just one insertion display area. is there.

  In FIG. 71, UI 7175 of mobile device 7100 presents PIP display 7180 during a video conference with a remote user of another mobile device. The PIP display 7180 of FIG. 71 includes three video displays, a background main display 7115, and two foreground inserts 7110 and 7160. In this example, the background main display 7115 presents a video of the mountain, which is assumed to be the video captured by the front or rear camera of the remote device. The front insertion display 7110 presents a video of a smile in the room, which is assumed to have been captured by the front or rear camera of the local device. The other foreground insert display 7160 displays the video of the singing guitar player, which is assumed to be the guitar player whose video is being captured by the other camera on the local device. Below the PIP display 7180 is an end conference button 7155 that the user may select (eg, through a single finger tap) to end the video conference. This PIP display is just one way to present a composite view of the video being captured by the remote and local devices. Some embodiments may provide other composite views, such as a tiled view or a different PIP display.

  FIG. 71 shows the rotation of the UI 7175 as six different operating stages 7120, 7125, 7130, 7135, 7140 and 7145. The first stage 7120 shows a UI 7175 during a video conference between a local user of the device and a remote user of a remote device.

  The second stage 7125 shows the UI 7175 after the user has begun to tilt the device 7100 sideways. In this example, device 7100 has begun to tilt device 7100 from a vertically held state to a horizontally held state, as indicated by arrow 7185. The appearance of UI 7175 has not changed. In other situations, the user may wish to tilt the device 7100 from a horizontally held state to a vertically held state, and in these situations the UI display 7175 may be optimized horizontally. Switch from the optimized view to the vertically optimized view.

  The third step 7130 shows the UI 7175 after the device 7100 has been tilted from being held vertically to being held horizontally. In this state, the appearance of the UI display 7175 has not changed yet. In some embodiments, the rotation operation is triggered after the device 7100 is tilted past a threshold amount and kept above this point for a period of time. In the example shown in FIG. 71, it is assumed that the UI display 7175 does not rotate due to the threshold amount and the rotation speed until a certain short interval elapses after the device is placed in the horizontal position. Other embodiments have different threshold amounts and waiting periods that trigger the rotation operation. For example, some embodiments may have a low threshold for triggering a rotation operation so that the UI 7175 always appears to be displayed in an upright position regardless of the orientation of the device 7100. . In other embodiments, a user of device 7100 may specify when a rotation operation can be triggered (eg, through menu preferences). Also, some embodiments may not delay rotation after the device is tilted beyond a threshold amount. Further, by using different embodiments, it is possible to trigger a rotation operation in different ways, such as by switching a switch on a mobile device, giving a voice audio command, in response to a menu selection, etc. It may be.

  The fourth step 7135 shows the UI 7175 after the rotation operation has started. Some embodiments animate the rotating display area to provide feedback to the user regarding the rotating motion. FIG. 71 shows an example of such an animation. Specifically, FIG. 71 shows, during its fourth stage 7135, that the display areas 7110, 7115 and 7160 begin to rotate together. Display areas 7110, 7115, and 7160 rotate about axis 7165 (ie, the z-axis) passing through the center of UI display 7175. Viewing areas 7110, 7115 and 7160 rotate the same amount in the opposite direction of rotation of device 7100 (eg, through tilting of device 7100). In this example, the device 7100 has been rotated 90 degrees clockwise (by moving from a vertically held state to a horizontally held state), and this rotation causes the display areas 7110, 7115, and 7160 to be inverted. Rotated 90 degrees clockwise. As the display areas 7110, 7115, and 7160 rotate, the display areas 7110, 7115, and 7160 shrink proportionately to fit the UI display 7175 so that the display areas 7110, 7115, and 7160 are still fully visible on the UI 7175. Some embodiments may provide a message indicating the status of the device 7100 (eg, by displaying the word “rotating”).

  Fifth stage 7140 shows UI 7175 after display areas 7110, 7115, and 7160 have been rotated 90 degrees counterclockwise from portrait view to landscape view. At this stage, the display areas 7110, 7115, and 7160 have been rotated, but have not yet expanded to the full width of the UI 7175. Arrow 7170 indicates that at the end of the fifth phase, display areas 7110, 7115 and 7160 begin to expand sideways to fit the full width of UI 7175. Different embodiments may not include this step, since the expansion can be performed simultaneously with the rotation in the fourth step 7135.

  The sixth step 7145 shows the UI 7175 after the display areas 7110, 7115 and 7160 have been expanded to occupy the entire display of the UI 7175. As mentioned above, other embodiments may implement this rotation differently. In some embodiments, simply rotating the screen of the device beyond a threshold amount may trigger a rotation operation regardless of the orientation of the device 7100.

  Also, other embodiments may provide different animations to show the rotation motion. The rotation operation performed in FIG. 71 includes rotating the UI display 7175 around the center of the UI display 7175. Alternatively, the display areas may be individually rotated about the central axis of these individual display areas. One such technique is illustrated in FIG. 72, which shows an alternative method of animating the rotation of the PIP display area 7180 of the UI 7175. The PIP display shown in FIG. 72 is the same PIP display 7180 as shown in FIG.

  FIG. 72 shows the PIP display 7180 as six different operating phases 7120, 7125, 7130, 7220, 7225 and 7230. The first three stages of the operation of the UI 7175 are the same as the first three stages of the operation described with reference to the UI 7175 of FIG. In the third phase for both FIGS. 71 and 72, the device has moved from a vertically held state to a horizontally held state, and rotation of UI 7175 has not yet begun.

  The fourth step 7220 shows an alternative way of animating the rotation. At this stage, the rotation operation has started. Specifically, the fourth step 7220 shows where the display areas 7110, 7115 and 7160 begin to rotate. Viewports 7110, 7115 and 7160 each rotate about an axis 7250 (ie, the z-axis) passing through the center of each of the displayportions. Viewing areas 7110, 7115, and 7160 rotate the same amount in the opposite direction of rotation of device 7100 (eg, due to tilt of device 7100). In this example, the device 7100 has been rotated 90 degrees in the clockwise direction (by moving from a state held vertically to a state held horizontally), and this rotation action causes the display areas 7110, 7115, and 7160 to be inverted. Rotate 90 degrees clockwise. As the display areas 7110, 7115 and 7160 rotate, they also shrink proportionally to fit the UI display 7175 so that the display areas 7110, 7115 and 7160 are still fully visible on the UI 7175.

  The fifth step 7225 shows the UI 7175 after the display areas 7110, 7115 and 7160 have been rotated 90 degrees counterclockwise from the vertical view to the horizontal view. At this stage, the display areas 7110, 7115, and 7160 have been rotated, but have not yet expanded to the full width of the UI 7175 or reached their final position. The final position of the display area in the PIP display 7115 is determined by the position of the display area in the PIP display shown in the first step 7120 (eg, the insertion display area 7110 at the lower left corner of the PIP display 7180 and the insertion display area at the lower right corner of the PIP display 7180). 7160).

  Arrow 7170 indicates that at the end of the fifth stage, display areas 7110, 7115 and 7160 begin to expand sideways until main display area 7115 fits the entire width of UI 7175. Further, arrow 7255 indicates that insertion display areas 7110 and 7160 have moved to reach the final position of PIP display 7180. In other words, the insertion display area 7110 moves downward toward the lower left corner of the PIP display 7180, and the other insertion display area 7160 moves to the lower right corner of the PIP display 7180. Different embodiments may perform this animation differently, for example, by using the snap and push operation shown in FIG. The sixth step 7230 shows the UI 110 after the display areas 7110, 7115 and 7160 have been expanded to occupy the entire final display of the UI 7175 and have reached their final position.

  As mentioned above, other embodiments may implement this rotation differently. For example, as also shown in FIGS. 36 and 37, some embodiments are captured by the local device to reflect the orientation of the local device after the rotation operation has been performed on the local device. Providing a rotation operation in which the orientation of the viewing area displaying the video is changed, some embodiments provide a remote device to reflect the orientation of the remote device after the rotation operation is performed on the remote device. Provides a rotating motion in which the orientation of the display area displaying the captured video is changed, and some embodiments provide a rotating motion in which the display area 1155 remains in the same position; Provides a different layout in the display area (eg, the layout of the display area 1155 in FIG. 12), or a combination thereof.

  In some embodiments, simply rotating the screen of the device beyond a threshold amount may trigger a rotation operation regardless of the orientation of the device 7100. Also, as described above, the local device and the remote device notify each other via the control communication channel that a rotation operation has been performed on one of the devices, and the other device has Make any corresponding changes possible. The animation of the rotation operation causes the video conference manager 1604 to re-synthesize the display area (group) at a different angle in the UI 1105 and to enlarge or reduce an image displayed in the display area (group), or to perform a dual operation. A change in the image processing behavior of the camera device can be caused.

7. Selecting a Remote View to Watch As described above, some embodiments allow a user of a dual camera portable device to select which camera to use for a video conference before or at the start of the video conference. . Instead of or in conjunction with this feature, according to some embodiments, the user of the device is the two videos being displayed in the video conference and from two video cameras on the remote device, From two video cameras on the local device. FIG. 73 shows the in-meeting selection of one video from two remote videos, and FIG. 74 shows the in-meeting selection of one video from two local videos.

  FIG. 73 illustrates remote video selection as six operating steps 7335, 7340, 7345, 7350, 7355 and 7360 of UI 7375 displayed on local device 7300. The first stage 7335 shows the UI 7375 with an initial PIP display 7390 being presented during a video conference with a remote user of a portable device having two cameras.

  As shown in the first step 7335, the initial PIP display 7390 includes three displays, a background main display 7315 and two foreground insert displays 7305 and 7310. The background display 7315 occupies most of the PIP display 7390, but the foreground insert displays 7305 and 7310 overlap a part of the background display 7315 on the UI 7375. In this example, the background display 7315 presents a video of the person in front of the microphone, which is assumed to be the video captured by the rear camera of the remote device. First front inset display 7305 presents a video of a male face, which in this example is assumed to be the video captured by one of the cameras of local device 7300. The second front insertion display 7310 presents a video of the person wearing the hat, which in this example is assumed to be the video captured by the front camera of the remote device.

  The initial PIP display 7390 is just one way to present a composite view of the video being captured by the cameras of the local and remote devices. Some embodiments may provide other composite views. For example, the background display may present video from one of the local device's cameras, and the smaller foreground insert display may present video from the front and rear cameras of the remote device. Also, in some cases, the PIP display includes only one background video display and one foreground video display, both of which come from remote devices. In some embodiments, the type of PIP display or default display mode may be specified by the user.

  The second step 7340 indicates the start of the video selection operation. In this example, the operation is initiated by activating a set of selectable UI items to be displayed on PIP display 7390. The set of selectable UI items presents options for selecting a remote video for display. In some embodiments, the set of selectable UI items may be activated (eg, by touching) by selecting any display area playing remote video on UI 7375. In other embodiments, the item may be activated by selecting any location on the UI 7375 (eg, by touching). Instead of, or in conjunction with, such a wake-up operation, in some embodiments, the user may perform other operations, such as through different touch screen operations or using one or more other physical inputs of the device. A set of selectable UI items can also be activated through.

  The third step 7345 displays a UI 7375 with an activated set of selectable UI items 7380 for selecting a remote video. In this example, a set of selectable UI items 7380 in the form of a pop-up menu is displayed in PIP display area 7390 and overlaps the PIP display. A set of selectable UI items 7380 (which can be implemented as selectable buttons) includes a “select R1” selectable UI item 7320 (eg, button 7320) and a “select R2” selectable UI item 7325 (eg, button 7325). And a “select both” selectable UI item 7330 (eg, button 7330) and a “cancel” selectable UI item 7385 (eg, button 7385). In this example, selecting the “Select R1” button 7320 causes the UI 7375 to display only the video captured by the remote device's rear camera (presented in the background display 7315). By selecting the "Select R2" button 7325, the UI 7375 displays only the video captured by the remote device's front-facing camera (presented in the foreground insert display 7310). By selecting the "select both" button 7330, the UI 7375 continues to display both videos captured by the front and rear cameras of the remote device. The operation is canceled by selecting a “cancel” button 7385. In some embodiments, videos captured by the local device are not affected by selections made in this menu.

  The fourth step 7350 shows the UI 7375 after the user has selected the “Select R1” button 7320 (eg, by tapping the button 7320 with his finger 7365). This selection is indicated by highlighting button 7320 on UI 7375. Some embodiments use different signage (e.g., the border of the selected item or the text highlighting of the selected item).

  The fifth step 7355 shows the animation of the UI 7375 after the user has selected a video from R1 for display. In this example, UI 7375 removes unwanted foreground insertion display area 7310 by sliding out of the right edge of PIP display 7390 as indicated by arrow 7370. Other embodiments utilize different animations to remove unwanted insertion display areas, such as fade out or fade over the insertion, move the insertion in a different direction, or simply remove the insertion instantly.

  A sixth step 7360 displays the UI 7375 during the video conference after the video selection operation has been completed. Video display area 7310 is no longer displayed on UI 7375. At this stage, the UI 7375 presents a new PIP display 7395 that includes the video display area 7315 as the background main display and the video display area 7305 as the insertion display.

  Also, in some embodiments, the video selection operation causes only the selected video to be displayed on the remote device, while in other embodiments, the operation has no effect on the remote device. In some embodiments, this video selection operation causes the remote device to stop transmitting unwanted video to the local device. In fact, in some embodiments, this video selection operation causes the camera of the remote device to stop capturing unwanted video. In some embodiments, these effects on the remote device may be overridden by the user of the remote device.

  The above example shows a case where the selected remote view is a view already displayed in the background main display. In some embodiments, when the user selects a remote view displayed in one of the inset displays, the selected remote view is displayed in the background main display. Some such embodiments use an animation in this case, such as the animation displayed in FIG. Further, selecting the remote video (s) causes the video conferencing manager 1604 to respond to user input, causing the video conference manager 1604 to composite only the selected remote video (s) in the composite display, such as an image of the dual camera portable device. Changes in processing behavior can be caused.

8. Selection of Local View to Watch FIG. 74 shows the selection of local video as the six operating phases 7435, 7440, 7445, 7450, 7455 and 7460 of the UI 7475 displayed on the local device 7400. The first stage 7435 shows a UI 7475 with an initial PIP display 7490 being presented during a video conference with a remote user of a portable device having at least one camera. PIP display 7490 differs from FIG. 73 in that background display 7415 presents a male video captured by the camera of the remote device and left foreground insert display 7410 has the guitar captured by the rear camera of the local portable device. 73 is similar to the first stage of FIG. 73 except that the right foreground insert display 7405 presents a video of a man wearing a hat captured by the front camera of the local mobile device 7400. . Thus, only one remote video is displayed and two local videos are displayed.

  The second step 7440 indicates the start of the video selection operation. In this example, the operation begins by activating a set of selectable UI items to be displayed on PIP display 7490 to select a remote video for display. In some embodiments, the set of selectable UI items may be activated (eg, by touching) by selecting any display area playing local video on UI display 7475. In other embodiments, the item may be activated by selecting (eg, by touching) any location on the UI display 7475. Instead of or in conjunction with such a wake-up operation, in some embodiments, the user may be able to perform other operations, such as by different touch screen operations or using one or more other physical inputs of the device. A set of selectable UI items can also be activated through.

  The third step 7445 displays a UI 7475 with an activated set of selectable UI items 7480 for selecting a local video. In this example, a set of selectable UI items 7480 in the form of a pop-up menu is displayed in a PIP display area 7490 that overlaps the PIP display. The set of selectable UI items 7480 includes “select L1” selectable UI item 7420 (eg, button 7420), “select L2” selectable UI item 7425 (eg, button 7425), and “select both”. A UI item 7430 (eg, button 7430) and a “cancel” selectable UI item 7485 (eg, button 7485) for canceling the operation are included. In this example, by selecting the "select L1" button 7420, the UI 7475 displays only the video captured by the local device's rear camera (presented in the foreground insert display 7410). By selecting the "select L2" button 7425, the UI 7475 displays only the video captured by the local device's front camera (presented in the foreground insert display 7405). By selecting the "Select Both" button 7430, the UI 7475 will continue to display both videos captured by both cameras on the local device, and the operation will be canceled by selecting the "Cancel" button 7485. You. In some embodiments, videos captured by the remote device are not affected by selections made through this menu.

  The fourth step 7450 shows the UI 7475 after the user has selected the “select L2” button 7425 (eg, by tapping the button 7425 with his finger 7465). This selection is indicated by highlighting button 7425 on UI display 7475. Some embodiments use different signage (e.g., the border of the selected item or the text highlighting of the selected item).

  A fifth step 7455 displays the UI 7475 animation after the user has selected a video from L2 for display. In this example, UI 7475 removes unwanted foreground insert display 7410 by sliding out of the left edge of PIP display 7490, indicated by arrow 7470. Other embodiments utilize different animations to remove unwanted insertion display areas, such as fade out or fade over the insertion, move the insertion in a different direction, or simply remove the insertion instantly.

  The sixth step displays the UI 7475 during the video conference after the video selection operation is completed. Video display area 7410 is no longer on UI 7425. At this stage, the UI 7475 presents a new PIP display 7495 including the remote video display 7415 as the background main display and the local video display 7405 as the insert display. In some embodiments, this video selection operation only affects the local display, since both video captures are still being transmitted to the remote device. Other embodiments stop capturing from the removed camera.

  The above example shows a case where the selected local view is a view already displayed in the background main display. In some embodiments, if the user selects a local view displayed in one of the inset displays, the selected local view is displayed in the main background display. Some such embodiments use an animation in this case, such as the animation displayed in FIG. Other embodiments use the inserted remote view if the local view in the background main display has been removed.

  Similar to the remote view selection operation described above with respect to FIG. 73, selecting a local video (s) will cause only the selected remote video (s) to be displayed in a composite display to the videoconference manager 1604 in response to user input. Can cause changes to the image processing behavior of the local dual camera portable device, such as compositing. Selecting the local video (s) may also cause a change in the operation of the camera (s) on the local device. For example, according to some embodiments, the camera of the unselected video stops transmitting unwanted video to the remote device, while in other embodiments, the camera stops capturing unwanted video. stop.

9. Selecting a Local View to Transmit The above subsection shows changes in the video display during a conference. According to some embodiments, a user of a dual camera mobile device may also select which camera to use for a video conference before the video conference starts. FIG. 75 illustrates a pre-conference selection of one video from two videos captured by a user's dual camera portable device for use in a video conference.

  FIG. 75 illustrates the selection of local video for use in video conferencing as the eight stages of operation of the UI 7500. First stage 7502 shows a UI 7500 of a dual camera portable device 7518 with an initial PIP display 7542 presented after the user has requested to start a video conference with a remote user of the portable device.

  As shown in the first step 7502, the initial PIP display 7542 includes two displays: a background main display 7520 and a foreground insert display 7522. The background main display 7520 occupies most of the display screen of the device, while the foreground insertion display 7522 is smaller than and overlaps the background main display. In this example, the background display 7520 presents a video of the person holding the guitar, which is assumed to be the video captured by the rear camera of the device. The foreground insert display 7522 presents a video of the person wearing the hat, which in this example is assumed to be the video being captured by the front camera of the device.

  This initial PIP display 7542 is just one way to present a composite view of the video being captured by the local device camera. Some embodiments may provide other composite views. For example, the background display may present video from the front camera of the device, and the smaller foreground insert display may present video from the rear camera of the device. Similarly, according to some embodiments, the two videos are displayed in two side-by-side display areas (eg, left and right display windows, or top and bottom display windows) or two diagonally aligned display areas in the UI 7500. sell. In some embodiments, the type of PIP display or default display mode may be specified by the user. Below the PIP display is a selectable UI item 7540 (eg, button 7540) labeled “End Conference” that allows the user to end the video conference by selecting an item.

  In a first step 7502, the user of the mobile device 7518 has requested a video conference with the remote user and is waiting for the remote user to respond. This waiting period is described by the notation "Preview, waiting for response ..." at the bottom of the display.

  The second step 7504 indicates the start of the video selection operation. In this example, operation is initiated by activating a set of selectable UI items to be displayed on PIP display 7542. The set of selectable UI items presents options for selecting local video to transmit to a remote device for a video conference. In some embodiments, the set of selectable UI items selects (eg, touches) any location on the UI display 7500 during the pre-conference time while waiting for the remote user to respond. Can be invoked by Instead of, or in conjunction with, such a wake-up operation, in some embodiments, the user may perform other operations, such as through different touch screen operations or using one or more other physical inputs of the device. Can activate a selectable UI item.

  Third step 7506 shows UI 7500 with an activated set of selectable UI items 7526 for the user to select a video. In this example, a set of selectable UI items 7526 in the form of a pop-up menu is displayed in PIP display area 7542 and overlaps the PIP display. In this example, the set of selectable UI items includes “Transmit L1” item 7528 (eg, button 7528), “Transmit L2” item 7530 (eg, button 7530), and “Transmit Both” item 7532 (eg, button 7532) and a “cancel” item 7534 (eg, button 7534). In this example, selecting the "Transmit L1" button 7528 causes the UI 7500 to transmit only the video captured by the device's rear camera to the remote device during the videoconference. By selecting the “Transmit L2” button 7530, the UI 7500 transmits only the video captured by the front camera of the device to the remote device during the video conference. By selecting the “Transmit Both” button 7532, the UI 7500 transmits both the video captured by the front and rear cameras of the device to the remote user for a video conference and selects the “Cancel” button 7534 This cancels the operation.

  The fourth step 7508 shows the UI 7500 after the user has selected the “Transmit L1” button 7528 (eg, by tapping the button 7528 with his finger 7524). This selection is indicated by highlighting button 7528 on PIP display area 7542. Some embodiments use different signage (e.g., the border of the selected item or the text highlighting of the selected item).

  Fifth step 7510 shows the animation of UI 7500 after the user has selected video from the rear camera of the device for transmission to the remote device. In this example, UI 7500 removes unwanted foreground insert display 7522 by sliding out of the right edge of PIP display 7542, indicated by arrow 7536. In a sixth step 7512, the inset display 7522 has been completely removed from the PIP display area 7542. Different embodiments use different animations to remove unwanted display areas, such as fade out or fade over the display area, move the display area in different directions, or simply remove the display area instantaneously.

  The seventh step 7514 shows the animation of the UI 7500 after the remote user has accepted the video conference request. Acceptance of the video conference request is highlighted by the removal of the "Preview, Waiting for Response ..." notation on the display. At this stage, as shown by arrow 7538, the size of the background display area 7520, which is video from the rear camera of the device, gradually decreases toward the lower left corner of the PIP display area 7542. The background display 7520 shrinks so that the UI 7500 can show behind the display area 7520 a display area 7544 containing video from the remote user's camera. Some embodiments shrink the local camera to different locations, use a tiled composite view of the two videos being displayed, or make the remote view an inset view area of the PIP view.

  The eighth step 7516 shows the UI 7500 after the video selection operation has been completed. The UI 7500 presents a new PIP display 7546 including an inset display 7520 of the video captured from the local device and a background display 7544 of the video transmitted from the remote device.

B. Bandwidth and Frame Rate In some embodiments, adjusting the size of the viewing area of the remote mobile device during a video conference causes the local mobile device to be driven by two cameras of the local mobile device (ie, a front camera and a rear camera). The bandwidth allocated to each captured video can be reallocated. FIG. 76 shows two examples of such bandwidth reallocation between two cameras of a local device.

  Each of the examples in FIG. 76 includes a rear camera sensor 7605 of the local device, a front camera sensor 7610 of the local device, a video conference module 7615 of the local device, and a UI 7635 of the remote portable device 7620. The rear camera sensor 7605 and the front camera sensor 7610 capture video from the rear camera and the front camera of the local device, respectively. The captured video is sent to video conferencing module 7615, which processes the video and transmits it to a remote device for display on UI 7635.

  In FIG. 76, the UI 7635 of the remote device presents the composite display. The composite display shows the video captured by the front and rear cameras of the local device. The video from the front camera captures a man wearing a tree and a hat, and the video from the rear camera captures a mountain landscape. As shown in FIG. 76, the two videos may be displayed on the UI 7635 in many different ways based on the configuration of the display area for displaying the video, and also based on the size of the display area. In each example, video conferencing module 7615 first allocates full output bandwidth to each of the videos according to the relative size of the display area at the remote device. Specifically, the video displayed in the larger display area of UI 7635 is assigned a larger portion of the total bandwidth, and the video displayed in the smaller display area of UI 7635 is assigned a smaller portion of the total bandwidth. Can be assigned. In some embodiments, if the videos are displayed in a display area of the same size, the total output bandwidth is evenly allocated to the two videos.

  The amount of bandwidth allocated to each of the two videos may affect the way each video is processed. For example, a video may require higher bandwidth than that allocated to the video. In such cases, the video frame rate is adjusted or the size of the video image is reduced to accommodate the lower bandwidth. Although reducing the frame rate of a video makes the video appear to be “broken,” reducing the size of the video image also reduces the area in which the video is displayed. Thus, if a video is allocated a certain amount of bandwidth, some embodiments adjust the video frame rate to allow the video to be transmitted within the allocated bandwidth. Reduce the size of the image or perform a combination of both. Those skilled in the art will appreciate that it is possible to allow video to be transmitted within the allotted bandwidth, even if the frame rate and average frame size adjustments are changed to obtain optimal overall video quality. Will be understood.

  Example (1) of FIG. 76 shows one scenario in which the bandwidth is reallocated in two operation stages of the UI 7635. The UI 7635 of the remote device 7620 in the first step 7670 presents a composite display including two displays. Of the two displays, one is at the top of UI 7635 and the other is at the bottom of UI 7635. In this example, the upper display area 7625 shows the video being captured by the front camera of the local device, and the lower display area 7630 shows the video being captured by the rear camera of the local device. As shown in a first step 7670, the upper display area 7625 is larger than the lower display area 7630. Thus, video from the front camera of the local device is allocated 80% of the bandwidth, and video from the rear camera of the local device is allocated 20% of the bandwidth. Adjust the video frame rate and / or scaling size to allow video from the local device's rear camera to be transmitted from the local device to the remote device within the allocated bandwidth.

  The second step 7675 shows the UI 7635 after the user of the remote device has increased the size of the lower display area so that the size of the upper display area 7625 and the size of the lower display area 7630 are approximately the same. As a result, each of the videos is reallocated by video conferencing module 7615 to 50% of the total bandwidth.

  Example (2) of FIG. 76 illustrates another scenario for performing bandwidth reallocation in two stages of operation of the UI 7635. In the first stage 7680 of example (2), the UI 7635 presents a PIP display. The PIP display includes two displays, a background main display area 7650 and a foreground insertion display area 7655. The background display area 7650 occupies most of the PIP display, but the foreground insertion display area 7655 is smaller than and overlaps the background main display area 7650. In this example, the background display area 7650 presents the video being captured by the front camera of the device. Insert display area 7655 presents the video being captured by the rear camera of the device. As shown at this stage, the background display area 7650 is larger than the insertion display area 7655. Thus, video from the front camera of the device is allocated 80% of the bandwidth and video from the rear camera of the device is allocated 20% of the bandwidth. Adjust the video frame rate and / or scaling size to allow video from the local device's rear camera to be transmitted from the local device to the remote device within the allocated bandwidth.

  The second step 7865 shows the UI 7635 after the user of the remote device has swapped the display of the two videos. Specifically, background display area 7660 presents the video that is currently being captured by the rear camera of the device, and insert display area 7665 is now being captured by the front camera of the device. Present the video. The video from the rear camera of the device is allocated 80% of the bandwidth and the video from the front camera of the device is allocated 20% of the bandwidth because the viewing area of the two videos has changed. Thus, the frame rate and / or scale size of the video from the front camera of the local device is reduced. Those skilled in the art will appreciate that the bandwidth distribution described in FIG. 76 is merely an example, and that other techniques for allocating bandwidth to two cameras during a video conference are possible.

1. Frame Rate ControlSimilar to the frame rate control operation during a conference described above, some embodiments reduce the rate at which images of video captured by each camera of a dual camera portable device are transmitted to other devices in a video conference. Coordination or maintenance may be required on an individual basis. Some of these embodiments provide similar techniques as described above. For example, some embodiments control the frame rate of each camera by adjusting the VBI of the sensor module 415 of each camera. Other embodiments also provide additional techniques such as, for example, frame dropping, which may be performed by the sensor module 415 and / or the general purpose transmission buffer 3120 of each camera.

2. Bandwidth Control Through Scaling As described above, during a videoconference between a dual camera portable device and another device, the amount of image data that can be transmitted over one or more network connections for a particular amount of time (ie, Network connection bandwidth) can be limited. In order to maximize and maintain the throughput of the network connection, different embodiments of the dual camera portable device provide different ways to control the amount of image data transmitted over the network connection in a particular amount of time. In some embodiments, the throughput is an average success rate of message delivery over a communication channel (eg, a network connection).

  When transmitting images captured by both cameras of a dual camera portable device, one such approach is to control the amount of image data transmitted over the network connection by using one or the other of the dual camera portable devices. Resize images from both cameras. While some embodiments reduce the size of images captured by a dual camera portable device to reduce the amount of image data transmitted over a network connection, other embodiments reduce the size of images captured over a network connection. The size of the image is increased to increase the amount of image data to be transmitted.

  Some embodiments maintain the height-to-width ratio of the image when scaling (ie, uniform scaling). Other embodiments scale the image such that the height-to-width ratio of the scaled image is different from the original image (ie, anamorphic scaling).

  Moreover, scaling can be performed at different stages of the image processing process. Scaling in some embodiments may be performed by a camera sensor. In such embodiments, the camera sensor may discard rows or columns (ie, pixel values) of the image data. In some of such embodiments, the remaining image data is interpolated to smooth the appearance of the image.

  Scaling in other embodiments is performed by the scaler module 455 of the CIPU 400. In some embodiments, scaling is performed by video conference manager 1604, as described above, and in other embodiments, the scaler is performed by an encoder. Thus, different embodiments of a dual camera portable device perform scaling differently.

3. Bit Rate Control Some embodiments provide different mechanisms for managing the bit rate at which video captured by the camera of a dual camera portable device is encoded. In some embodiments, the dual camera portable device includes a rate controller for each camera. Some embodiments provide a constant bit rate management scheme. In this scheme, each of the rate controllers is set to a fixed bit rate such that the overall bit rate of video from both cameras on the mobile device is constant. Other embodiments provide a priority scheme in which one of the two videos from the device's camera is always preferred over the other when it is necessary to reduce the overall bit rate.

  In some embodiments, the arbitrator module manages two rate controllers for two cameras. FIG. 77 shows an example of such an arbitrator module. As shown in FIG. 77, the rate controller 7700 sets the bit rate for the front camera, and the rate controller 7705 sets the bit rate for the rear camera. The rate controller transmits an image from the camera sensor to the encoder 7715. The arbitrator module 7710 is connected to both rate controllers and has the available bandwidth, each of the two videos, to ensure that both videos can be transmitted to the remote device under the available bandwidth. The bit rate setting for each of rate controllers 7700 and 7705 is controlled in a number of ways based on information such as the video size. Further, arbitrator 7710 may be configured to implement the fixed rate or priority scheme described above.

  In some other embodiments, the two rate controllers for the two cameras can communicate with each other. Under this scheme, rate controllers can exchange information for each video and set the video bit rate accordingly. Some examples of rate controller rate management mechanisms are provided. However, many other different mechanisms are possible.

4. Video Processing Some embodiments of a dual camera portable device process images captured by both cameras of the dual camera portable device differently in different situations. For example, when processing a PIP composite image that includes images captured by both cameras of a dual camera portable device, some embodiments selectively perform a TNR process 2000 on the PIP composite image. Some of these embodiments perform the TNR process 2000 only on the main image of the PIP composite image, while others of these embodiments perform only on the inserted image of the PIP composite image. To perform the TNR process 2000.

  As another example of the processing of images captured by both cameras of a mobile device, some embodiments adjust the display area displaying the video, such as by a user (e.g., stretching the insertion of a PIP display, Based on various changes to the video conference, such as defining a region of interest in the video being played, swapping the main / insert of the PIP display, changing the total available bandwidth, etc. The scaled image. Some of these embodiments scale the image in the manner described above. That is, the image can be scaled, for example, by the encoder 1655, the video conference manager 1604, the scaler module 455, and the camera sensor (ie, 405a or 405b) used to capture the image.

5. Encoding As described above, some embodiments transmit video from both cameras of a dual camera portable device. Thus, these embodiments may encode video captured by both cameras for transmission to a remote device during a video conference. Different embodiments provide different ways to encode video for transmission. FIG. 78 illustrates a technique for processing a video to be transmitted using a multiplexer (MUX) 7815, an encoder module 7825, a buffer 7830, and a combining module 7835.

  The MUX 7815 receives one input signal based on the selection signal, and outputs the selected input signal to the encoder 7825. For example, if the selection signal indicates to MUX 7815 that it will receive an input signal from C1, MUX 7815 selects that input signal and outputs that signal. This selection signal may be provided in a number of ways, such as a command from video conference manager 1604. Through the MUX 7815, the encoder 7825 alternately encodes the image received from the MUX 7815 into a bitstream format, and stores the encoded image in the buffer 7830. Combining module 7835 combines (ie, multiplexes) one or more bitstreams stored in buffer 7830 and outputs a single bitstream.

  The operation of this encoding technique will now be described with respect to three stages 7860, 7865 and 7870. In a first step 7860, the MUX 7815 is configured to receive the image 7805 captured by the camera C1 and output it to the encoder 7825 for encoding. Encoder 7825 encodes the received image to generate bitstream 7850, which is then stored in buffer 7830. The second stage 7865 is similar to the first stage 7860, except that the MUX 7815 is configured to receive the image 7810 captured by the camera C2 and output it to the encoder 7825 for encoding. Again, the encoder encodes the received image to generate a bitstream 7855, which is stored in buffer 7830. In a third step 7870, the combining module 7835 reads the bitstreams 7850 and 7855 from the buffer 7830 and combines them into one bitstream for transmission to a remote device.

  FIG. 79 shows another approach for encoding two videos from a dual camera portable device for transmission to a remote device during a video conference. In this approach, a video frame (ie, an image) from the first camera of the mobile device and another video frame from the second camera of the mobile device are combined into one video frame, and then the combined video frame is: Encoded into a bit stream to be transmitted to the remote device. As shown in FIG. 79, this method includes a synthesizer 7915, a buffer 7920, and an encoder 7925.

  As shown, the combiner 7915 combines the image 7905 from the first camera and the image 7910 from the second camera to form a combined image 7955. Different embodiments combine image 7905 and image 7910 differently. For example, the combiner 7915 of some embodiments can combine images by aligning two adjacent images with each other as shown in FIG. Composite images 8030 and 8035 show two exemplary composite images using this technique. In composite image 8030, image 7905 from the first camera is aligned with image 7910 from the second camera. On the other hand, a composite image 8035 shows an image 7905 aligned to the left of the image 7910.

  In some other embodiments, the compositor 7915 can composite the two images 7905 and 7910 by overlaying the two images 7905 and 7910 on a larger background image. The composite image 8040 in FIG. 80 shows an exemplary composite image using this technique. In composite image 8040, image 7905 and image 7910 are obliquely aligned and overlaid on the blank image (ie, image 7905 is at the upper left corner of the background image and image 7910 is at the lower right corner). In some embodiments, the camera sensors may be different sizes and thus capture images with different pixel resolutions. In such an embodiment, combiner 7915 can combine image 7905 and image 7910 in a manner similar to that illustrated by composite image 8045 in FIG. After combining the two images, the combiner 7915 stores the combined image in the buffer 7920. The encoder 7925 reads the combined image from the buffer 7920, encodes the combined image into a bitstream, and sends it to the remote device of the video conference.

  Here, the operation will be described with respect to the combiner 7915, the buffer 7920, and the encoder 7925 shown in FIG. First, the first camera sends the image 7905 to the combiner 7915 as part of the sequence of images in the video. At the same time, the second camera sends another image 7910 to the combiner 7915 as part of the sequence of images in the video. Thereafter, the combiner 7915 combines the image 7905 and the image 7910 to form a combined image 7955 by the above-described means. Next, the synthesizer 7915 transmits the synthesized image 7955 to the buffer 7920. Thereafter, the buffer 7920 stores the combined image before transmitting the combined image to the encoder 7925. Finally, encoder 7925 encodes the composite image into a bitstream and sends the bitstream to a remote device of the video conference.

  FIG. 81 illustrates yet another approach for encoding two videos from a dual camera portable device for transmission to a remote device during a video conference. In this approach, two videos from the device are displayed in a composite display, a screenshot of the composite display is taken, and encoded into a bitstream to be transmitted to the remote device. As shown in FIG. 81, this technique includes an encoder 8115. In some embodiments, encoder 8115 encodes the composite image and sends it to a remote device.

  Here, the operation will be described with respect to the encoder 8115 shown in FIG. First, video from two cameras of a dual camera portable device is displayed in a composite view of the device screen. The composite display can present the video in any manner. For example, the composite display in some embodiments can present two videos on a PIP display, such as the PIP display 8105 shown in FIG. In other embodiments, the composite display may present the two videos in two display areas side-by-side or two display areas aligned diagonally. A screenshot of the PIP display 8105, such as the image 8110, is taken and transmitted to the encoder 8115. The encoder then encodes the sequence of screenshots into a bitstream 8120, and then transmits the bitstream 8120 to a remote device of the video conference. While several different approaches for encoding two videos have been described above, other approaches are possible.

6. Decoding Some embodiments of a dual camera portable device may receive a bitstream encoded according to the techniques described above with respect to FIGS. In such an embodiment, the dual camera portable device may receive information (eg, via a video conferencing control channel) indicating the technique used to encode the video. FIG. 82 illustrates one technique for decoding a bitstream of two videos received from another device over a communication network for display on a dual camera portable device during a video conference. Specifically, this technique is used to decode a bitstream encoded by the encoding technique described above with respect to FIG.

  As shown in FIG. 82, this approach uses a separation module 8235, buffers 8230 and 8290, and a decoder module 8225. Demultiplex module 8235 decomposes (ie, demultiplexes) the bitstream into one or more bitstreams and stores the bitstreams in buffer 8230. Decoder 8225 reads the encoded bitstream, decodes it to generate video, and then stores the video in buffer 8290.

  Here, the operation of this method will be described with respect to the separation module 8235, the buffers 8230 and 8290, and the decoder module 8225 shown in FIG. Initially, a dual camera portable device receives a bitstream 7845 from another device over a communication network in a video conference (eg, at a networking manager 1614). Since the received bitstream is a multiplexed bitstream of two bitstreams, the separation module 8235 decomposes the received bitstream into two bitstreams 8255 and 8260. Each encoded bitstream represents video data captured from one of the two cameras of the device. Thereafter, the separation module 8235 stores the bit streams 8255 and 8260 in the buffer 8230.

  Thereafter, decoder 8225 reads bitstream 8250, one of the two bitstreams 8255 and 8260, from buffer 8230, and decoder 8225 decodes bitstream 8250, generates video 8280, and buffers video 8280. 8290. Decoder 8225 also decodes the other of bit streams 8255 and 8260, and stores the generated video in buffer 8290. Here, both videos can be read from buffer 8290 and stored, or displayed on a dual camera portable device.

  FIG. 83 illustrates a method of decoding a bitstream encoded by the method described with reference to FIG. 79. As shown in FIG. 83, this technique includes a decoder 8325, a buffer 8320, and a decomposer 8315.

  In some embodiments, decoder 8325 receives a bitstream encoded according to the technique shown in FIG. 79 and decodes the bitstream into one or more composite images, which are then stored in buffer 8320 Is done. Decomposer 8315 extracts two images from each composite image. In order to extract the two images from the composite image, the decomposer 8315 may include information indicating the location of each image within the composite image (eg, from a device in a video conference where the images were composited and coded, through a video conference communication control channel). Received information).

  Here, the operation of this method will be described with respect to the decoder 8325, the buffer 8320, and the decomposer 8315 shown in FIG. Initially, the decoder 8325 receives a video bitstream, such as the bitstream created by the technique described with respect to FIG. 79, from another mobile device in a video conference. The decoder 8325 decodes the bit stream into one or more composite images including the composite image 7955, and stores the composite image in the buffer 8320. Thereafter, buffer 8320 stores the combined image before transmitting the combined image to decomposer 8315. When the decomposer receives the composite image 7955 from the buffer 8320, it decomposes the composite image 7955 into two images 7905 and 7910. These images are the same as images 7905 and 7910 in FIG.

  When the bitstream is received from a system such as the system described in FIG. 81, a decoder such as decoder 8325 in FIG. 83 decodes the bitstream into a sequence of screenshots. The sequence of screenshots is displayed as a video on the screen of the device without further processing.

VI. Multiple Sources As described above, video may be captured by both cameras of a dual camera portable device in a video conference and transmitted to another device. Some embodiments, instead of transmitting video captured from both cameras of a dual camera mobile device, different media content displayed on the dual camera mobile device along with the video captured from the camera of the dual camera mobile device Alternatively, any content may be transmitted. In other words, these embodiments can transmit content from several sources along with the video captured by the camera of the dual camera portable device.

  FIG. 84 conceptually illustrates another software architecture for the video conferencing and processing module of the dual camera portable device of some embodiments. The video conferencing and processing module of FIG. 84 is the same as the video conferencing and processing module except that the video conferencing and Similar to the 16 videoconferencing and processing modules 1600.

  The media source module 8470 of some embodiments routes media content between the video conferencing module 8402 and the storage device 8475. Examples of media content include video, images, documents, and music. Other embodiments store other types of media content on the storage device 8475. In some embodiments, the storage device 8475 is an internal storage device (eg, RAM), while in other embodiments the storage device 8475 is an external storage device (eg, a compact flash (CF) card, a secure digital (SD) card. Etc.).

  In some embodiments, the screen capture module 8480 routes the image of the content displayed on the display of the dual camera portable device through the display driver 8485. In some embodiments, the display driver 8485 is responsible for capturing content on the display and converting the content to an image. Different embodiments capture different content displayed on the display. For example, some embodiments capture all content displayed on the display. Other embodiments capture a particular viewport of the display (eg, the viewport of the currently active window, the viewport of the PIP display, etc.).

  An example operation of some of the videoconferencing and processing modules will now be described with reference to FIG. To transmit media content along with video captured from the camera of a dual camera portable device, the video conferencing module 8402 of some embodiments does not read the images from the CIPU 1650, but instead uses the video conferencing manager 1604 to control the media source module 8470. The same operations as the video conference module 1602 described above with reference to FIG. To transmit an image of the content displayed on the display of the dual camera mobile device, some embodiments of the video conference manager 1604 retrieves the image of the content displayed on the display of the dual camera mobile device through the display driver 8485. . Some embodiments perform similar processing (e.g., perspective correction, resizing, etc.) on media content or images of content displayed on a display, as performed on images read from CIPU 1650. , But other embodiments do not perform any processing.

  The above description describes some of the examples of transmitting content from various sources along with video captured by the camera of a dual camera portable device. However, other embodiments can transmit other different types of content. For example, in a video conference involving multiple participants, some embodiments transmit video received from one device and video captured by a camera of a dual camera portable device to another device in the video conference. Thus, any number of different types of content from any number of sources can be transmitted along with the video captured by the camera of the dual camera portable device.

VII. Video Conferencing with Multiple Participants The section above relating to video conferencing describes a video conference with two participants. However, a video conference with multiple participants (ie, three or more participants) using the mobile device of some embodiments is also possible. In some embodiments, all participants of a multi-participant video conference can see and hear each other. In another embodiment, one participant (eg, a broadcaster) can see and hear all other participants, and all other participants can see and hear the broadcaster. Provide a multi-participant broadcast video conference that can be heard, but other participants cannot see and hear each other (unless authorized by the broadcaster, for example).

A. Multi-participant Video Conference User Interface During a multi-participant video conference, some embodiments display the video conference participants and select a particular participant (s) to see. Provides a variety of different UIs. For example, some embodiments of the mobile device provide a UI that simultaneously displays all participants in a video conference with multiple participants, and the user of the mobile device can display one of the participants (e.g., (By stretching the image of the selected participant). FIG. 85 shows an example of such a UI.

  This figure shows all participants of a video conference with multiple participants simultaneously on the UI 8530 of the mobile device 8500, and shows the sequence of operations for selecting one of the participants to see, in five different stages of the UI 8530. 8505, 8510, 8515, 8520 and 8525. The first stage 8505 shows the UI 8530 after a multi-participant video conference has been established between the other three users of the other device. As illustrated, the UI 8530 includes a composite display 8535 and a display area 1155. Synthetic display 8535 includes four display areas 8565, 8570, 8575, and 8580 that display images captured by the cameras of the participants of the video conference by a plurality of participants. In this example, display area 8565 shows a user of mobile device 8500 (ie, display area 8565 displays images captured by the front camera of mobile device 8500). The display area 1155 is the same as the above-described display area 1155 in FIG.

  The second step 8510 shows that the user of the portable device 8500 starts the participant selection operation by selecting one of the display areas of the composite display area 8530. Specifically, a second step 8510 indicates that the user selects display area 8570 (eg, by tapping display area 8570 with finger 8550).

  The third stage 8515 of the UI 8530 shows the composite display 8555 after the participant selection operation is completed. Some embodiments provide an animation (not shown) to indicate the transition between the second stage 8510 and the third stage 8515. The composite display 8555 includes, as a background display area, a PIP display 8560 (that is, a display area 8570) indicating the display area of the participant selected in the second step 8510, and a user display area 8565 as an insertion display area of the PIP display 8560. Included as In this example, PIP display 8560 shows an image of selected display area 8570 that has been expanded horizontally to fit horizontally. In some embodiments, the image is not stretched and the image in the selected viewport maintains its portrait orientation (ie, the extra white space on each side of the background viewport, as shown in FIG. Filled with). In addition, the composite display 8555 also includes a composite display 8585 showing reduced images of the two unselected display areas 8575 and 8580.

  The fourth step 8520 shows that by selecting the PIP display 8560 (eg, by tapping the PIP display 8560 with the finger 8550), the user of the portable device 8500 has begun a participant deselect operation. The fifth step 8525 shows the composite display 8535 after the completion of the participant deselection operation.

  FIG. 85 illustrates an exemplary sequence of operations for simultaneously displaying all participants of a video conference with multiple participants, performing a participant selection operation, and performing a participant deselection operation. Other sequences of operation are possible. For example, after the third step 8515, instead of initiating a participant deselection operation, the user may replace the newly selected display area in the display area 8585 with the background display area of the PIP display 8560 (i. One of the unselected display areas displayed on the composite display 8585 can be selected in order to replace the selected display area. Therefore, the user can replace the display area in the display area 8585 with the background display area of the PIP display 8560 any number of times at any time during a video conference with a plurality of participants. Also, at any time during a video conference with multiple participants, the user may perform a participant deselect operation to return to composite display 8535. Further, different embodiments allow a user to select a particular participant in different ways, such as by flipping a switch on the mobile device 8500, giving a voice command, and the like.

  Some embodiments provide techniques for automatically selecting participants based on, for example, audio detection. In such an embodiment, when one of the participants speaks, the display area of that participant is automatically selected as the background display area of PIP display 8560. When a different participant speaks, that participant's display area is automatically selected as the background display area of PIP display 8560. In some embodiments, if none of the participants in the multi-participant video conference are speaking, the display will display the composite display 8535 after a defined amount of silence (eg, 3 seconds). In some embodiments, nothing happens on the UI 8530 of the mobile device 8500 when the user of the mobile device 8500 speaks.

  FIG. 86 illustrates another exemplary sequence of operations for simultaneously displaying all participants of a video conference with multiple participants and selecting one of the participants to watch. FIG. 86 illustrates this operation on the UI 8645 of the portable device 8500 as seven different stages 8505, 8605, 8610, 8615, 8620, 8625 and 8630 of the UI 8645. The first step 8505 is the same as the first step 8505 shown in FIG. 85, which shows the UI 8645 after a video conference with multiple participants between the other three users of the other device has been established.

  A second step 8605 indicates that the user of the mobile device 8500 has begun a participant selection operation by selecting the display area 8570 (eg, by placing two fingers on the display area 8570). The third step 8610 shows a transition step of the participant selection operation. At this stage, the user drags the two fingers away from each other while increasing the display area 8570 and filling the display area of what was used to create the composite display 8535. This example shows that the display area 8570 is selected, but any of the other display areas 8565, 8575, and 8580 can be selected. In some embodiments, the user of the mobile device 8500 can be prevented from selecting the user's viewport (ie, viewport 8565 in this example).

  The fourth step 8615 of the UI 8645 shows the PIP display 8635 after the participant selection operation has been completed. Some embodiments require the user to continue dragging the fingers away from each other until the display area 8570 fills the background display area 8640 of the PIP display 8635, while other embodiments require that the user continue to drag the fingers away from each other. Before the user releases the finger, it is only necessary that the user perform a drag operation that is greater than a certain threshold amount (eg, longer than a certain distance or longer than a certain amount of time). Is done. If the user's drag action meets or exceeds a certain threshold amount, UI 8645 continues to expand display area 8570 until it fills background display area 8640 of PIP display 8635. Otherwise, the participant selection operation is not completed and the UI 8645 returns to the composite display 8535. As shown, the selected display area (ie, display area 8570) is the background display area 8640 of the PIP display 8635, and the user's display area 8565 is the insertion display area of the PIP display 8635. Some embodiments provide an animation (not shown) to indicate the transition between the third stage 8610 and the fourth stage 8615.

  A fifth step 8620 initiates the participant deselect operation by the user of the portable device 8500 selecting the display area 8640 of the PIP display 8635 (eg, by placing two fingers on the background display area 8640). Show where you are. The sixth step 8625 indicates a transition step of the participant deselection operation. This stage shows the user dragging the fingers toward each other to shrink the display area, which is used to be the background display area 8640 of the PIP display 8635. Similar to the operation described in the third step 8610, some embodiments require that the user's drag operation be greater than a threshold amount before the user releases the finger (eg, longer than a certain distance, Or longer than a certain amount of time). Otherwise, the participant deselection operation is not completed and the UI 8645 returns to the PIP display 8635. The seventh step 8630 of the UI 8645 shows the composite display 8535 after the completion of the participant deselection operation.

  FIG. 86 illustrates another exemplary sequence of operations for simultaneously displaying all participants in a video conference with multiple participants, performing a participant selection operation, and performing a participant deselection operation. However, according to some embodiments, a user of the portable device 8500 may repeatedly perform a participant selection operation and a participant deselection operation. FIG. 87 illustrates one such embodiment.

  Specifically, FIG. 87 illustrates an exemplary sequence of performing a participant selection operation and a participant deselection operation multiple times in the UI 8730, through seven different stages 8505, 8705, 8615, 8710, 8715, 8720, and 795 of the UI 8730. 8725. The first step 8505 is the same as the first step 8505 in FIGS. 85 and 86 described above. The second step 8705 of FIG. 86 differs from the second step 8705 of FIG. 86 in that the user selects the display area 8570 by tapping the display area 8570 once (instead of placing two fingers on the display area 8570). Is the same as The third step 8615 is the same as the fourth step 8615 in FIG. 86 because it shows the PIP display 8635 after the completion of the participant selection operation. The fourth step 8710 is similar to that except that the user selects the background display area 8640 of the PIP display 8645 by tapping the background display area 8640 once (instead of placing two fingers on the background display area 8640). This is similar to the fifth step 8620 in FIG. 86.

  The fifth step 8715 is the same as the seventh step 8630 in FIG. 86 because the composite display 8535 is shown after the participant deselection operation is completed. The sixth step 8720 is similar to the second step 8510, except that a participant selection operation is performed on the display area 8575. Similarly, the seventh step 8725 is similar to the third step 8705 because the selected display area (ie, display area 8575) is shown as the background display area 8640 of the PIP display 8635. Although FIG. 87 illustrates only a small number of participant selection operations and participant deselection operations, any number of such operations may be performed during a video conference with multiple participants.

  Additionally, some embodiments provide a UI that can display a different number of participants during a video conference. For example, the UI of some embodiments displays only some of the participants in a multi-participant video conference when the mobile device is held in an upright position (ie, portrait orientation), and the mobile device is in landscape orientation. Display additional participants when held in position (ie, sideways). Other embodiments display all participants when the mobile device is held in a landscape position. In addition, some embodiments provide animations that show transitions between different positions and / or orientations of the mobile device, similar to those shown in FIGS. 34, 35, 36, and 37. Other different animations are possible.

  As another example of a UI that displays a different number of participants during a video conference, some embodiments allow a user of a mobile device to select multiple participants to view simultaneously during a video conference. it can. Referring to the first stage 8505 of FIG. 85 for illustration, according to some of these embodiments, a user of the portable device 8500 may display two or more of the display areas 8565, 8570, 8575, and 8580 (eg, (By tapping the corresponding display area of the composite display 8535). The selected display area may be displayed in various manners, such as any of the composite display, the PIP display, and the display configuration shown in FIG. 65, among other types of display configurations of a plurality of participants. Moreover, while examples of some embodiments have been described, one of ordinary skill in the art will appreciate that different embodiments may select and display multiple participants of a video conference with multiple participants in many different ways. Like.

B. User Interface for Broadcast Video Conferencing with Multiple Participants As mentioned earlier, broadcast video conferencing with multiple participants involves one participant seeing and listening to all other participants. Only the other participants are not able to hear or see each other. To facilitate multi-participant broadcast video conferencing, some embodiments provide a number of different UIs for displaying broadcasters and other participants of a multi-participant broadcast video conference. I do. For example, some embodiments provide a student-teacher-like UI layout that is similar to the layout of the third stage 8515 shown in FIG. Accordingly, the student and teacher UI layouts of some embodiments will be described with respect to this stage thereafter.

  In these embodiments, only the broadcaster is displayed over the entire display area of PIP display 8560 (ie, the insertion display area is not displayed). Other participants of the broadcast-type video conference with a plurality of participants are displayed below a PIP display 8560 similar to the display area displayed on the composite display 8585. In some embodiments, a prescribed number of other participants are displayed in the composite display 8585 when the mobile device is in portrait mode, but additional or all participants are identified by the mobile device as described above. Similarly, it can be displayed on the composite display 8585 in the case of the horizontal mode. Further, other embodiments provide different UIs for displaying broadcasters and other participants of a multi-participant broadcast video conference.

C. Controlling Audio for Multiple Participant Video Conferencing In addition, the mobile devices of some embodiments provide different techniques for controlling the audio of multiple participant video conference participants. For example, with some embodiments of the mobile device, the user of the mobile device can view a video conference with multiple participants through a single set of volume controls (eg, a volume slider) displayed on the UI of such an embodiment. Can control the audio of each participant. In another embodiment, the mobile device allows the user of the mobile device to use each participant's audio in a video conference with a plurality of participants by a set of volume controls, such as a volume slider, displayed in a table display area of each participant. Volume can be controlled individually. Some embodiments provide only a mute button instead of a set of volume controls. Thus, in some such embodiments, the user of the portable device can only mute or unmute all participants in a video conference with multiple participants, while in other such embodiments, The user of the mobile device can individually mute or unmute each participant in a video conference with multiple participants. Further, other techniques for controlling the audio of a participant in a video conference with multiple participants are possible, such as by switching a switch on the portable device, giving a voice command, and the like.

VIII. Electronic Systems Many of the functions and uses described above are implemented as software processes specified as sets of instructions recorded on computer readable storage media (also referred to as computer readable media). When the instructions are executed by one or more processing units (eg, one or more processors, processor cores, or other processing units), the processing unit (s) perform the operations indicated in the instructions. Let it. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, and the like. Computer readable media does not include carrier waves and electronic signals through wireless or wired connections.

  As used herein, the term "software" is meant to include firmware resident in read-only memory or applications stored on magnetic storage and can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while leaving separate software inventions. In some embodiments, multiple software inventions may be implemented as separate programs. Finally, any combination of separate programs that together implement the software invention described herein is included within the scope of the invention. In some embodiments, a software program defines one or more specific machine implementations that perform and perform the operations of the software program when installed to run on one or more electronic systems.

  Some embodiments are implemented as software processes that include one or more application programming interfaces (APIs) in an environment in which the calling program code interacts with other program code being invoked through one or more interfaces. . Various function calls, messages or other types of invocation, which may further include various types of parameters, may be transferred via an API between the calling program and the code being called. Further, the API may provide calling program code with the ability to use data types or classes defined in the API and implemented in the called program code.

  At least certain embodiments include an environment in which the calling software component interacts with the called software component through an API. Methods for operating through an API in this environment include one or more function calls, messages, other types of invocations, or transfer of parameters through the API.

  In some embodiments, one or more application programming interfaces (APIs) can be used. For example, some embodiments of the media exchange module 310 (or 910) may access various video processing and encoding functions described in FIGS. 3 and 9, such as the functions of the TNR module 1900 described in FIG. Next, the API set is provided to other software components.

  An API is an interface implemented by a program code component or a hardware component (hereinafter referred to as an “API implementation component”), and different program code components or hardware components (hereinafter referred to as an “API caller component”). Can be used to use one or more functions, methods, procedures, data structures, classes, and / or other services provided by the API implementation component. An API can define one or more parameters passed between the API caller component and the API implementation component.

  The API allows the developer of the API calling component (which may be a third-party developer) to take advantage of the specific functionality provided by the API implementation component. There may be one API caller component, or there may be multiple such components. An API can be a source code interface provided by a computer system or program library to support a service request from an application. An operating system (OS) can have multiple APIs to allow applications running on the OS to call one or more of these APIs, and services (such as program libraries) An application using the service may have multiple APIs to call one or more of those APIs. An API can be specified in terms of a programming language that can be interpreted and executed or compiled when building the application.

  In some embodiments, the API implementation component can provide multiple APIs, each providing a different view of or a different aspect of accessing the different aspects of the functionality implemented by the API implementation component. For example, one API with an API-implementing component can provide a first set of functions and can be exposed to third-party developers, and another API with an API-implementing component is hidden (not exposed) and functions And a further set of functions, such as testing or debugging functions not included in the first set of functions. In other embodiments, the API-implementing component itself can call one or more other components via its underlying API, and thus can be both an API-calling component and an API-implementing component.

  The API defines the language and parameters used by the API caller component when accessing and using a particular function of the API implementation component. For example, an API caller component accesses a particular function of an API-implemented component through one or more API calls or invocations (eg, implemented by functions or methods) exposed by the API, and calls or invokes the API. Pass data and control information using parameters via The API implementation component may return a value through the API in response to an API call from the API calling component. The API defines the syntax and results of the API call (eg, how to invoke the API call, and what the API call does), but the API calls the function specified by the API call. It is not necessary to clarify how is realized. Various API calls are transferred via one or more application programming interfaces between the caller (API caller component) and the API implementation component. Transferring an API call may include issuing, starting, invoking, calling, receiving, returning, or responding to a function call or message. In other words, the transfer can describe the operation by either the API caller component or the API implementation component. A function call or other invocation of the API can send or receive one or more parameters through a parameter list or other structure. Parameters are constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods, or other data or other items that are passed through an API. It can be a way of.

  Moreover, data types or classes can be provided by an API and implemented by an API implementation component. Thus, the API calling component declares variables, uses pointers to such types or classes, uses constant values of such types or classes, or uses the definitions provided in the API, or Can be instantiated.

  In general, an API may be used to access services or data provided by an API implementation component, or to initiate execution of an operation or calculation provided by the API implementation component. By way of example, each of the API implementation component and the API caller component may be any one of an operating system, library, device driver, API, application program, or other module (API implementation component and API caller, respectively). It should be understood that the components may be the same type of module or different types of modules). In some cases, the API-implemented components may be implemented, at least in part, in firmware, microcode, or other hardware logic. In some embodiments, the API allows client programs to use services provided by a software development kit (SDK) library. In other embodiments, applications or other client programs can use the API provided by the application framework. In these embodiments, the application or client program may incorporate calls to functions and methods provided by the SDK and API, or may use data types or objects defined in the SDK and provided by the API. The application framework, in these embodiments, can provide a main event loop to a program that responds to various events defined by the framework. The API allows an application to define events and responses to events using the application framework. In some implementations, the API call includes capabilities or states related to aspects such as input capabilities and states, output capabilities and states, processing capabilities, power states, storage device capacities and states, communication capabilities, and the like. The capabilities or state of the hardware device can be reported to the application, and the API can be implemented, in part, by firmware, microcode, or other low-level logic that executes, in part, on hardware components.

  The API calling component can be a local component (ie, on the same data processing system as the API implementing component) or a remote component (ie, the API implementing component) that communicates with the API implementing component through the API over a network. And on a different data processing system). The API-implementing component may function as an API-calling component (ie, make API calls to APIs exposed by different API-implementing components), and the API-calling component may be a different API-calling component. , May function as an API implementation component by implementing the APIs exposed in.

  The API allows multiple API calling components written in different programming languages to communicate with the API implementing component (therefore, the API is used to translate calls and return values between the API implementing component and the API calling component). Functions can be included). However, the API may be implemented in terms of a particular programming language. The API caller component, in one embodiment, includes an API set from the OS provider and another API set from the plug-in provider and another API set from another provider or creator of another API set (eg, a software library). API) from different providers such as

  FIG. 88 is a block diagram illustrating an exemplary API architecture that may be used in some embodiments of the present invention. As shown in FIG. 88, the API architecture 8800 includes an API implementation component 8810 (eg, an operating system, library, device driver, API, application program, software, or other module) that implements the API 8820. The API 8820 specifies one or more functions, methods, classes, objects, protocols, data structures, formats and / or other mechanisms of an API implementation component that may be used by the API caller component 8830. The API 8820 can specify at least one calling convention that defines how functions of the API implementation component 8810 receive parameters from the API calling component 8830, and how the function returns results to the API calling component. . An API caller component 8830 (eg, an operating system, library, device driver, API, application program, software or other module) may access the API implementation component 8810 defined by the API 8820 to access and use the functions of the API 8820. API call via The API implementation component 8810 can return a value to the API caller component 8830 through the API 8820 in response to the API call.

  It will be appreciated that API implementation component 8810 may include additional functions, methods, classes, data structures, and / or other mechanisms that are not defined through API 8820 and are not available to API caller component 8830. It is understood that the API caller component 8830 may be on the same system as the API implementation component 8810, or may be remote and access the API implementation component 8810 using the API 8820 over a network. I want to. FIG. 88 shows a single API caller component 8830 interacting with API 8820, but other API caller components that may be written in a different language (or the same language) than API caller 8830 use API 8820. It should be understood that this may be done.

  The API implementation component 8810, the API 8820, and the API caller component 8830 can be stored on a machine-readable medium, which can be any machine for storing information in a form readable by a machine (eg, a computer or other data processing system). Including the mechanism. For example, a machine-readable medium includes a magnetic disk, an optical disk, a random access memory, a read-only memory, a flash memory device, and the like.

  FIG. 89 is an example of a dual camera mobile computing device architecture 8900. An implementation of a mobile computing device can include one or more processing units 8905, a memory interface 8910, and a peripheral device interface 8915. Each of these components that make up the computing device architecture can be a separate component or can be integrated into one or more integrated circuits. These various components may also be coupled together by one or more communication buses or signal lines.

  Peripheral interface 8915 may be coupled to various sensors and subsystems, including camera subsystem 8920, wireless communication subsystem (s) 8925, audio subsystem 8930, I / O subsystem 8935, and the like. Peripheral device interface 8915 allows communication between the processor and the peripheral device. Peripherals such as orientation sensor 8945 or acceleration sensor 8950 can be coupled to peripheral interface 8915 to facilitate orientation and acceleration functions.

  The camera subsystem 8920 may be coupled to one or more light sensors 8940, for example, a charge coupled device (CCD) light sensor, a complementary metal oxide semiconductor (CMOS) light sensor. A camera subsystem 8920 coupled to the sensor can facilitate camera functions such as image and / or video data capture. Wireless communication subsystem 8925 may serve to facilitate communication functions. Wireless communication subsystem 8925 may include a radio frequency receiver, a radio frequency transmitter, an optical receiver, and an optical transmitter. The wireless communication subsystem may be implemented to operate on one or more communication networks, such as a GSM network, a Wi-Fi network, a Bluetooth network, and the like. An audio subsystem 8930 is coupled to the speakers and microphone to facilitate audio-enabled functions such as audio recognition, digital recording, and the like.

  The I / O subsystem 8935 includes transfers between input / output peripheral devices, such as displays, touch screens, etc., and the CPU's data bus through the peripheral interface. The I / O subsystem 8935 includes a touch screen controller 8955 and another input controller 8960 to facilitate these functions. A touch screen controller 8955 is coupled to the touch screen 8965 and can detect contact and movement on the screen using any of a plurality of touch sensitivity techniques. Other input controllers 8960 may be coupled to other input / control devices such as one or more buttons.

  The memory interface 8910 can be coupled to a memory 8970, which can include non-volatile memory such as high speed random access memory and / or flash memory. The memory can store an operating system (OS) 8972. OS 8972 may include instructions for handling basic system services and for performing hardware-dependent tasks.

  The memory stores communication instructions 8974 to facilitate communication with one or more additional devices, graphical user interface instructions 8976 to facilitate graphic user interface processing, and image / video related processing and functions. Image / video processing instructions 8978 for facilitation, telephone instructions 8980 for facilitating telephone related processes and functions, and media exchange and facilitation for facilitating processes and functions associated with media communication and processing. Processing instructions 8982, camera instructions 8984 to facilitate camera-related processes and functions, and video conferencing instructions 8986 to facilitate video conferencing processes and functions can also be included. The instructions identified above need not be implemented as a separate software program or module. Various functions of the mobile computing device may be implemented in hardware and / or software including one or more signal processing and / or application specific integrated circuits.

  The embodiments described above can include a touch I / O device 9001 that can receive touch input to interact with a computing system 9003 via a wired or wireless communication channel 9002, as shown in FIG. Touch I / O device 9001 may be used to provide user input to computing system 9003 instead of or in combination with other input devices such as a keyboard, mouse, and the like. One or more touch I / O devices 9001 can be used to provide user input to the computing system 9003. The touch I / O device 9001 may be an integral part of the computing system 9003 (eg, a touch screen on a laptop) or may be separate from the computing system 9003.

  Touch I / O device 9001 can include a touch sensor panel that is wholly or partially transparent, translucent, non-transparent, opaque, or any combination thereof. The touch I / O device 9001 may be a touch screen, a touch pad, a touch screen functioning as a touch pad (eg, a touch screen replacing a laptop touch pad), a touch screen or touch coupled or combined with any other input device. It may be implemented as a pad (eg, a touch screen or touch pad disposed on a keyboard) or any multi-dimensional object having a touch-sensitive surface for receiving touch input.

  In one example, touch I / O device 9001 implemented as a touch screen can include a transparent and / or translucent touch sensor panel partially or wholly disposed over at least a portion of a display. . According to this embodiment, touch I / O device 9001 is operative to display graphical data sent from computing system 9003 (and / or another source) and to receive user input. I do. In other embodiments, the touch I / O device 9001 may be implemented as an integrated touch screen with a touch sensor component / device integrated with a display component / device. In still other embodiments, the touch screen may be used as an auxiliary or additional display screen to display auxiliary or the same graphical data as the main display and receive touch input.

  Touch I / O device 9001 is capable of measuring capacitance, resistance, optical, acoustic, inductive, mechanical, chemical, or any phenomenon that can be measured with respect to the occurrence of one or more touches or near touches near device 9001. May be configured to detect a location of one or more touches or near touches on the device 9001. Software, hardware, firmware, or any combination thereof, may be used to process a measurement of a detected touch in order to identify and track one or more gestures. The gesture may correspond to a steady or non-stationary, single or multiple, touch or near touch on the touch I / O device 9001. Gestures are touch I / O devices, such as tapping, pushing, fixing, rubbing, twisting, reorienting, changing pressure and pressing, at essentially the same time, adjacent or consecutively It can be performed by moving one or more fingers or other objects in a particular manner on 9001. The gesture may be characterized by pinching, sliding, swiping, rotating, bending, dragging, or tapping with any other finger or fingers or any other finger, It is not limited to these. A single gesture may be performed by one or more users with one or more hands, or by any combination of these.

  The computing system 9003 may display a graphical user interface (GUI) on a display using the graphical data. The GUI may be configured to receive touch input via touch I / O device 9001. Since the touch I / O device 9001 is implemented as a touch screen, it can display a GUI. Alternatively, the GUI may be displayed on a separate display from the touch I / O device 9001. The GUI may include graphical elements that are displayed at specific locations within the interface. The graphical elements may include various virtual input devices to be displayed, including, but not limited to, a virtual scroll wheel, a virtual keyboard, a virtual knob, a virtual button, any virtual UI, and the like. The user can perform a gesture at one or more specific locations on the touch I / O device 9001 that can be associated with a graphical element of the GUI. In other embodiments, the user can perform a gesture at one or more locations independent of the location of the GUI graphical element. Gestures performed on the touch I / O device 9001 directly and indirectly manipulate and control graphical elements such as cursors, icons, media files, lists, text, the front or parts of images in the GUI. Can change, move, move, activate, start, or generally affect. For example, in the case of a touch screen, a user can interact directly with the graphical element by performing a gesture through the graphical element on the touch screen. Alternatively, touchpads generally provide indirect interaction. The gesture may also affect a GUI element that is not being displayed (eg, causing a user interface to be displayed) or may affect other operations within the computing system 9003 (eg, a GUI). Affecting the state or mode of the application or operating system). The gesture may or may not be performed on the touch I / O device 9001 together with the displayed cursor. For example, when a gesture is performed on a touchpad, a cursor (or pointer) may be displayed on a display screen or a touchscreen, and the cursor may be displayed on the touchpad to interact with graphical objects on the display screen. May be controlled via a touch input of the user. In other embodiments where the gesture is performed directly on the touchscreen, the user can interact directly with the object on the touchscreen regardless of whether a cursor or pointer is displayed on the touchscreen.

  Feedback may be provided to the user via the communication channel 9002 in response to or based on a touch or near touch on the touch I / O device 9001. Feedback may be sent in a variable or non-variable manner, optically, mechanically, electrically, olfactory, acoustically, etc., or any combination thereof.

  These functions described above can be implemented in digital electronic circuits, in computer software, firmware or hardware. The techniques may be implemented using one or more computer programs. The programmable processor and the computer may be included in or packaged as a mobile device. Processes and logic flows may be performed by one or more programmable processors and by one or more programmable logic circuits. General purpose computing and storage devices and special purpose computing and storage devices may be interconnected through a communications network.

  Some embodiments include electronic components, such as microprocessors, storage devices, and memories that store computer program instructions on a machine-readable or computer-readable medium (also referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). including. Some examples of such computer-readable media are RAM, ROM, read-only compact disc (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), read-only digital Application discs (eg, DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (eg, DVD-RAM, DVD-RW, DVD + RW, etc.), flash memory (eg, SD card, miniSD card, microSD) Cards, etc.), magnetic and / or solid-state hard drives, read-only Blu-Ray® disks and recordable Blu-Ray disks, UDO (ultra density optical) disks, any other optical media or magnetic Including the body and a floppy disk (registered trademark). The computer readable medium can store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Examples of computer programs or code include machine code, such as generated by a compiler, and files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

  Although the above description primarily refers to a microprocessor or multi-core processor executing software, some embodiments may be implemented in one or more applications, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Performed by one or more integrated circuits. In some embodiments, such an integrated circuit executes instructions stored on the circuit itself.

  The terms "computer," "server," "processor," and "memory," as used herein and in any claims of this application, refer to electronic devices or other technical devices. These terms exclude a person or group of people. For clarity, the terms "display" or "display" mean to display on an electronic device. The term "computer-readable medium" as used in this specification and in any claims of this application is fully limited to tangible physical objects that store information in a form readable by a computer. These terms exclude any wireless signals, wired download signals and any other ephemeral signals.

  FIG. 91 conceptually illustrates an exemplary communication system 9100 used to connect some participants in a video conference according to some embodiments. As shown, communication system 9100 includes a number of portable devices 9115, a number of cellular base stations (or Node Bs) 9110, a number of radio network controllers (RNCs) 9105, and a core network 9125. . The cellular base station and RNC are collectively referred to as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access Network (UTRAN) 9130. Each RNC 9105 is connected to one or more cellular base stations 9110 and is collectively referred to as a radio access network (RAN).

  Each cellular base station 9110 covers a service area 9120. As shown, the mobile devices 9115 in each service area are wirelessly connected to the serving cellular base station 9110 in the service area 9120 via a Uu interface. The Uu interface uses a protocol stack with two planes, a control plane and a user plane. The user plane supports circuit switched data streams, packet switched data streams and broadcast data streams. The control plane carries network signaling messages.

  Each cellular base station is connected to the RNC via an Iub interface. Each RNC 9105 is connected to the core network 9125 by an Iu-cs interface and an Iu-ps interface. The Iu-cs interface is used for circuit-switched services (eg, voice), and the Iu-ps interface is used for packet-switched services (eg, data). The Iur interface is used to connect two RNCs to each other.

  Thus, communication system 9100 supports both circuit-switched services and packet-switched services. For example, a call can be made by transmitting call data (for example, voice) through a circuit switch of the communication system 9100 by a circuit switching service. Conducting a video conference by transmitting video conference data through a packet switch of the communication system 9100 using a packet exchange service using a transport protocol layer such as UDP or TCP via an Internet layer protocol such as IP. Can be. In some embodiments, the transfer of a call to a video conference (eg, handoff) as described above in the video conference setup section uses circuit-switched and packet-switched services supported by a communication system such as communication system 9100. . That is, in such an embodiment, the call is conducted through a circuit switch of the communication system 9100 and the video conference is conducted through a packet switch of the communication system 9100.

  Although the exemplary communication system of FIG. 91 illustrates a UTRAN wireless mobile communication system that is a third generation (3G) technology, in some embodiments, in order to connect some of the participants in the conference, Other 3G communication systems such as second generation (2G) communication systems, 3GPP2 Evolution Data Optimized / Evolution Data Only (EV-DO) and Third Generation Partnership Project 2 (3GPP2) code division multiple access IX (CDMA IX) It should be noted that 4th generation (4G) communication systems, wireless local area networks (WLANs) and World Wide Interoperability for Microwave Access (WiMAX) communication systems can be used. Examples of 2G systems include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), and Enhanced Data Rate for GSM Evolution (EDGE). The 2G communication system architecture is shown in FIG. 91, except that the 2G communication system architecture uses a radio base station device (BTS) instead of the Node B 9110 and uses a base station controller (BSC) instead of the RNC 9105. Similar to architecture. In a 2G communication system, the A interface between the BSC and the core network is used for circuit switching services, and the Gb interface between the BSC and the core network is used for packet switching services.

  In some embodiments, the communication system 9100 is operated by a service carrier, which first provisions the mobile device 9115 so that the mobile device 9115 can use the communication system 9100. Some embodiments provision the mobile device 9115 by configuring and registering a Subscriber Identity Module (SIM) card with the mobile device 9115. In other embodiments, the mobile device 9115 is instead configured and registered using the memory of the mobile device 9115. In addition, additional services can be provisioned (after the customer purchases the mobile device 9115), such as GPRS, multimedia messaging services (MMS), and data services such as instant messaging. When provisioned, the mobile device 9115 is activated, thereby enabling the use of the communication system 9100 by the service carrier.

  Communication system 9100 is a private communication network in some embodiments. In such embodiments, the mobile devices 9115 can communicate (eg, make voice calls, exchange data) between each other (eg, mobile devices 9115 provisioned in the communication system 9100). In another embodiment, communication system 9100 is a public communication network. Accordingly, the mobile device 9115 can communicate with other devices external to the communication system 9100 in addition to the mobile device 9115 provisioned in the communication system 9100. Some of the other devices external to communication system 9100 include telephones, computers and other devices connected to communication system 9100 through other networks, such as the public switched telephone network or another wireless communication network.

  The Long Term Evolution (LTE) specification is used to define a 4G communication system. FIG. 92 conceptually illustrates an example of a 4G communication system 9200 used to connect several participants in a video conference according to some embodiments. As shown, the communication system 9200 includes a number of mobile devices 9115, a number of evolved Node Bs (eNBs) 9205, a mobility management entity (MME) 9215, a serving gateway (S-GW) 9220, It includes a packet data network (PDN) gateway 9225 and a home subscriber server (HSS) 9235. In some embodiments, communication system 9200 includes one or more MMEs 9215, one or more S-GWs 9220, one or more PDN gateways 9225, and one or more HSSs 9235.

  The eNB 9205 provides an air interface to the mobile device 9115. As shown, each NB 9205 covers a service area 9210. The mobile device 9115 in each service area 9210 is wirelessly connected to the eNB 9205 in the service area 9210 through an LTE-Uu interface. FIG. 92 also shows eNBs 9205 connected together via an X2 interface. Further, the eNB 9205 is connected to the MME 9215 through the S1-MME interface, and is connected to the S-GW 9220 through the S1-U interface. The eNB 9205 is generically referred to as an evolved UTRAN (E-TRAN) 9230.

  The eNB 9205 performs functions such as radio resource management (for example, radio bearer control, connection mobility control, and the like), path control of user plane data to the S-GW 9220, signal measurement and measurement report, and selection of an MME when a mobile device is attached. provide. The functions of the MME 9215 include: tracking and paging of mobile devices in idle mode, activation and deactivation of radio bearers, selection of S-GW 9220 when attaching mobile devices, termination of non-access stratum (NAS) signaling, and user interaction with HSS 9235. Including authentication.

  The functions of the S-GW 9220 include (1) routing and forwarding of user data packets, and (2) management and storage of mobile device contexts such as IP bearer service parameters and network internal routing information. The function of the PDN gateway 9225 is to provide a connection from the mobile device to an external packet data network (not shown) by being an egress and ingress of traffic destined for the mobile device. A mobile station may have simultaneous connections with multiple PDN gateways to access multiple packet data networks. The PDN gateway 9225 also functions as a mobility anchor between WiMAX and 3GPP technologies such as 3GPP2 (for example, CDMA IX and EV-DO) and technologies other than 3GPP.

  As shown, the MME 9215 is connected to the S-GW 9220 through an S11 interface and to the HSS 9235 through an S6a interface. The S-GW 9220 and the PDN gateway 9220 are connected through an S8 interface. The MME 9215, the S-GW 9220, and the PDN gateway 9225 are collectively referred to as an evolved packet core (EPC). EPC is a key component of the System Architecture Evolution (SAE) architecture, which is the core network architecture of the 3GPP LTE wireless communication standard. EPC is a pure packet system. For example, EPC does not have an audio media gateway. Services such as voice and SMS are provided by application functions that are packet switched and routed and utilize EPC services. Thus, using the transition from a call to a video conference as an example, in some embodiments, both the call and the video conference take place through the packet switch of communication system 9200. In some such embodiments, the packet-switched channel used for the call continues to be used for audio data of the video conference after the call ends. However, in other such embodiments, a different packet-switched channel is created (e.g., during the establishment of a videoconference), and the audio data is sent to the newly created packet rather than the packet-switched channel at the end of the call. It is transmitted over the switching channel.

  Further, the amount of bandwidth provided by these different technologies ranges from 44 kilobits per second (kbps) for GPRS to over 10 megabits per second (Mbps) for LTE. For LTE, a download speed of 100 Mbps and an upload speed of 50 Mbps are expected in the future.

  Although the present invention has been described with reference to many specific details, those skilled in the art will recognize that the present invention may be embodied in other specific forms without departing from the scope of the invention. . Further, some figures conceptually illustrate the process. Certain operations of these processes need not be performed in the exact order shown and described. Certain operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Moreover, the process may be implemented using several sub-processes or as part of a larger macro process.

  Also, many embodiments have been described above as a video conference between two dual camera portable devices. However, many of these embodiments may not be used when involving a video conference between a dual camera mobile device and another device, such as a single camera mobile device, a computer, a telephone with video conferencing capabilities. As will be appreciated by those skilled in the art. Furthermore, many of the embodiments described above can be used in single camera portable devices and other computing devices with video conferencing capabilities. Thus, it will be apparent to one skilled in the art that the present invention is not limited by the above details, but is defined by the appended claims.

Claims (145)

A computer-readable storage medium of a first mobile device, comprising:
The storage medium stores a computer program,
The computer program comprises:
A set of instructions for presenting a selectable user interface (UI) item on the first mobile device for switching from the audio call to a video conference during an audio call over a wireless communication network with a second device;
An instruction set for receiving a selection of the selectable UI item;
An instruction set for initiating the video conference without ending the audio call;
A set of instructions for ending the audio call before enabling the first device and the second device to present exchanged audio and video data through the video conference. Storage medium.   The instruction set for presenting the selectable UI items includes an instruction set for continuously presenting the selectable UI items in a display area that can be displayed on a display screen of the first mobile device during the audio call. The storage medium according to claim 1, comprising: The computer program comprises:
Displaying the first video captured by the first mobile device in a first display area while waiting for the second video captured by the second device to reach the first mobile device. Instruction set,
An instruction set for reducing the size of the first display area when the second video is received at the first mobile device, for indicating the second video in a second display area of the first mobile device. The storage medium according to claim 1, further comprising:   4. The storage medium according to claim 3, wherein the first display area overlaps the second display area when the size of the first display area is reduced.   When the size of the first display area is reduced, the first display area and the second display area form a picture-in-picture display such that the first display area functions as an inserted photograph overlapping the second display area. The storage medium according to claim 3, wherein:   The storage medium of claim 1, wherein the audio call ends before the first device and the second device can exchange audio and video data through the video conference.   After the first device and the second device start exchanging audio data and video data through the video conference, the first device and the second device present the exchanged audio data and video data. 2. The storage medium according to claim 1, wherein the audio call is terminated before the user starts calling.   The computer program further comprises a set of instructions for establishing a communication channel from the first portable device to the second device to establish the audio call between the first device and the second device. The storage medium according to claim 1, wherein:   The storage medium of claim 1, wherein the first mobile device is a mobile communication device for communicating with another device in the wireless communication network.   The storage medium according to claim 1, wherein the first mobile device is a smartphone.   The storage medium of claim 1, wherein the first device and the second device are provisioned in the wireless communication network to make an audio call with another device through the wireless communication network.   The first mobile device is provisioned in the wireless communication network to make an audio call with another device through the wireless communication network, while the second device is used to make an audio call with another device through the wireless communication network. The storage medium according to claim 1, wherein the storage medium is not provisioned.   The wireless communication network is a first wireless communication network, and the second device performs an audio call with another device through a second wireless communication network different from the first wireless communication network or through another communication network. 13. The storage medium according to claim 12, wherein the storage medium is provisioned.   The storage medium according to claim 1, wherein the first device and the second device are smartphones.   The storage medium according to claim 1, wherein the communication network is a private wireless communication network.   The storage medium according to claim 1, wherein the communication network is a public wireless communication network.   The wireless communication network includes a circuit switch for routing audio calls and a packet switch for routing data, and the audio calls are routed through the circuit switches of the wireless communication network during the video conference. The audio data and the video data being exchanged are exchanged as packets through the packet switch, and the packets are transported between the first device and the second device over an Internet Protocol (IP) communication link. The storage medium according to claim 1, wherein: A computer-readable storage medium of a first mobile device, comprising:
The storage medium stores a computer program for setting up a video conference during a call between the first mobile device and the second device;
The computer program comprises:
During a call between the first device and the second device, a first UI layout having a selectable user interface (UI) item for switching from the call to the video conference is presented on the first mobile device. Instruction set,
A set of instructions for presenting a second UI layout having a first video captured by the first device in response to receiving the selection of the selectable UI item;
In response to receiving acceptance of the video conferencing request from the second device, a second video captured by the second device is displayed to indicate a transition from the second UI layout to a third UI layout. An instruction set for reducing the size of the first video while superimposing the first video on
The storage medium according to claim 3, wherein the third UI layout indicates a picture-in-picture (PIP) layout in which the first video overlaps at least a part of the second video.   The storage medium of claim 18, wherein the first device and the second device are portable devices provisioned to make a call over at least one public wireless communication network of one wireless service carrier. .   19. The PIP display according to claim 18, wherein the PIP display has a main display area indicating the second video and an insertion display area indicating the first video, and the insertion display area is smaller than the main display area. A storage medium according to claim 1.   The computer program reduces the size of the first display area for displaying the first video so as to show a second display area for displaying the second video after the first display area. 19. The method of claim 18, further comprising: providing a set of instructions for providing an animated initial presentation of the second video in response to receiving acceptance of the video conference request from a device. Storage medium.   The third UI layout has a composite display, the composite display has a first display area showing the first video and a second display area showing the second video, and the first display and the second display. 19. The storage medium according to claim 18, wherein the storage media have the same size and are located side by side.   19. The storage medium according to claim 18, wherein the first UI layout, the second UI layout, and the third UI layout are within a same display area of a UI of the first mobile device. A computer readable storage medium of a first mobile device, wherein the storage medium stores a computer program, wherein the computer program comprises:
Instructions for presenting at the first device a composite display having a first video captured by the first device and a second video captured by a second device participating in a video conference with the first device. Set and
A set of instructions for receiving input at the first device to change the composite display during the video conference;
An instruction set for changing the composite display based on the received input. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video, wherein the insertion display is smaller than the main display, and At least partially overlap,
The instruction set for receiving the input includes: (i) an instruction set for receiving a selection of the insertion display; and (ii) an instruction set for receiving an input indicating a moving direction of the insertion display.
The storage medium according to claim 24, wherein the instruction set for changing includes an instruction set for moving the insertion display in the moving direction. The first mobile device includes a touch sensor display screen,
The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video, wherein the insertion display is smaller than the main display, and At least partially overlap,
The instruction set for receiving the input includes: (i) an instruction set for associating the position of the insertion display with a touched position of the touch sensor display screen; and (ii) the insertion display in accordance with the association. An instruction set for selecting, and (iii) an instruction set for determining that the position of the contact with the touch sensor display screen has moved while the contact is maintained,
The storage medium according to claim 24, wherein the instruction set for changing the composite display includes an instruction set for moving the insertion display in the moving direction of the contact.   The instruction set for moving the insertion display includes, while the insertion does not reach a snap position in the composite display designated as a position in the composite display for locating one end of the insertion display, 27. The storage medium of claim 26, including a set of instructions for moving the inserted display even after contact with the screen has been completed.   The storage medium according to claim 27, wherein the snap position is a corner of the PIP display. An instruction set for moving the insertion display includes:
An instruction set for moving the insertion display even after the contact with the screen is completed, if the movement of the contact position exceeds a specific amount;
27. The storage medium according to claim 26, further comprising: an instruction set for not moving the insertion display after the contact is completed if the movement of the contact position does not exceed the specific amount. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video, wherein the insertion display is smaller than the main display, and At least partially overlap,
The instruction set for receiving the input includes: (i) an instruction set for receiving a selection of the insertion indication; and (ii) an instruction set for receiving an input indicating movement of the insertion indication.
The storage medium according to claim 24, wherein the instruction set for changing includes an instruction for moving the insertion area.   The storage medium according to claim 24, wherein the insertion display moves from a first corner of the PIP display to a second corner of the PIP display. The input is generated when a display screen of the first device is rotated,
The storage medium according to claim 24, wherein the instruction set for changing includes an instruction set for rotating the composite display.   The storage medium according to claim 32, wherein the screen of the first device is rotatable independently of a main body of the first device.   33. The storage medium according to claim 32, wherein when the first device rotates, the screen of the first device rotates.   33. The instruction set for rotating the composite display includes an instruction set for presenting an animation showing rotation of the composite display about an axis passing through the center of the composite display. Storage media.   The storage medium according to claim 35, wherein the animation includes reducing the composite display so that the entire composite display is displayed during rotation of the composite display.   The storage medium according to claim 36, wherein at the end of the animation, the composite display is expanded so as to match the orientation of the screen.   33. The storage medium according to claim 32, wherein the screen of the first device rotates in a first direction, and the composite display rotates in a second direction opposite to the first direction. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video;
The insertion display is smaller than the main display, overlaps at least a part of the main display,
33. The method of claim 32, wherein if the main display and the insert display are in a portrait orientation prior to the rotation, the instruction set includes an instruction to change the orientation of the insert display to a landscape orientation. Storage medium. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video;
The insertion display is smaller than the main display, overlaps at least a part of the main display,
The computer program further includes an instruction set for receiving a notification of the rotation of the second device and an instruction set for changing the orientation of the main display from a first orientation to a second orientation. The storage medium according to claim 32. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video;
The insertion display is smaller than the main display, overlaps at least a part of the main display,
The input is generated when the first device is rotated;
The storage medium according to claim 24, wherein the instruction set for changing includes an instruction set for rotating the PIP display. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video;
The insertion display is smaller than the main display, overlaps at least a part of the main display,
The instruction set for receiving the input includes an instruction set for receiving selection of a part of the insertion display and movement of the part,
The storage medium according to claim 24, wherein the instruction set for changing includes an instruction set for changing a size of the insertion display.   43. The storage medium of claim 42, wherein moving the portion includes moving the portion to the center of the insertion display, and wherein the changing includes reducing a size of the insertion display.   43. The storage medium according to claim 42, wherein the movement of the part includes moving the part in a direction away from the center of the insertion display, and the change includes increasing the size of the insertion display.   43. The storage medium according to claim 42, wherein the selected portion is a side of the insertion display.   43. The storage medium according to claim 42, wherein the selected portion is a corner of the insertion display. The composite display is a picture-in-picture (PIP) display having an insertion display indicating the first video and a main display indicating the second video;
The insertion display is smaller than the main display, overlaps at least a part of the main display,
The instruction set for changing includes an instruction set for exchanging the insertion display and the main display so that the insertion display indicates the second video and the main display indicates the first video. The storage medium according to claim 24, wherein:   The storage medium according to claim 24, wherein the instruction set for changing includes an instruction set for exchanging a position of the first video and a position of the second video in the composite display. The instruction set for receiving the input includes an instruction set for selecting the first video,
The storage medium of claim 24, wherein the instruction set for changing comprises an instruction set for moving the first video with respect to the composite display. A computer-readable storage medium storing a computer program for execution by a processor, wherein the computer program is for adjusting operation of a first mobile device in a video conference with a second mobile device, wherein the computer program Is
A set of instructions for receiving video captured by a camera of the second mobile device;
A set of instructions for displaying a picture-in-picture display of the video captured by the second device and the video captured by the first device on the first device;
An instruction set for detecting a change in screen orientation of the first mobile device;
A command set for rotating the PIP display on the screen of the first device based on the change in the orientation of the screen of the first device. A computer-readable storage medium storing a computer program for execution by a processor, wherein the computer program is for adjusting operation of a first mobile device in a video conference with a second mobile device, wherein the computer program Is
A set of instructions for receiving video captured by the second mobile device, the video being displayed on a screen of the first mobile device;
A set of instructions for generating a display of the video captured by the second mobile device, along with another display of the video captured by the first mobile device;
A set of instructions for receiving user input instructing to move a display of one of the videos toward a screen orientation of the first mobile device;
Instructions for moving the one display of the video to another location on the screen of the first mobile device based on the user instruction. A method of processing images for a first camera and a second camera of a mobile device using a shared pipeline, the method comprising:
Receiving a first set of images captured by the first camera of the mobile device;
Processing the first set of images using a first configuration of the shared pipeline;
Receiving a second set of images captured by the second camera of the mobile device;
Processing the second set of images using a second configuration of the shared pipeline that is different from the first configuration.   The method of claim 52, wherein the first configuration of the shared pipeline is stored in a first set of registers included in the shared pipeline.   The method of claim 53, wherein the second configuration of the shared pipeline is stored in a second set of registers included in the shared pipeline.   The method of claim 54, wherein the first register set and the second register set are different.   54. The method of claim 53, further comprising reading a value stored in the first set of registers prior to processing the first set of images.   55. The method of claim 54, further comprising reading a value stored in the second set of registers before processing the second set of images. A method of processing images for a first camera and a second camera of a mobile device using a shared pipeline, the method comprising:
Processing a first image from the first camera using the shared pipeline during a vertical blanking interval of the second camera;
Processing the second image from the second camera using the shared pipeline during a vertical blanking interval of the first camera.   59. The method of claim 58, wherein the vertical blanking period of the first camera is equal to the vertical blanking period of the second camera. A method for controlling a frame rate of a camera of a mobile device, comprising:
Capturing a first set of images at a first frame rate using a first vertical blanking interval of the camera of the mobile device;
Adjusting the first vertical blanking interval of the camera to a second vertical blanking interval for capturing an image at a second frame rate;
Capturing a second set of images at the second frame rate using the second vertical blanking interval of the camera. Transmitting the first set of images to the second device at the first frame rate during a video conference;
Transmitting the second set of images to the second device at the second frame rate during the videoconference. A computer readable storage medium for a portable device having at least a first camera and a second camera, the storage medium storing a computer program for encoding a video picture captured by the camera, wherein the computer program comprises:
A set of instructions for identifying a first rate controller parameter based at least on a first value quantifying one characteristic of the first camera;
A set of instructions for using the first rate controller parameters to configure a rate controller set to adjust a bit rate for encoding a video picture captured by the first camera;
A set of instructions for identifying a second rate controller parameter based at least on a second value quantifying the same characteristic of the second camera, wherein the first value and the second value are different. An instruction set, wherein the first rate controller parameter and the second rate controller parameter are different;
Instructions for using the second rate controller parameters to configure the rate controller set to adjust a bit rate for encoding a video picture captured by the second camera. Characteristic storage medium.   63. The storage medium of claim 62, wherein the rate controller set includes one rate controller.   An instruction set for using the first rate controller parameter to configure the rate controller set includes an instruction set for configuring a first rate controller, and the second set of instructions for configuring the rate controller set. 63. The storage medium of claim 62, wherein the instruction set for using rate controller parameters includes an instruction set for configuring a second rate controller.   63. The storage medium of claim 62, wherein the instruction set for identifying the first rate controller parameter includes an instruction set for calculating the first rate controller parameter.   63. The storage medium according to claim 62, wherein the characteristics of the first camera and the second camera are camera noises representing an amount of picture artifact generated in the cameras.   63. The storage medium according to claim 62, wherein the characteristic of the first camera and the second camera is a resolution of the camera. The first value and the second value are visual masking intensities that quantify the visibility of coding artifacts in encoding a video picture captured by the camera;
63. The storage medium according to claim 62, wherein the first rate controller parameter and the second rate controller parameter are a first quantization parameter and a second quantization parameter. The first camera and the second camera are for capturing video while the mobile device is used for a video conference;
63. The storage medium of claim 62, wherein the computer program further comprises instructions for switching from the first camera to the second camera during the video conference. The computer program comprises:
An instruction set for storing a set of state information while encoding a video picture from the first camera;
The state information for configuring the rate controller set to adjust a bit rate for encoding the video picture captured by the second camera after switching from the first camera to the second camera. 70. The storage medium of claim 69, further comprising: a set of instructions for using the set.   71. The storage medium of claim 70, wherein the set of state information includes a target bit rate for encoding a captured video picture.   71. The storage medium of claim 70, wherein the state information set includes at least one of a buffer occupancy, a maximum buffer occupancy, and data relating to the size of a recently encoded video picture. . A mobile device,
A first camera and a second camera for capturing a video picture;
(I) a first configuration based on a first rate controller parameter identified based on a first value quantifying a characteristic of the first camera; and (ii) a second configuration quantifying the same characteristic of the second camera. A rate controller set having a second configuration based on a second rate controller parameter identified based on the value, wherein the first value and the second value are different, and the difference causes the first rate controller parameter to be different. And the second rate controller parameter is different, wherein the rate controller set generates a quantization parameter based on the rate controller parameter, a rate controller set,
A video encoder for encoding a video picture using the quantization parameters received from the set of rate controllers.   74. The mobile device of claim 73, wherein the rate controller set includes one rate controller, and the first configuration and the second configuration are different configurations of the rate controller.   The rate controller set includes a first rate controller and a second rate controller, (i) the first configuration is a configuration of the first rate controller, and (ii) the second configuration is a configuration of the second rate controller. The mobile device of claim 73, wherein: The first value and the second value are visual masking intensities that quantify the visibility of coding artifacts in encoding a video picture captured by the camera;
The mobile device of claim 73, wherein the first rate controller parameter and the second rate controller parameter are a first quantization parameter and a second quantization parameter. The first camera and the second camera are for capturing video while the mobile device is used for a video conference;
The mobile device comprises:
A module for enabling switching from one camera to another during the video conference;
A storage device for storing a set of state information while encoding a video picture from any of the cameras,
Configuring the rate controller set to adjust a bit rate for encoding the video picture captured by the other camera after a switch from one camera to the other camera is performed. The mobile device of claim 73, wherein a set of state information is used.   74. The mobile device of claim 73, wherein the video encoder comprises a quantizer that quantizes encoding of the video picture based on the quantization parameter.   74. The mobile device of claim 73, wherein the characteristics of the first camera and the second camera are camera noises representing an amount of picture artifacts that have occurred in the camera.   The mobile device of claim 73, wherein the characteristic of the first camera and the second camera is a camera resolution. A method for encoding video captured by a first camera and a second camera of a mobile device, comprising:
Defining a first process for generating a first rate controller parameter based at least on a first value quantifying one characteristic of the first camera;
Defining a second process for generating a second rate controller parameter based at least on a second value quantifying the same characteristic of the second camera, wherein the first value and the second value are Different processes and
A bit rate for encoding a video picture captured by either the first camera or the second camera based on the generated first rate controller parameter or the generated second rate controller parameter. Defining a set of rate controllers for coordination.   The method of claim 81, wherein the rate controller set includes one rate controller, and wherein the first configuration and the second configuration are different configurations of the rate controller.   The rate controller set includes a first rate controller and a second rate controller, (i) the first configuration is a configuration of the first rate controller, and (ii) the second configuration is a configuration of the second rate controller. The method of claim 81, wherein A computer readable storage medium of a first mobile device including a first camera and a second camera, wherein the storage medium stores a computer program for performing a video conference between the first mobile device and a second device. , The computer program comprises:
An instruction set for selecting the first camera to capture an image;
An instruction set for transmitting an image captured by the first camera to the second device;
A set of instructions for receiving a selection of the second camera to capture an image during the video conference;
A command set for terminating transmission of the image captured by the first camera during the video conference and transmitting the image captured by the second camera of the first mobile device to the second device; A storage medium characterized by having.   The computer program displays an image captured by each camera of the first mobile device on a display of the first mobile device whenever transmitting the image captured by the camera to the second device. The storage medium of claim 84, further comprising an instruction set for performing   The computer program further comprises a set of instructions for animating a transition from the video image captured by the first camera to the video image captured by the second camera. 90. The storage medium according to 85. The computer program comprises:
A set of instructions for receiving an image captured by the camera of the second device during the video conference;
85. The storage medium of claim 84, further comprising: a set of instructions for displaying the received image captured by the camera of the second device on a display of the first mobile device. The computer program comprises:
A set of instructions for receiving an image captured by the camera of the second device during the video conference;
The received image captured by the camera of the second device may be combined with the image captured by the camera of the first mobile device capturing an image being transmitted during the videoconference. The storage medium of claim 84, further comprising: a set of instructions for displaying on a display of the mobile device.   The computer program includes a selectable user interface item for instructing the first mobile device to switch cameras during the video conference while simultaneously displaying the image captured and transmitted by the first camera. The storage medium of claim 88, further comprising a set of instructions for displaying simultaneously.   The instruction set for selecting the second camera further comprises an instruction set for receiving an input for instructing the first mobile device to select the second camera through a user interface of the first mobile device. 85. The storage medium according to claim 84, comprising:   85. The instruction set for selecting the second camera further comprises an instruction set for receiving a request from the second device to switch from the first camera to the second camera. The storage medium according to the above.   The first device and the second device exchange control messages during the video conference, and the request from the second device is one of the control messages exchanged during the video conference. 92. The storage medium according to claim 91, wherein A computer readable storage medium of a first mobile device including a first camera and a second camera, wherein the storage medium stores a computer program for performing a video conference between the first mobile device and a second device. And the computer program comprises:
(1) a display area for displaying images captured by the camera of the first mobile device and (2) transmitted to the second device during a real-time video conference session;
A selectable user interface (UI) item for switching the camera of the first mobile device during the video conference, wherein the first mobile device is configured to switch the first mobile device when the camera switching occurs. A graphical user interface (GUI) having a selectable user interface item for terminating transmission of an image captured by one of the cameras of the device and transmitting an image captured by the other of the devices to the second device. A storage medium characterized by having.   The storage medium of claim 93, wherein the display area is also for displaying an image transmitted by the second device during the video conference.   The storage medium according to claim 93, wherein the display area and the selectable UI item are simultaneously displayed during the video conference.   The storage medium of claim 93, wherein the computer program comprises a network manager for receiving a request from the second device to switch cameras of the first mobile device during a video conference.   The first device and the second device exchange control messages during the video conference, and the request from the second device is one of the control messages exchanged during the video conference. 97. The storage medium according to claim 96. A method for defining a computer program for a first mobile device to conduct a video conference with a second device, wherein the first mobile device comprises a first camera and a second camera, the method comprising:
(1) defining a display area for displaying images captured by a camera of the first mobile device and transmitted to the second device during a real-time video conference session;
Defining a selectable user interface (UI) item for switching the camera of the first mobile device during the video conference, wherein when the camera switching occurs, the first mobile device is configured to: Terminating the transmission of the image captured by one of the cameras of the first mobile device and transmitting the image captured by the other of the camera of the device to the second device.   The method of claim 98, wherein the display area is also for displaying an image transmitted by the second device during the video conference.   The method of claim 98, wherein the display area and the selectable UI item are displayed simultaneously during the video conference.   100. The method of claim 98, further comprising defining a network manager to receive a request from the second device to switch cameras of the first mobile device during a video conference.   The first device and the second device exchange control messages during the video conference, and the request from the second device is one of the control messages exchanged during the video conference. 102. The method of claim 101, wherein A computer readable storage medium of a first mobile device, the storage medium storing a computer program for enabling the first device to remotely control the second mobile device during a video conference with a second mobile device. Storing the computer program,
A set of instructions for transmitting an image captured by the camera of the first device to the second device;
A set of instructions for receiving an image captured by the camera of the second device;
Transmitting a command to the second device over a communication channel of a real-time communication session to the second device to instruct the second device to perform an operation of modifying the image captured by the camera of the second device. And a set of instructions.   The storage medium according to claim 103, wherein the communication channel is a control channel.   The storage medium according to claim 103, wherein the operation is an exposure operation.   The storage medium according to claim 103, wherein the operation is a focus operation.   The storage medium according to claim 103, wherein the operation is a zoom operation.   The storage medium of claim 103, wherein the computer program further comprises a set of instructions for displaying the image captured by the camera of the second device.   The storage medium of claim 103, wherein the command is received through a graphical user interface of the first device.   110. The storage medium according to claim 109, wherein the command is transmitted by touching a position on the display screen of the first device where the image captured by the camera of the second device is displayed. . A computer-readable storage medium for a first mobile device, wherein the storage medium stores a computer program for enabling the first device to remotely control the second device during a video conference with a second mobile device. Wherein the computer program has a graphical user interface (GUI),
A first display area for displaying an image captured by a camera of the first device during a real-time communication session;
A second display area for displaying an image captured by the camera of the second device during the real-time communication session;
A user interface for sending a command to the second device during the real-time communication session to instruct the second device to perform an operation of changing the image captured by the camera of the second device; (UI) item.   The storage medium according to claim 111, wherein the operation is an exposure operation.   The storage medium according to claim 111, wherein the operation is a focus operation.   The storage medium according to claim 111, wherein the operation is a zoom operation.   The storage medium of claim 111, wherein the command is transmitted over a communication channel of the real-time communication session.   The storage medium of claim 115, wherein the communication channel is a control channel for the real-time communication session.   The storage medium of claim 111, wherein the first display area is an insertion within the second display area. A method for defining a computer program for enabling said first mobile device to remotely control a second mobile device during a video conference, comprising:
Defining a display for displaying video captured from the second mobile device on the first mobile device;
Camera adjustment for receiving at the first mobile device an input to instruct the second mobile device to perform an operation of changing the image captured by the camera of the second device during the video conference. Defining an operation.   The method of claim 1, further comprising: defining a first network manager for the first mobile device, relaying the received input to a second network manager of the second mobile device during the video conference. 118. The method according to 118.   The first network manager relays data to the second network manager during the videoconference via a packet transmitted in Internet Protocol (IP), wherein the packet is used to control the operation of the second mobile device. 120. The method of claim 119, comprising a control packet for relaying input data received at a first mobile device.   119. The method of claim 118, further comprising defining a user interface for receiving the input by detecting that the location of the display has been touched during the video conference.   The method of claim 118, wherein the operation is an exposure operation. A computer readable storage medium of a first mobile device, wherein the storage medium stores a computer program for managing a video conference between the first mobile device and a second device, wherein the computer program comprises:
A set of instructions for identifying a first allowable bit rate for transmitting video conference data to the second device over a communication channel;
An instruction set for identifying a current bit rate that is less than the first allowable bit rate;
An instruction set for receiving network data for the communication channel from the second device;
A set of instructions for determining from the received network data that the communication channel maintains the current bit rate increase;
An instruction set for increasing the current bit rate;
An instruction set for repeatedly performing the receiving operation, the determining operation, and the increasing operation until it is determined that the communication channel does not maintain the current bit rate increase. Storage medium.   The storage medium of claim 123, wherein the second device is a mobile device that communicates with the first mobile device through a wireless communication network having a limited bandwidth.   An instruction set for repeatedly performing the receiving operation, the determining operation, and the increasing operation is that accurate network data is relayed from the second device based on the previous gradual increase of the current bit rate. 128. The storage medium of claim 123, wherein the storage medium is executed repeatedly at a specific time rate that allows. The computer program comprises:
Even after the repetitive execution of the receiving operation, the determining operation, and the increasing operation is completed, receiving network data related to the communication channel from the second device, and using the received network data to enable use of the communication channel. A set of instructions to identify changes in various resources;
The storage medium of claim 123, further comprising: a set of instructions for increasing or decreasing the current bit rate based on the identified change in the available resources of the communication channel.   The storage medium according to claim 123, wherein the instruction set for determining includes an instruction set for determining that the current bit rate has reached the first allowable bit rate. The received data includes available bandwidth measurement data provided by the second device;
The set of instructions for determining is a second allowed bit rate lower than the first allowed bit rate, the second allowed bit rate being derived from the available bandwidth measurement data provided by the second device. 124. The storage medium of claim 123, comprising an instruction set for determining that the current bit rate has reached a bit rate.   The storage medium of claim 123, wherein the received data includes a one-way signal delay calculated by the second device.   130. The storage of claim 129, wherein the one-way signal delay is calculated by the second device by examining a delay of an audio packet transmitted from the first mobile device to the second device. Medium.   131. The audio packet delay of claim 130, wherein the audio packet delay is checked by checking a time code difference in the audio packet for a difference in receiving the audio packet by the second device. Storage medium.   The received data is used by the second device by examining a size of a video packet sent to the second device against a difference in a time at which the second device received the packet. 128. The storage medium of claim 123, comprising possible bandwidth measurement data.   The storage medium of claim 123, wherein the received network data includes packet loss data.   The storage medium of claim 123, wherein the received network data includes round trip delay time data.   The instruction set for receiving the network data includes an instruction set for transmitting network data transmitted in a real-time transport protocol (RTP) stream and identified in the RTP stream by an RTP extension header. The storage medium according to claim 123, wherein A method performed by a first mobile device to manage a video conference between a first mobile device and a second device, the method comprising:
Identifying a first allowable bit rate for transmitting video conference data to the second device over a communication channel;
Identifying a current bit rate that is less than the first allowable bit rate;
Receiving network data for the communication channel from the second device;
Determining from the received network data that the communication channel maintains the current bit rate increase;
Increasing the current bit rate;
Repeatedly performing the receiving operation, the determining operation, and the increasing operation until it is determined that the communication channel does not maintain the current bit rate increase.   139. The method of claim 136, wherein the second device is a mobile device communicating with the first mobile device over a wireless communication network having a limited bandwidth.   The step of repeatedly performing the receiving operation, the determining operation, and the increasing operation enables accurate network data to be relayed from the second device based on the previous gradual increase of the current bit rate. 139. The method of claim 136, wherein the method is performed repeatedly at a specific time rate. Even after the repetitive execution of the receiving operation, the determining operation, and the increasing operation is completed, receiving network data related to the communication channel from the second device, and using the received network data to enable use of the communication channel. Identifying changes in critical resources;
136. The method of claim 136, further comprising: increasing or decreasing the current bit rate based on the identified change in the available resources of the communication channel.   138. The method of claim 136, wherein the determining comprises determining that the current bit rate has reached the first allowed bit rate. The received data includes available bandwidth measurement data provided by the second device;
The step of determining may include a second allowable bit rate lower than the first allowable bit rate, the second allowable bit rate being derived from the available bandwidth measurement data provided by the second device. 137. The method of claim 136, comprising determining that the current bit rate has been reached.   137. The method of claim 136, wherein the received data includes a one-way signal delay calculated by the second device.   143. The method of claim 142, wherein the one-way signal delay is calculated by the second device by examining a delay of an audio packet transmitted from the first portable device to the second device. .   The received data is used by the second device by examining a size of a video packet sent to the second device against a difference in a time at which the second device received the packet. 137. The method of claim 136, including including possible bandwidth measurement data.   138. The method of claim 136, wherein receiving the network data comprises receiving network data transmitted in a real-time transport protocol (RTP) stream and identified in the RTP stream by an RTP extension header. the method of.