Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
  • H04N5/445—Receiver circuitry for the reception of television signals according to analogue transmission standards for displaying additional information
  • H04N5/45—Picture in picture, e.g. displaying simultaneously another television channel in a region of the screen
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/14—Display of multiple viewports
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/4223—Cameras
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
  • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
  • H04N21/4316—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
  • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
  • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
  • H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
  • H04N5/445—Receiver circuitry for the reception of television signals according to analogue transmission standards for displaying additional information
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2352/00—Parallel handling of streams of display data

Definitions

• This disclosure relates generally to display systems and methods.
• features for capturing, displaying and/or recording two or more pictures in a picture (PIPs) system with tracking capability on a display are disclosed.
• Some of the desired features in display devices include displaying a picture in a picture (PIP) on a display of the device.
• PIP picture in a picture
• the ability to display a PIP allows a viewer to watch video on the main screen as well as other videos in the PIP. For example, one may watch a live football game as it's being played in a main screen while also watching video highlights of another game, or other information, in a smaller picture window.
• a single PIP does not allow for watching more than one other video at the same time as watching the main video.
• a viewer while watching a football game on the main video screen, may want to watch more than one other football game in smaller picture windows.
• Conventional systems that display only one PIP therefore compromise the enjoyment and utility of the display.
• the method comprises displaying a video on a screen of the device; selecting a first object in the video; displaying a first PIP on the screen, wherein the first PIP comprises the first object; and displaying a second PIP on the screen, wherein the second PIP comprises a second object.
• the system comprises a selection module configured to recognize selection of a first object to track in a first video; and a screen configured to display the first video, a first PIP comprising the first object as it is being tracked, and a second PIP comprising a second object.
• embodiments of a system for displaying a plurality of PIPs on an electronic device comprise means for displaying a video on a screen of the device; means for selecting a first object in the video; means for tracking the first object; means for displaying a first PIP on the screen, wherein the first PIP displays the first object as it is being tracked; and means for displaying a second PIP on the screen, wherein the second PIP displays a second object.
• a non-transient computer readable medium configured to store instructions that when executed by a processor perform a method for displaying a plurality of PIPs on an electronic device.
• the method comprises displaying a video on a screen of the device; selecting a first object to track in the video; displaying a first PIP on the screen, wherein the first PIP displays the first object as it is being tracked; and displaying a second PIP on the screen, wherein the second PIP displays a second object.
• Figure 1 depicts a front view of an embodiment of a display having two pictures in a picture (PIPs) with one of the PIPs showing a tracked object on the display.
• PIPs picture
• Figure 3 depicts a front view of the display of Figure 1 with three PIPs, where two of the PIPs are displaying tracked objects.
• Figure 7 is a flow chart diagramming one embodiment of a method for displaying and recording two PIPs on a display, where the position of the first PIP moves if it is interfering with the tracked object it is displaying.
• Figure 8 is a flow chart diagramming one embodiment of a method for displaying and recording four PIPs on a display, where the position of the PIP moves if it is interfering with the respective tracked object it is displaying.
• one of the PIP's may display a view taken from a second imaging sensor on the display device.
• the objects in the scene captured from the second image sensor may be shown in their own PIP window on the same screen as the objects viewed by the first image sensor.
• the second image sensor may be viewing an entirely different field of view as the image sensor.
• the first image sensor may be on the front of a cell phone, and the second image sensor may be on the back of the cell phone.
• multiple PIPs allow for objects from the first field of view to be displayed on the screen along with objects from the second field of view.
• the second imaging sensor may be on an entirely different device. For instance, a separate video source may provide the video to display in one or all of the PIPs.
• the first object 112 is in the same field of view as that shown by the display 110.
• one or both displayed objects 120, 124 being shown in the PIPs 116, 118 are in the same field of view as each other.
• the second object is in a different field of view as shown in the main video 105 of the display 110.
• the second object is a user of the device 100, and the user's face is displayed as the second displayed object 120 in the second PIP 116.
• the two PIPs 116, 118 may be resized.
• the second PIP 116 may be expanded to the size shown by resized PIP 122.
• the second PIP 116 may thus have larger dimensions in four directions.
• the second PIP 116 may also have smaller dimensions, as well as different shapes, line weightings, etc.
• the boundary of the second PIP 116 may be dashed as shown by the boundary of the resized PIP 122.
• the location or locations on the display 110 to which the first PIP 118 moves when it interferes with the first object 112 may be controlled. For instance, when the first PIP 118 interferes with the first object 112, the first PIP 118 may move from its original location to a second location that is not interfering with other objects or specified areas on the display 110. In some embodiments, when the first PIP 118 interferes with the first object 112, the first PIP 118 may move to a location where the first PIP 118 will not interfere with the first object 112 or with the second PIP 116.
• More than four PIPs may be useful, for example, when recording and/or viewing microscopic images, such as a cluster of microorganisms, or viewing many objects, such as a herd of many animals.
• the various PIP's may be used to individually track the many microorganisms or animals of interest.
• the first PIP 118, the second PIP 116, the third PIP 126 and a fourth PIP 134 may be shown on the display 110.
• each PIP may display separate objects.
• the first PIP 118 may display the first displayed object 124, shown as a star-shaped object
• the second PIP 116 may display the second displayed object 120, shown as a face
• the third PIP 126 may display a third displayed object 128, shown as a triangular-shaped object
• the fourth PIP 134 may display a fourth displayed object 136, shown as a circular-shaped object.
• Embodiments with four PIPs 116, 118, 126, 134 possess the same abilities and features as other embodiments with only two or three PIPs, as discussed above.
• the four PIPs 116, 118, 126, 134 can similarly be resized, manually or automatically relocated, have changed transparency properties, may show tracked objects, may show objects from the same or from different fields of view, may display objects from different video sources, etc.
• Any features discussed above with respect to embodiments with two or three PIPs, though not explicitly addressed with respect to embodiments with four PIPS may also be implemented, mutatis mutandis, in embodiments with four or more PIPs.
• any features discussed with respect to embodiments with four PIPs, though not explicitly addressed with respect to embodiments with two or three PIPS may also be implemented, mutatis mutandis, in embodiments with two, three or more than four PIPs.
• the various embodiments of the display 110 discussed with respect to Figures 1-4 may be embodied on a wide range of devices and systems.
• Figure 5 depicts an embodiment of one such device 500.
• the device 500 may incorporate any of the features of the various components as discussed above with respect to Figures 1-4, including tracking objects, resizing and/or relocating PIPs, changing PIP properties such as transparency, displaying the PIPs in different configurations, etc.
• the device 500 may be a mobile device, such as a tablet as shown, or a handheld, portable, laptop or desktop computer.
• the device 500 may be a small, handheld computing device, with a display screen with touch input and/or a miniature keyboard, weighing less than 2 pounds (0.91 kg).
• the device 500 may also be manufactured by companies such as Apple®, Nokia®, HTC®, LG®, BlackBerry®, and Motorola Mobility®.
• the device 500 has an operating system (OS), and can run various types of application software, such as apps or applications.
• the device may be equipped with Wi-Fi, Bluetooth, and GPS capabilities that can allow connections to the Internet and other Bluetooth-capable devices, such as an automobile or a microphone headset.
• One or more imaging sensors 512 which may be a camera or cameras, or a media player feature for video and/or music files may be on the device 500 along with a stable battery power source such as a lithium battery.
• the device 500 may be a smartphone. This may be a mobile phone built on a mobile operating system, with advanced computing and connectivity capabilities. It may combine the functions of a personal digital assistant (PDA), email functionality, and a mobile phone.
• PDA personal digital assistant
• the device 500 may also have the functionality of portable media players, low-end compact digital cameras, pocket video cameras, and GPS navigation units to form one multi-use device 500.
• the device 500 also includes a high-resolution touchscreen and web browsers that display standard web pages as well as mobile-optimized sites. High-speed data access may be provided by Wi-Fi, mobile broadband, near field communication (NFC) and/or Bluetooth.
• the first viewed object 504 may be in a field of view that is in a first direction 505 relative to the device 500, as depicted.
• the second viewed object 502 such as a user of the device 500, may be in a field of view that is in a second direction 507 relative to the device 500, as depicted.
• the directions 505, 507 of the two fields of view of the device 500 are in opposite directions.
• the field of view in the direction 507 is viewed by an imaging sensor 512, which may be a camera.
• the field of view in the opposite direction 505 is viewed by an imaging sensor (not shown), which may be another camera, on the opposite side of the device 500 as that of the imaging sensor 512. Therefore, the first displayed object 516 in the first PIP 506 is from a different field of view of the second displayed object 514 in the second PIP 508.
• the main video 503 may display objects from the field of view in the direction 505.
• the first PIP 506 displays the first viewed object 504 which is in that same field of view.
• the second PIP 508 displays the second viewed object 502 from the field of view in the direction 507.
• the two PIPs 506, 508 are overlaid onto the main video 503.
• the PIPs 506, 508 completely obscure the parts of the main video 503 that are "behind” or "underneath” the PIPs 506, 508.
• the PIPs 506, 508 partially obscure the parts of the main video 503 that are "behind” or “underneath” the PIPs 506, 508.
• the PIPs 506, 508 are transparent to some percentage, such as 25%, 50%, 75% etc.
• the various videos or objects being shown by the various PIPs by the devices, systems and methods disclosed herein may also be connected with other video sources or displays.
• the videos shown in the PIPs may originate from a device or source other than the device on which the PIPs are being shown. Further, in some embodiments, the videos shown in the PIPs on one device may also be shown on other devices and/or displays. This is discussed in further detail herein, for example with respect to Figure 6.
• Figure 6 depicts a block diagram of one such embodiment of a device 600 with a display 625.
• the device 600 may be a mobile device, such as those discussed above with respect to Figure 5.
• the device 600 may also be a cell phone, digital camera, personal digital assistant, or the like. It may also be a more stationary device such as a desktop personal computer, video conferencing station, or the like.
• the device 600 may comprise a recording system having modules for displaying, recording and/or managing multiple PIPs.
• the device 600 has a set of components including a processor 620 linked to an imaging sensor 615.
• a working memory 605, storage 610, electronic display 625, and memory 630 containing various modules are also in communication with the processor 620.
• the processor 620 may be a general purpose processing unit or a processor specially designed for imaging applications. As shown, the processor 620 is connected to the working memory 605 and the memory 630.
• the modules in memory 630 may be software, such as programs or applications.
• a plurality of modules may be in the device 600. These modules include instructions that configure the processor 620 to perform various image processing and device management tasks.
• the modules may include modules related to PIP, as well as traditional photographic applications, high dynamic range imaging, panoramic video, or stereoscopic imaging such as 3D images or 3D video.
• the memory 630 stores an imaging sensor control module 635, object of interest detection module 640, touch screen input module 655, settings management module 660, window display module 670, preview control module 675, operating system 680, selection module 685, recording module 690, and PIP display module 695.
• the working memory 605 may be used by processor 620 to store a working set of processor instructions contained in the modules of memory 630. Alternatively, working memory 605 may also be used by processor 620 to store dynamic data created during the operation of device 600.
• the processor 620 is configured by the several modules stored in the memory 630.
• the imaging sensor control module 635 may include instructions that configure the processor 620 to capture images with imaging sensor 615.
• the imaging sensor control module 635 may also include instructions that adjust the focus position of imaging sensor 615. Therefore, processor 620, along with image capture control module 635, imaging sensor 615, and working memory 605 represent one means for capturing an image using an imaging sensor.
• Recording module 690 may include instructions that configure the processor 620 to record the captured images.
• Touch screen input module 655 may include instructions that configure the processor 620 to receive touch inputs from a touch screen display, for example, display 625.
• the settings management module 660 may include instructions to manage various parameter settings for device 600. For example, parameters related to the configuration of the various PIPs may be managed by module 660.
• the window display module 670 may include instructions to manage the layout of data, such as zoom level, within the PIPs generated on display 625 on device 600. For example, the display 625 may include more than one image "window,” such as a PIP, within it. Some "windows" may display data at differing scales. Instructions within window display module 670 may configure the processor to translate data related to each of these sub windows into display commands for display 625.
• the preview control module 675 includes instructions that configure the processor 620 to display a preview window on the display 625 according to the methods described above.
• the preview control module 675 may include instructions that call subroutines in the imaging control module 635 in order to configure the processor 620 to capture a first image using the imaging sensor 615.
• the preview control module 675 may then call the object of interest detection module 640 to detect objects of interest in a first image captured by the imaging sensor 615. Instructions in the preview control module 675 may then invoke the settings management module 660 to determine how the operator has configured the preview window to display on the display 625, for example, in a PIP.
• Window display module 670 may invoke instructions in operating system 680 to control the display and cause it to display the appropriate preview window configuration on electronic display 625.
• the preview control module 675 may also include instructions that configure the processor 620 to verify, or seek verification from a user, of selected and/or detected objects of interest on display 625.
• the operating system module 680 is a mobile operating system and the device 600 is a mobile device.
• the mobile operating system used by the mobile device such as a smartphone, may be Google's Android, Apple's iOS, Symbian, Blackberry Ltd's BlackBerry 10, Samsung's Bada, Microsoft's Windows Phone, Hewlett-Packard's webOS, embedded Linux distributions such as Maemo and MeeGo, Mozilla's Firefox OS, Canonical Ltd.'s Ubuntu Phone, Tizen, or others.
• the mobile device can receive multiple operating system module 680 updates over its lifetime.
• the processor 620 may write data to the storage module 610. While the storage module 610 is represented graphically as a traditional disk device, those with skill in the art understand multiple embodiments could include either a disk based storage device or one of several other type storage mediums to include a memory disk, USB drive, flash drive, remotely connected storage medium, virtual disk driver, or the like.
• Figure 6 depicts a device comprising separate components to include a processor, imaging sensor, and memory
• the memory components may be combined with processor components to save cost and improve performance.
• Figure 6 illustrates two memory components, to include memory component 630 comprising several modules, and a separate memory 605 comprising a working memory, one with skill in the art would recognize several embodiments utilizing different memory architectures.
• the device 600 may have a design that utilizes ROM or static RAM memory for the storage of processor instructions implementing the modules contained in memory 630.
• the object of interest detection module 640 may communicate data related to the first viewed object 504 to the processor 620 which may then display the first object 516 in the main video 503.
• the first object 516 may be selected on the display 501 by, for example, the touch screen input module 655.
• a user of device 600 may touch the area of the display 501 corresponding to the first object 516.
• the touch screen input module 655 may then take this input and communicate it to the processor 620 for further processing.
• the first PIP 506 may be moved or resized using the PIP display module 695 or the window display module 670.
• tracking of the first object 510 in the first PIP 506 may be indicated by the settings management module 660 and carried out by the imaging sensor control module 635 or the object of interest detection module 640.
• any of the functions that may be carried out with the modules of Figure 6 involving the first object 510 in the first PIP 506 may also be carried out on the second displayed object 514 in the second PIP 508.
• the various sets of instructions in the modules of Figure 6 to be carried out on the various features shown in Figure 5 may be carried out by various modules and need not be limited to just one module for one capability.
• the object of interest detection module 640 may be used to detect the first object 510, it is understood that another module, for instance the window display module 670 may instead or in addition provide this functionality. It is therefore understood that the functionalities recited of the various modules and components of Figure 6 as applied to the features of Figure 5 are not the sole modules or components capable of carrying out the aforementioned functions but are merely listed as examples of how the disclosed implementations may be implemented.
• step 730 the first PIP is moved or otherwise altered on the display.
• the first PIP is moved or relocated to a different location on the display. In some embodiments, this movement is automatic such that the first PIP will move without any input from a user or viewer of the display.
• the location or locations on the display to which the first PIP moves may be repeated. For instance, the first PIP may always move to a same second location when it interferes with the first object. The first PIP may then move back to its original location if it interferes with the first object in this second location.
• the first PIP may move to a location where the first PIP will not interfere with the first object, with the second PIP, with the second object, or with any other indicated locations.
• Many other configurations of moving the first PIP are within the ordinary skill of the art and are thus within the scope of the present disclosure.
• the first PIP may alter its appearance. This may be for example to facilitate viewing the first object on the display if the first object overlaps with the first PIP. In some embodiments, the first PIP may disappear when there is interference and then reappear. In some embodiments, the first PIP may become transparent. For example, the first PIP 118 may become fifty percent transparent. Further, combinations of relocating and/or altering the appearance of the first PIP may be implemented. Also, any of the features or capabilities related to interference and/or moving as discussed above with respect to Figures 1 and 2 may be implemented in the method 700.
• the process 800 begins with step 805 wherein a video on a screen or display of a device is displayed.
• the video displayed in step 805 may comprise one or several objects of interest, which may be moving or stationary.
• This step 805 may further include receiving a selection of a first object in the video, detecting the first object, and/or tracking the first object805and/or detecting and tracking the first object.
• the process 800 then moves to step 810 where the first object is displayed in a first PIP on the display. 800Next, in step 815, it is determined whether the location of the first PIP on the display interferes with the location of the first object on the display.
• the process 800 moves to step 870 where the main video, the first PIP, the second PIP, the third PIP and the fourth PIP are recorded.
• the video is the main video on the display over which the first and second PIPs are situated.
• the video, the first PIP, the second PIP, the third PIP and the fourth PIP are all recorded in the same display.
• the recording may be of the main video with the four PIPs situated over the main video.
• the video, the first PIP, the second PIP, the third PIP and the fourth PIP are all recorded in different displays or otherwise in different files.
• the main video may be recorded on one display or file, the first PIP recorded in a first display or file, the second PIP recorded in a second display or file, etc.
• portions of process 800 are performed.
• process 800 may be performed with only two or three PIPs.
• process 800 is performed in different orders.
• the first PIP may be located first
• the third PIP may be located second.
• Other variations such as these to the process 800 are within the scope of the present disclosure.
• a software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non- transitory storage medium known in the art.
• An exemplary computer-readable storage medium is coupled to the processor such the processor can read information from, and write information to, the computer-readable storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an ASIC.
• the ASIC may reside in a user terminal, camera, or other device.
• the processor and the storage medium may reside as discrete components in a user terminal, camera, or other device.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Computer Hardware Design (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Business, Economics & Management (AREA)
• Marketing (AREA)
• Controls And Circuits For Display Device (AREA)
• Transforming Electric Information Into Light Information (AREA)
• Studio Devices (AREA)

Priority Applications (3)

Applications Claiming Priority (4)

Publications (1)

Family

ID=52582732

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (35)

Citations (16)

Family Cites Families (21)

• 2014
  • 2014-02-20 US US14/185,683 patent/US9973722B2/en not_active Expired - Fee Related
  • 2014-08-19 JP JP2016538968A patent/JP2016538601A/ja active Pending
  • 2014-08-19 CN CN201480046888.0A patent/CN105519097B/zh not_active Expired - Fee Related
  • 2014-08-19 EP EP14767172.1A patent/EP3039858A1/en not_active Ceased
  • 2014-08-19 WO PCT/US2014/051732 patent/WO2015031111A1/en not_active Ceased

Patent Citations (16)

Also Published As

Similar Documents

Legal Events