TW201319952A - Identifying visual media content captured by camera-enabled mobile device - Google Patents

Abstract

Automatic identification of media content is at least partially based upon visually capturing a still or video image of media content being presented to a user via another device. The media content can be further refined by determining location of the user, capturing an audio portion of the media content, date and time of the capture, or profile/behavioral characteristics of the user. Identifying the media content can require (1) distinguishing a rectangular illumination the corresponds to a video display; (2) decoding a watermark presented within the displayed image/video; (3) characterizing the presentation sufficiently for determining a particular time stamp or portion of a program; and (4) determining user setting preferences for viewing the program (e.g., close captioning, aspect ratio, language). Thus identified, the media content appropriately formatted can be received for continued presentation on a user interface of the mobile device.

Description

Identify visual media content captured by camera-enabled mobile devices

The present invention relates to mobile operating environments and, more particularly, to visually identifying visual media content captured by a camera-enabled mobile device.

For decades, the development of digital image processing has attempted to automate certain visual capabilities, such as image recognition. Computer vision has tried to identify obstacles in order to achieve self-directed viewing. Optical character recognition relies on techniques such as detecting skew in an image and performing character shape associations. The monitoring system attempts to identify biometric data such as faces to maintain security.

An example of image processing is to generate a digital key signature for each known segment of a video broadcast program, which digital signature can then be matched by a digital key signature generated for the unknown segment. This technology is used by broadcast advertisers for automatic tracking to see what commercials are being broadcast in a particular market. This type of processing benefits from the ability to sample high-fidelity versions of broadcast video signals. Additionally, such sampling and analysis can be performed by devices that have high processing capacity in time. Thus, the device performing image processing is typically not mobile or intended for consumer use.

Advances in technology have produced smaller and more powerful processing equipment. For example, there are currently a variety of portable, personal computing devices that are small, lightweight, and easy to carry, including wireless computing devices such as portable wireless telephones, personal digital assistants (PDAs), and paging devices. More specifically, portable radiotelephones, such as cellular phones and Internet Protocol (IP) phones, are capable of transmitting voice and data packets over a wireless network. In addition, many such wireless telephones include other types of devices, and such types of devices are also included herein.

Digital signal processors (DSPs), image processors, and other processing devices are often used in portable personal computing devices having digital cameras or displaying images or video data captured by digital cameras. The processing device described above can be used to provide video and audio functions for processing received data such as image data or performing other functions. Digital imaging technology enables small devices to capture image data and enhance and send image data.

There are situations in which a user is using media content at a certain venue, but wants to watch or read the material on their mobile device while on the go. It is desirable to continue watching or reading the content after leaving the scene, and the user wishes to be able to easily locate the content without the need for a laborious search. This is especially the case in the absence of direct access to the source of media content. In addition, unlike voice recognition, various complex situations can occur when automatically visually recognizing segments of media content. The orientation of the image and the unrelated images within the field of view can complicate efforts to capture images or video segments via the viewfinder of the portable personal computing device.

A brief summary of one or more aspects is provided below to provide a basic understanding of the aspects. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key or critical elements in all aspects or the scope of any or all aspects. The sole purpose of the summary is to present some of the embodiments of the invention

In one aspect, a method for identifying visual media content is provided. Receive images from the camera of the mobile device. Detects the quads contained in the image. The visual media content contained within the quadrilateral is captured to identify the visual media content.

In another aspect, at least one processor for identifying visual media content is provided. The first module is for receiving images from a camera of the mobile device. The second module is for detecting a quadrangle included in the image. The third module is configured to capture visual media content included in the quadrilateral to identify the visual media content.

In another aspect, a computer program product for identifying visual media content is provided. Non-transient computer readable storage media contains multiple sets of code. The first set of codes is used to cause the computer to receive images from the camera of the mobile device. The second set of codes is used to cause the computer to detect the quadrilateral contained in the image. The third set of codes is for causing the computer to capture visual media content contained within the quad to identify the visual media content.

In yet another aspect, an apparatus for identifying visual media content is provided. Means are provided for receiving images from a camera of the mobile device. A member for detecting a quadrilateral contained in the image is provided. Means are provided for capturing visual media content contained within the quad to identify the visual media content.

In yet another aspect, an apparatus for identifying visual media content is provided. The camera of the mobile device produces an image. The computing platform detects a quadrilateral contained in the image received from the camera and captures visual media content contained within the quadrilateral to identify the visual media content.

To achieve the above and related objects, the one or more aspects include the features that are fully described below and particularly pointed out in the claims. The specific description of the one or more aspects is set forth in the following description and the annexed drawings. However, the features are only a few of the various ways in which the principles of the various embodiments can be applied, and the description is intended to include all such aspects and equivalents.

There are occasions where it is necessary to identify and transmit media content (eg, text, images, video) being viewed via another component. Thereby the user can use the media content in a convenient manner. For example, a user can read media content, such as text-based news or entertainment articles contained in printed periodic publications or displayed on a computer monitor. Similarly, media content can be graphical, such as an illustrative picture or photo. As another example, a user may view a scene in which visual media content is being displayed. In order to quickly capture the content being viewed for later retrieval, the user can conveniently use the camera function. In order to quickly capture and later query the entire content of text-based or graphic articles or visual media content, users can use camera-enabled mobile devices (eg, smart phones, portable game consoles, personal digital assistants) and many more).

Various aspects will now be described with reference to the drawings. In the following description, numerous specific details are set forth However, it will be apparent that the various aspects can be implemented without such specific details. In other instances, well-known structures and devices are illustrated in block diagram form in order to facilitate the description.

Referring initially to FIG. 1, mobile device 100 is illustrated as device 102, device 100 identification visual media content 104 presented by the display 106 for the user 108 viewing the outside. Camera 110 of mobile device 102 produces digital image 112 . Computing platform 114 to detect 116 quadrilateral, quadrilateral 116 and 106 corresponding to the external display is included in the digital image received from the camera 110 112. The computing platform 114 can direct the higher resolution imaging performed by the camera 110 to include the quadrilateral 116 to capture the visual media content 104 contained within the quadrilateral 116 . The computing platform 114 , the remote server 118, or both, can analyze the visual media content 104 for identification with respect to the repository 120 having the media content file 122 . A more complete version 126 of the identification information 124 or visual media content 104 can then be transmitted to the mobile device 102 via the airway 128 for presentation to the user 108 in the user interface 130 .

In FIG. 2 , a method or sequence of operations 200 for identifying visual media content is illustrated. An image is received from a camera of the mobile device (block 202 ). A quadrilateral contained within the image is detected (block 204 ). The visual media content contained within the quadrilateral is retrieved to identify the visual media content (block 206 ).

In one exemplary use, consider the case where the user is watching media content, such as a movie, on a television, but chooses to view the remaining portion on the mobile device. The user points the camera of the mobile phone to the TV. The mobile device is triggered to recognize the program on the running TV via the viewfinder of the mobile device by processing the input frame. In particular, the image on the TV is cropped from the background by utilizing the expectation that the television is a quadrangle in the center of the viewfinder and is generally brighter than the surrounding environment. The captured portion of the content (perhaps preprocessed for machine vision recognition) can be sent to the server to view which movie the sequence came from. For example, the above system can use a hash table to quickly view which scenes in which movies should be further examined. The key to a hash table is to use a local bar graph of the color found in the frame. For example, if the upper left quadrant of image 1 has 50% blue primitives, 30% white primitives, and 20% black primitives, and then changes to 30% blue primitives after a given time, With 50% white primitives and 20% black primitives, this feature combined with the other three quadrants will be used to reduce the scene from the frame in the movie library. Based on the reduced set, the program will then be repeated based on frequency rather than color. If the upper left quadrant has 700 edge primitives at a given time and becomes 400 edge primitives after 300 frames, the pattern will further streamline the collection. Based on this reduced set of fragments, the system can use SIFT or some other feature-based extraction method to reduce the exact frame. Once the name and time stamp of the movie is found, the device can connect to a dedicated server, purchase and then download the entire movie, but start the stream title from the point where the user is currently watching the movie on the TV.

In addition to the image detection method, it is also possible to use a microphone to capture audio from the TV and use the audio in the hash view function to assist in the determination of the media content. Alternatively, the mobile device can employ an image processing algorithm on the local end or on the remote server to identify legitimate forensic video watermarks. Video watermarks can contain timestamps, customer identifiers, and content identifiers to allow extraction of such data, even after compression and after multiple bit-to-analog-digital conversions.

In another aspect, if the user is reading an article, magazine, newspaper, book, etc. on the PC, but chooses to access the content on the mobile device. The user takes a photo of the content. For example, the camera uses a macro mode to focus on objects that are less than 2 inches from the lens and has sufficient resolution for optical character recognition. Thus, the search engine can be used to search for the identified digital string and present the best article match to the user on the user interface for selection. The identified matching articles can be bookmarked or downloaded for future viewing. If the content is copyright protected and/or not found online, then watermarking techniques can be used to determine if the user is the rights holder of the content. If the watermarking technique is not used but the content is still copyright protected, the user can enter the subscription identifier from the physical copy of the content (eg, book, journal) to access the content.

Alternatively, if the article is not found, the system can push a similar article of the topic or an article of the same author.

In one exemplary aspect, in Figures 3A-3C , a method or sequence of operations 300 for capturing and recognizing visual media content within a detected external display imaged by a camera is provided. The user directs the camera of the wireless mobile device to a display or monitor (block 302 ). The user selects user controls for capturing image content (block 304 ).

In one aspect, the mobile device is enabled to capture one type of visual media content (eg, text, graphic images, video images). In another aspect, the mobile device can receive an indication of what type of visual media content to retrieve or have retrieved. As a further alternative, the mobile device can automatically determine the type of visual media content among several options. For such purposes, the mobile device can determine the intent or appropriateness for the text capture (block 306 ). If so, the capture can be directed to a high contrast (usually black and white) text without native motion (block 308 ). The mobile device can also determine the intent or appropriateness for the image capture (block 310 ). If so, the target can be colored and have varying contrast, but there is still no native motion (block 312 ). The mobile device can also determine the intent/appropriateness for video capture (block 314 ). If so, the target can have a native motion (block 316 ).

In an exemplary aspect, at block 317 , the color conversion process supports the above decisions. It can be useful to transform from the input color space (usually from RGB) to the luminance-chrominance space because the system can determine the amount of color in the image. The amount of color in the image under study can be determined based on Cb and Cr, where Cb and Cr are in the median of the range (eg, 128), indicating no color. Alternatively or additionally, a particular color such as white and black or blue may be detected, the color representing the print. Alternatively or additionally, a consistent background color can be detected, wherein the background color can be discarded in order to rely on the image or digital content. Alternatively or additionally, a Gabor filter may be used to determine whether the content has a regular pattern at a certain frequency, and a pattern having a rule at a certain frequency may represent text having a particular font size.

A quadrilateral image or video source can be found in the low resolution camera preview program (block 318 ), which will be discussed with reference to FIG. 3B . With further reference to FIG. 3A , alternatively, there may be sufficient processing power to use the higher resolution initial capture. In another aspect, the user can assist by focusing the viewfinder on a portion of the camera preview (block 320 ). In another aspect, the mobile device displays the identified source or display candidate source for selection by the user (block 322 ). In the latter case, the user interface receives the user selection and proceeds to the decision image/video source (block 324 ). If desired or enabled, the user can assist with the prompt by aligning or focusing the camera or clicking on the viewfinder (block 326 ). In some implementations, the mobile device can send a region of interest (ROI) to the camera for optimal resolution settings (block 328 ). The mobile device retrieves one (or more) frames from the area (block 330 ).

The media content of the captured frame(s) is identified (block 332 ). In various aspects, the identification may be performed by the mobile device, by decentralized processing between the mobile device and the remote server, or primarily by a remote server, as further described with respect to FIG. 3C . With continued reference to FIG. 3A , the mobile device can download the media content (block 334 ).

In FIG. 3B , an exemplary method 318 for viewing an image/video source of a quadrilateral within an image is provided (block 318 ). If a lower camera preview resolution is used, one or more of the initial images may have VGA resolution on the "n" frames (n = 1 means no motion) (block 336 ).

With further reference to FIG. 3B , an ROI map having 255 values can be established in which the ROI map can be blurred due to a decrease in value upon clicking of the user input (block 338 ). In some instances, the detection may utilize the tendency of the display or monitor to have a brighter illumination level than other surfaces in the room. To this end, a "glowing" ROI map is created by thresholding the maximum illuminance for values > x (eg, achieving 20% of the remaining primitives), such as defined by red-green-values (RGB) ( Block 340 ). In some instances where video capture is desired, detection can take advantage of the fact that there is a change in display motion between frames. To this end, a "motion" ROI map is created by the difference between each frame and the previous "m" frame (eg, m=3). The maximum difference ([Delta]) can be recorded to help remove signal interference (block 342 ). The image from the viewfinder can be cropped based on the weight of the ROI map (block 344 ). The cropped image is sent to the fast corner detector (block 346 ). Can compare threshold primitives ( 6) A closer corner point (CP) is clustered (block 348 ). If the CP is within the complete nxn region within the bright map, the CP can be deleted (block 350 ). If the movement of the m × m region in the bright FIG entirely, may be deleted CP (block 352).

A candidate for identifying a quad based on the deleted CP (block 354 ):

(i) no convex surface (the sum of the angles is 360°) (block 356 );

(ii) any internal angle > 110° (block 358);

(iv) video aspect ratio (4:3, 16:9) (block 360 );

(v) 1/25 of the area ≧ image (block 362 );

(vi) having two (2) equal adjacent corners (block 364 ); and

(vii) A candidate for the quadrilateral is associated based on the depth view (block 365 ).

Thus, a group representing the association of the shape of the monitor or display is identified.

For the last solution using depth inspection, a specific corner group can be determined to have a certain depth by using depth focusing capability. Thus, the corners can be deleted based on the corners in the foreground depth or background depth that is considered to be unrelated to the set of candidate corners. The depth information can be used to determine the set of corners at the same depth level in the image.

U.S. Patent Application "System and Method to Generate Depth Data Using Edge Detection", filed on August 5, 2008, in which the inventor is Babak Forutanpour, Serial No. 12/185,887, Publication No. 20100033617 A1. Additional disclosure regarding the use of depth inspection is described in the application to the assignee and is hereby incorporated by reference in its entirety.

Four (4) corners of the candidate are added to the main list (block 366 ). The quadrilateral shape formed from the set of corners in the master list is selected such that a large pseudo-quadrant that substantially encompasses the entire image is not allowed to surround the smaller quadrilateral. For example, a quadrangle larger than one-fifth of the image area is not allowed to surround other candidate quadrilaterals. In an exemplary scenario, any quadrilateral that occupies 80% of another quadrilateral is deleted (block 368 ).

The corners may be deleted based on detecting a border shape and a uniform color representing the monitor/display (block 370 ). For another example, the bar graphs of the left half and the right half of the quadrilateral should match. Alternatively or additionally, the upper half of the quadrilateral and the lower half of the bar graph should match. In another aspect, one side of the bezel is allowed to be different from the side of the side of the opposite side in order to accommodate asymmetric placement of user controls, audio speakers, mounting or support structures, and the like. In an exemplary aspect, the match is calculated within a certain limit (eg, 20%) of the total primitive via a difference between the binned histogram for one side and the other side, where the limit can It is different. Alternatively, if converted to a hue saturation value (HSV), the average hue can be limited to a low value (eg, within 10%). In an exemplary scenario, the deleted border (1/14 of the width) has a bar graph with any quadrilateral that is farther than the standard deviation of 1.5.

Due to this exemplary recognition, a quadrilateral corresponding to the display or monitor can be identified from the image.

In FIG. 3C , an exemplary method or sequence of operations 332 for identifying media content of the captured frame(s) is provided. The ability to cooperate with the remote server is determined (block 371 ). For example, this capability may be limited due to available resources, interference, channel fading, transmit power limitations, user usage restrictions, and the like. To make adjustments for some or all of these considerations, the bandwidth of the connection to the server can be tested (block 372 ). Device performance constraints (eg, speed and availability of a central processing unit (CPU), configuration of digital signal processing hardware/software, etc.) can be determined (block 374 ). The cost of the user preference or usage bandwidth can be accessed (block 376 ). The capability constraint may also be a power limitation based on the power consumption of the image processing performed by the local end or the power required to transmit the variable number of image data. Capability constraints can also be correlated to the end-to-end time used to process and send image data.

In some instances, one characteristic of capability is decisive. For example, a low-performance mobile device may not be able to perform additional digital image processing, so the original image data must be uploaded regardless of channel restrictions.

In another example, a set of possible modes is determined based on which portions of the digital image processing can be performed at the local or remote end. The user experience can then be enhanced based on user preferences (eg, implementation cost), system preferences for traffic optimization, or by reducing the time to complete the digital image processing to select for an optimal solution.

For example, a look-up data table (LUT) can be accessed to determine image processing for media content identification between the device and the server (block 378 ). Therefore, in one scheme, the local processing mode is considered appropriate. For example, a very low connection is detected, it is determined that a capable device is available or has been selected (block 380 ). Performing an n × n + bar graph based on edge detection & Scale Invariant Feature Transform (the SIFT) (block 382). The feature vector is sent to the server to save bandwidth (block 384 ).

In another aspect, the sharing processing mode is considered appropriate. For example, a media connection is detected, the local and remote components are determined to be available for sharing processing, or the mode has been selected (block 386 ). The n×n based bar graph + edge detection is performed without performing SIFT (block 388 ). The result is sent to the server (block 390 ).

In another scenario, the remote processing mode is considered appropriate. For example, the zone device is determined to be unable to perform the process, decides that the quick connect is available, or has selected the mode (block 392 ). Instead of processing, the captured segment is sent to the server (block 394 ).

After the remote server has completed any remaining processing and matched for the remote server media content directory, the mobile device receives a candidate list of media content matches (block 396 ). In one scenario, consider presenting the constraints of the above matching on the mobile device. In addition, user assistance (eg, additional alignment of the camera, interaction with the user interface, etc.) may be required to uniquely identify the media content. To this end, it may be decided that no matches are obtained and more media content needs to be retrieved (block 397 ). Alternatively or additionally, a limited match (e.g., three) may be determined, the matches being adapted to be presented at the user interface for selection by the user (block 398 ). Alternatively or additionally, a large number of matches can be identified. In response, the mobile device can retrieve more content or send a higher resolution version of the captured content for use by the server in the match (block 399 ).

In another exemplary aspect, in FIG. 4 , method 400 performs image analysis using a camera-enabled communication device or device, such as a smart phone. A frame of low resolution mode from the camera viewfinder can be stored (block 402 ). Alternatively, image analysis may be performed for a higher resolution snapshot or snapshot sequence that is automatically or manually selected (block 404 ). The algorithm is executed to extract a display of a portion of the viewfinder image corresponding to an external display or monitor (e.g., a television monitor) that is presenting the media content (block 406 ). For convenience, the area may be identified as a liquid crystal display (LCD), which is a type of display commonly used in e-books, televisions, and computers, although it is contemplated that other techniques consistent with aspects of the present invention can be used.

In FIG. 5 , an exemplary lower resolution image 500 of a room includes a display 502 viewed from the side.

With further reference to FIG. 4 , the selection of the correct portion may involve, in part, performing a series of processes, either entirely at the local end or performed decentralized at the local end and at the remote end or all at the far end (block 408 ).

For example, a Harris corner detector can be executed to view all corners in the region of interest (ROI) (block 410 ). The entire arrangement and combination of the found corners can be checked until a set is created (block 412 ) to:

(1) The content in the area has an average brightness that is much higher than the average brightness of the overall image, and the ratio is expressed as " 1 " (block 414 ).

(2) The lines connecting the four corners are uniform and have substantially the same hue in the hue saturation value (HSV) space (block 416 ). For example, the border elements of the inspection monitor/display have a substantially or substantially identical color with one or both pairs of opposing sides having approximately the same width. In an exemplary aspect, it may be decided that the border elements have an RGB match within one threshold (eg, 20%) between each other. The allowable border thickness can be based on the LUT. For example, when the area within the quadrilateral is 1/10 of the image, the border should be 1/30 of the horizontal width of the x-axis of the image. It would be expected that the 640 x 480 image would be a television with a border of approximately 20 primitives.

(3) Deleting the perspective of the four (4) points from the area that does not match the framing of the object at or below the height of the eye (eg, deleting the roof light) (block 418 ).

In FIG. 6 , a schematic lower resolution image 600 of the ROI portion of the room (including the display 602 viewed from the side) has detected a plurality of corners 604 , which may be used to automatically define the display 602 to enable A candidate for media content 606 is retrieved.

In FIG. 7 , the set of candidate primitives 700 for a room derived from corner detection includes a plurality of points defining external and internal points 702 and 704 of display 706 , and image points 708 and within media content 710 . The extraneous points 712 outside of display 706 need to be selectively deleted.

With further reference to Figure 4 , an accurate corner of the LCD display is now found, and then optionally, if the perspective is greater than a certain threshold p , the smart phone can better correct the perspective (block 420 ).

For example, the threshold p can be based on the length ratio of the opposing lateral sides. For example, consider the ratio p ₁ indicating that the lateral sides are within 90% of each other. It can be determined that the matching algorithm is robust enough to enable matching without correcting the distortion caused in the captured video image. For another example, consider the ratio p ₂ indicating that the lateral sides are between 90% and 70% of each other. Correction may be required to correct for distortion due to perspective in this range. For another example, consider the ratio p _3, indicating the lateral sides opposite to each other larger differences in size and thus can not be corrected and if the cause doubt to find the right quadrilateral. Suppose the user does not attempt to capture at this non-perpendicular angle. It should also be recognized that vertical and horizontal perspectives may use different thresholds.

In one aspect, any rotation of the image from the vertical plane relative to the camera viewpoint can be corrected. You can find a pair of parallel lines (for example, the top/bottom side, or the left/right side) and rotate the entire image digitally so that the two lines are 0 or 90 degrees relative to the image, where the calculation is more Close to the angle.

Create a bar chart image n × n rectangular or square grid, e.g., n = 3 (block 422).

The number of primitives in each region is calculated by hard or soft edges (block 424 ). Various definitions can be used for hard (sharp) edges and soft (fuzzy) edges. For example, a "hard edge" primitive is this primitive: a neighboring primitive that is n from the primitive (eg, n=2 in all directions) is far larger than the primitive of the primitive itself or A much smaller value, such as a threshold >120. If the value of a primitive is between two neighbors and the two neighbors have different values from each other, the primitive is on the "soft edge". The degree of sharpness in the image can indicate discontinuities in depth, discontinuities in the surface direction, changes in material properties, or changes in scene illumination.

The n ² red green blue (RGB) values and hard edge values + soft edge values are combined into the data payload and transmitted (block 426 ). By sending not only the color information to the server but also the number of hard/soft primitives, the server can use this information to search the frame from the movie library for blocks with similar characteristics. In short, just sending color information may not be enough. It may not be enough to know how many red/green/blue a block has. In the case of excessive RGB in the image block, a modified screening program can be used. For example, the candidate list can also be narrowed down by sending information about the block having 45 primitives on the hard edge and 39 primitives on the soft edge. In the ideal case of no transmission or processing constraints, the entire block can be sent to enable the server to subtract the two images frame by frame.

Media content (eg, movies, television, news articles, radio broadcasts, podcast programs, etc.) is recognized and formatted for continued presentation on the mobile device (block 428 ). If the user does not have sufficient rights to use the material, then a data rights subscription can be made (block 430 ).

With the foregoing, the user is provided with a convenient way to continue to use specific media content via the mobile device without the cumbersome processing of identifying, locating and accessing the media content via the mobile device.

Referring to Figure 8 , a system 800 for identifying visual media content is illustrated. For example, system 800 can reside at least partially within a user equipment (UE). It will be appreciated that system 800 is represented as including a plurality of functional blocks, which may be functional blocks representing functions implemented by a computing platform, a processor, a software, or a combination of the above (e.g., firmware). System 800 includes a logical grouping 802 of a plurality of electronic components that are capable of acting in conjunction. For example, logical grouping 802 can include an electrical component 804 for receiving images from a camera of a mobile device. Additionally, logical grouping 802 can include an electronic component 806 for detecting quadrilaterals contained within the image. For another example, logical grouping 802 can include an electronic component 808 for capturing visual media content contained within a quad to identify the visual media content. Further, system 800 may include memory 820, storage 804 for executing electronic components - 808 functions associated instruction. Although memory is shown outside the body 820, it is to be understood that the electronic component 804 - One or more among 808 can exist within memory 820.

In Figure 9 , an apparatus 902 for recognizing visual media content is illustrated. A component 904 is provided for receiving images from a camera of the mobile device. A providing member 906 is provided for detecting a quadrilateral contained in the image. A component 908 is provided for capturing visual media content contained within the quadrilateral to identify the visual media content.

10 is a block diagram of a particular mobile device 1000 , including a bar graph, a corner detector, and a scale invariant feature transform (SIFT) generator 1064 . Mobile device 1000 can be implemented in a portable electronic device and includes a signal processor 1010 , such as a digital signal processor (DSP), coupled to memory 1032 . A bar graph, corner detector, and scale invariant feature transform (SIFT) generator 1064 is included in signal processor 1010 . In an illustrative example, the corner detector and SIFT generator 1064 operate in accordance with or in any combination as described with respect to Figures 1-7 .

Camera interface 1068 is coupled to signal processor 1010 and is also coupled to a camera, such as camera 1070 . The camera interface 1068 can be adapted to automatically capture a scene in response to a single image capture command (such as a user "clicking" on a shutter controller or other image capture input), either automatically or in response to a signal generated by the DSP 1010 . Images. Display controller 1026 is coupled to signal processor 1010 and display device 1028 . An encoder/decoder (transcoder) 1034 can also be coupled to the signal processor 1010 . Speaker 1036 and microphone 1038 can be coupled to transcoder 1034 . Wireless interface 1040 can be coupled to signal processor 1010 and wireless antenna 1042 .

The signal processor 1010 is adapted to detect corners in the image data based on changes in luminance values between adjacent data points as previously described. The signal processor 1010 is further adapted to generate image material 1046 , such as a depth map or other form of depth material, as previously described, the image material 1046 being exported using a collection of image data. By using depth focusing capabilities, it can be decided that certain corner groups have a certain depth. Thus, the corners can be deleted based on the corners in the foreground depth or background depth that is considered to be unrelated to the set of corner candidates. In an exemplary scenario, in addition to using the brightness and motion ROI maps, when the object is blurred or sharp, the camera can scan the lens to view the focus level. Based on this information, it can be determined whether there is no edge, a soft edge, or a hard edge. Corners at the same depth can be considered to be coplanar. Alternatively, the three-dimensional coordinates of the corners may be determined based in part on the depth information to determine coplanar points that are not perpendicular to the camera.

The image material may include video material from camera 1070 , image data from wireless transmission via antenna 1042, or data from other sources such as external devices coupled via a universal serial bus (USB) interface (not shown). These are all illustrative and not limiting examples.

Display controller 1026 is configured to receive the processed image material and provide the processed image data to display device 1028 . Additionally, memory 1032 can be configured to receive and store processed image data, and wireless interface 1040 can be configured to receive processed image data for transmission via antenna 1042 .

In one particular embodiment, signal processor 1010 , display controller 1026 , memory 1032 , transcoder 1034 , wireless interface 1040, and camera interface 1068 are included in system-in-package or on-wafer system device 1022 . In one particular embodiment, input device 1030 and power source 1044 are coupled to mobile on-wafer system device 1022 . Moreover, in one particular embodiment, as shown in FIG. 10 , display device 1028 , input device 1030 , speaker 1036 , microphone 1038 , wireless antenna 1042 , camera 1070, and power source 1044 are external to system-on-chip device 1022 . However, each of display device 1028 , input device 1030 , speaker 1036 , microphone 1038 , wireless antenna 1042 , camera 1070, and power source 1044 can be coupled to a component of system-on-chip device 1022 , such as an interface or controller.

In an exemplary scenario, the mobile device can perform identification and transmission of media content using multiple input multiple output (MIMO) cellular communication capabilities. In one exemplary embodiment, MIMO system employs multiple (N _T) transmit antennas and multiple (N _R) receive antennas for data transmission. By the N _T transmit antennas and N _R MIMO channels receive antennas may be decomposed into N _S independent channels is also referred to as spatial channels, where _{N S ≦ min {N T,} N R}. Each of the N _S independent channels corresponds to one dimension. The MIMO system can provide higher performance (e.g., higher throughput and/or greater reliability) if additional dimensions established by multiple transmit and receive antennas are utilized.

The MIMO system can support time division duplex ("TDD") and frequency division duplex ("FDD"). In a TDD system, the forward link transmission and the reverse link transmission are on the same frequency range, such that the mutual principle allows the forward link channel to be estimated from the reverse link channel. This allows the access point to extract the transmit beamforming gain on the forward link when multiple antennas are available at one access point.

The teachings herein may be incorporated into a node (eg, a device) that uses various elements to communicate with at least one other node. Figure 11 illustrates several exemplary elements that may be used to implement communication between nodes. In particular, FIG. 11 illustrates a wireless device 1110 (eg, an access point) and a wireless device 1150 (eg, an access terminal) of a MIMO system 1100 . At device 1110 , the traffic data for the plurality of data streams is provided from data source 1112 to a transmit ("TX") data processor 1114 .

In some aspects, each data stream is transmitted via a respective transmit antenna. The TX data processor 1114 can format, encode, and interleave the traffic data for each data stream based on a particular coding scheme selected for each data stream to provide coded material.

The coded data of each data stream and the pilot frequency data can be multiplexed using OFDM technology. The pilot data is typically a known data pattern that is processed in a known manner, and the pilot data can be used at the receiver system to estimate the channel response. The multiplexed pilot and encoded data of the data stream is then modulated (ie, symbol mapped) based on a particular modulation scheme (eg, BPSK, QPSK, M-PSK, or M-QAM) selected for each data stream, To provide modulation symbols. The data rate, encoding, and modulation for each data stream can be determined via instructions executed by processor 1130 . Data memory 1132 can store code, data, and other information used by processor 1130 or other components of device 1110 .

Modulation symbols for all data streams can be provided to a TX MIMO processor 1120, processor 1120 may further process these modulation symbols (e.g., for OFDM). TX MIMO processor 1120 then provides 1122a through 1122t N _T modulation symbol streams to N _T transceivers ( "XCVR"), each transceiver having a transmitter (TMTR) and a receiver (RCVR). In some aspects, TX MIMO processor 1120 can use beamforming weights for the symbols of the data stream and the antenna from which the symbol is being transmitted.

Each transceiver 1122a through 1122t receives and processes a respective symbol stream to provide one or more analog signals and further conditions (e.g., amplifies, filters, and upconverts) analog signals to provide for transmission over a MIMO channel. Modulated signal. 1124a through 1124t are then transmitted from N transceivers 1122a through 1122t _T modulated signals from N _T antennas.

At the device 1150 by N _R antennas 1152a through 1152r receive the transmitted modulated signals, the received signal from each antenna 1152a-1152r provided to a respective transceiver ( "XCVR") 1154a to 1154r. Each of the transceivers 1154a through 1154r can condition (e.g., filter, amplify, and downconvert) the respective received signals, digitize the conditioned signals to provide samples, and further process the samples to provide corresponding "received" symbols. flow.

A receive ( "RX") data processor 1160 then receives and processes the N _R received symbol streams from N _R transceivers 1154a through 1154r based on a particular receiver processing technique to provide N _T th "detection" symbol streams. The RX data processor 1160 then demodulates, deinterleaves, and decodes each detected symbol stream to recover the traffic data for the data stream. The processing performed by RX data processor 1160 is the inverse of the processing performed by TX MIMO processor 1120 and TX data processor 1114 at device 1110 .

Processor 1170 periodically determines which precoding matrix to use. Processor 1170 forms a reverse link message comprising a matrix index portion and a rank value portion. Data memory 1172 can store code, data, and other information used by processor 1170 or other components of device 1150 .

The reverse link message can include various types of information related to the communication link and/or the received data stream. The reverse link message may then be processed by a TX data processor 1138, modulated by a modulator 1180, conditioned by transceivers 1154a through 1154r, and transmitted back to the device 1110, from TX data processor 1138 further source 1136 Receive traffic data for multiple streams.

At device 1110 , the modulated signal from device 1150 is received by antennas 1124a - 1124t , adjusted by transceivers 1122a through 1122t , demodulated by a demodulator ("DEMOD") 1140 , and processed by RX data processor 1142 . Processing to extract the reverse link message sent by device 1150 . Processor 1130 then determines which precoding matrix to use for determining beamforming weights and then processes the extracted information.

Figure 11 also illustrates that the communication component can include one or more components for effecting image data transmission in the presence of interference. For example, interference ("INTER.") control element 1190 can coordinate with processor 1130 and/or other elements of device 1110 to transmit/receive signals to/from another device (eg, device 1150 ). Similarly, interference control component 1192 can coordinate with processor 1170 and/or other components of device 1150 to transmit/receive signals to/from another device (e.g., device 1110 ). It will be appreciated that for each device 1110 and 1150 , the functionality of two or more of the above elements may be provided by a single element. For example, a single processing element can provide the functionality of interference control element 1190 and processor 1130 , and a single processing element can provide the functionality of interference control element 1192 and processor 1170 .

Referring to FIG. 12 , an exemplary computing environment 1200 for implementing various aspects of the claimed subject matter includes a computer 1212 . Computer 1212 includes processing unit 1214 , system memory 1216, and system bus 1218 . System bus 1218 couples a plurality of system components to processing unit 1214 , such as but not limited to system memory 1216 . Processing unit 1214 can be any of a variety of available processors. Dual microprocessors and other multiprocessor architectures can also be used as the processing unit 1214 .

The system bus 1218 can be any of several types of bus bar structures including a memory bus or memory controller, a peripheral bus or an external bus and/or a local bus, and Use any type of available bus architecture, including but not limited to: Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Electronic Drive (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics (AGP), Personal Computer Memory Card International Association Bus (PCMCIA), FireWire (IEEE 1294), and Smart PC System Interface (SCSI).

System memory 1216 includes volatile memory 1220 and non-volatile memory 1222 . A basic input/output system (BIOS) is stored in non-volatile memory 1222 that contains the basic routine for communicating information between elements of computer 1212 , such as in a boot process. By way of example and not limitation, non-volatile memory 1222 may comprise read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electronic erasable programmable ROM (EEPROM) ) or flash memory. Volatile memory 1220 includes random access memory (RAM) that acts as an external buffer memory. By way of example and not limitation, RAM can take many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink. DRAM (SLDRAM), Rambus Direct RAM (RDRAM), Direct Rambus Dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM).

Computer 1212 also includes removable/non-removable, volatile/non-volatile computer storage media. For example, Figure 12 illustrates a disk storage device 1224 . Disk storage device 1224 includes, but is not limited to, a device such as a disk drive, a floppy disk drive, a tape drive, a Jaz drive, a compact disk, an LS-100 drive, a flash memory card, or a memory stick. In addition, the disk storage device 1224 may include a separate storage medium or a storage medium combined with other storage media, including but not limited to: a compact disk drive such as a compact disk ROM drive (CD-ROM), a CD recordable drive (CD-R drive) ), CD rewritable drive (CD-RW drive) or digital versatile disc ROM drive (DVD-ROM). To implement the connection of disk storage device 1224 to system bus 1218 , a removable or non-removable interface, such as interface 1226 , is typically used.

It will be appreciated that FIG. 12 depicts acting as an intermediary software between a user and a basic computer resource, which is described in a suitable operating environment 1200 . This type of software includes an operating system 1228 . OS 1228 may be stored in the disk storage device 1224, operating system 1228 to control and allocate resources of the computer system 1212. The system application 1230 utilizes the operating system 1228 to manage resources via the program module 1232 and program data 1234 stored in the system memory 1216 or disk storage device 1224 . It will be appreciated that a variety of operating systems or combinations of operating systems may be employed to implement the claimed subject matter.

The user enters commands or information into the computer 1212 via the input device 1236 . Input device 1236 includes, but is not limited to, pointing devices such as a mouse, trackball, stylus, trackpad, keyboard, microphone, joystick, gamepad, satellite dish, scanner, TV tuner card, digital Cameras, digital cameras, web cameras, and more. Such and other input devices 1238 via interface port connected via a system bus 1218 to the processing unit 1214. The interface 埠1238 includes, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device 1240 uses some of the same types of ports as input device 1236 . Thus, for example, USB 埠 can be used to provide input to computer 1212 and output information from computer 1212 to output device 1240 . Output adapter 1242 is provided to illustrate that there are some output devices 1240 in other output devices 1240 , such as monitors, speakers, and printers, requiring special adapters. By way of example and not limitation, the output adapter 1242 includes a video card and a sound card providing a means of connection between the output device 1240 and the system bus 1218 . It should be noted that other devices and/or device systems provide both input and output capabilities, such as remote computer 1244 .

Computer 1212 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer 1244 . The remote computer 1244 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other general network node, and the like, and typically includes as much as described with respect to the computer 1212. One or all components. For simplicity, only the memory storage device 1246 is illustrated for the remote computer 1244 . The remote computer 1244 is logically connected to the computer 1212 via the network interface 1248 and then physically connected via the communication connection 1250 . The network interface 1248 covers wired and/or wireless communication networks, such as regional networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Scepter Ring Network, and more. WAN technologies include, but are not limited to, point-to-point connections, circuit-switched networks such as Integrated Services Digital Network (ISDN) and variants, packet switched networks, and Digital Subscriber Line (DSL).

Communication connection 1250 refers to the hardware/software used to connect network interface 1248 to busbar 1218 . Although the communication connection 1250 is shown as being internal to the computer 1212 for clarity of illustration, the communication connection may also be external to the computer 1212 . The hardware/software necessary for the connection to the network interface 1248 includes (for example purposes only) internal and external technologies, such as data machines including general telephone-grade data machines, cable data machines, and DSL modems, ISDN. Adapter and Ethernet card.

Variations, modifications, and other embodiments of the subject matter described herein will be apparent to those skilled in the <RTIgt; Accordingly, the contents of this disclosure are not intended to be defined by the above-described exemplary description, but rather by the spirit and scope of the appended claims.

It should be apparent that the teachings herein may be embodied in various forms and that any specific structure or function disclosed herein is merely representative. Based on the teachings herein, those skilled in the art will recognize that a certain aspect disclosed herein can be implemented independently of other aspects, and two or more of these can be combined in various ways. For example, an apparatus or implementation may be implemented using any number of the approaches set forth herein. In addition, other structures or functions may be implemented in addition to or in place of one or more of the aspects set forth herein. As an example, a plurality of methods, apparatus, systems, and apparatuses described herein are described in the context of providing dynamic queries and recommendations in a mobile communication environment. Those skilled in the art will recognize that similar techniques can be applied to other communication and non-communication environments as well.

As used in the context of this case, the terms "content" and "object" are used to describe any type of application, multimedia file, image file, executable file, program, web page, script, file, submission, message, material, medium Any other type of media or information following the material or that can be rendered, processed or executed on the device.

As used in the context of this case, the terms "component", "system", "module" and the like are intended to mean a computer-related entity, or a hardware, software, executing software, firmware, mediation software, or The combinations are arbitrarily combined. For example, an element can be, but is not limited to, a program running on a processor, a processor, an object, an executable, a thread of execution, a program, or a computer. One or more components can reside in a executing program and/or a executing thread, and the components can be located in a computer or distributed across two or more computers. Moreover, the elements can be executed from a variety of computer readable media having various data structures stored thereon. The elements can communicate via the local and/or remote program, for example, based on a signal having one or more data packets (eg, data from a component that is signaled to the local system, the distributed system Another component in the interaction interacts with other systems via a network such as the Internet. In addition, as will be recognized by those skilled in the art, the elements of the systems described herein may be rearranged or supplemented by additional elements to facilitate the various aspects, objectives, and advantages described in connection with the elements. And so on, and is not limited to the precise configuration provided in the given figures.

In addition, general purpose processors, digital signal processors (DSPs), special application integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices designed to perform the functions described herein can be used. The various gates, or transistor logic devices, individual hardware components, or any combination of the above, are used to implement or perform the various exemplary logic, logic blocks, modules and circuits described in connection with the aspects disclosed herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any general processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such structure. Additionally, at least one processor can include one or more modules operable to perform one or more of the steps and/or operations described above.

In addition, the various aspects or structures described herein may be implemented as a method, apparatus, or article of manufacture using standard programming or engineering techniques. Furthermore, the operations or actions of the methods or algorithms described in connection with the aspects disclosed herein may be embodied in a hardware, a software module executed by a processor, or a combination of both. In addition, in some aspects, the operations or actions of a method or algorithm may be located on a machine readable medium or computer readable medium as at least one or any combination or combination of code or instructions, machine readable medium or computer Readable media can be incorporated into computer program products. Moreover, the term "article of manufacture" as used herein is intended to include a computer program that can be accessed from any computer readable device, carrier, or media. For example, computer readable media can include, but is not limited to, magnetic storage devices (eg, hard drives, floppy disks, tapes, etc.), optical disks (eg, compact discs (CDs), digital versatile discs (DVD), etc.) , smart cards and flash memory devices (eg cards, sticks, keyed disks, etc.). Moreover, various storage media described herein can represent one or more devices or other machine readable media for storing information. The term "machine readable medium" may include, but is not limited to, a wireless channel and various other media capable of storing, containing or carrying instructions or material.

In addition, various aspects are described herein in connection with mobile devices. Mobile devices can also be referred to as systems, subscriber units, subscriber stations, mobile stations, mobile devices, mobile devices, cellular devices, multi-mode devices, remote stations, remote terminals, access terminals, user terminals, users. Agent, user device or user device, etc. The subscriber station can be a cellular telephone, a wireless telephone, a Session Initiation Protocol (SIP) telephone, a Wireless Area Loop (WLL) station, a Personal Digital Assistant (PDA), a wirelessly connected handheld device, or connected to a wireless data modem or Other processing devices that implement similar mechanisms for wireless communication with processing devices.

In addition to the above, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or preferred. Instead, the use of the word "exemplary" is intended to provide a concept in a specific form. In addition, as used in this and the accompanying claims, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, "X uses A or B" is intended to mean any of the natural inclusive permutations unless otherwise specified or apparent from the context. That is, in this example, X can use A, X can use B, or X can use both A and B, and thus the statement "X uses A or B" is satisfied in any of the above examples. In addition, the article "a" or "an" or "an" or "an"

As used herein, the term "inference" or "inference" generally refers to a procedure for inferring or inferring the state of a system, environment, or user from a set of observations obtained through an event or material. Inference can be used to identify a particular context or action, or inference can produce, for example, a probability distribution of states. The inference can be based on probability - that is, the probability distribution of the state of interest is calculated based on considerations of data and events. Inference can also represent techniques for combining high-level events based on a set of events or materials. Such inferences result in new events or actions being formed from a set of observed events and/or stored event data, regardless of whether the events are closely related in time and regardless of whether the event and data are from one or several events and data. source.

100. . . Device

102. . . Mobile device

104. . . Visual media content

106. . . monitor

108. . . user

110. . . camera

112. . . Digital image

114. . . Computing platform

116. . . quadrilateral

118. . . Remote server

120. . . database

122. . . Media content file

124. . . Identification information

126. . . Media content

128. . . Air passage

130. . . user interface

200. . . Method for identifying visual media content

202. . . Receiving images from the camera of the mobile device

204. . . Detecting the quadrilateral contained in the image

206. . . Capturing visual media content contained within the quad to identify the visual media content

300. . . Method for capturing and recognizing visual media content in a detected external display imaged by a camera

302. . . Point the camera of the wireless mobile device

304. . . User selects control for capturing image content

306. . . Decide on the intent/appropriateness of the text

308. . . B/W-no motion

310. . . Decide on the intent/appropriateness of the image capture

312. . . Color - no movement

314. . . Decide on the intent/appropriateness of video capture

316. . . Color-sports

317. . . Convert image capture output in RGB format to YCbCr and process it for color and motion information

318. . . View quadrilateral images/video sources in a low resolution camera preview program

320. . . The user clicks on the viewfinder to assist

322. . . Display candidate sources for users to choose

324. . . Receive user selection/decision image/video source

326. . . User assisted by alignment/focus/click

328. . . Send a region of interest (ROI) to the camera for optimal resolution settings

330. . . Extract one (or more) frames from the area

332. . . Identifying the media content of one (or more) frames captured

334. . . Download media content

336. . . VGA resolution on "N" frames (n=1 means no motion)

338. . . Create an ROI graph with 255 values, which can be blurred due to a decrease in value when clicking on user input

340. . . Establish a "bright" ROI map by thresholding the maximum value (RGB) for values >x (for example, achieving 20% of the remaining pixels)

342. . . Create a "moving" ROI map via the difference between each frame and the previous "M" frame (eg, M=3), and record the maximum △ to indicate the removal of signal interference.

344. . . Crop the image from the viewfinder based on the weight of the ROI map

346. . . Feed fast corner detector

348. . . Contrast threshold primitives ( 6) Closer corner points (CP) for clustering

350. . . If the CP is in the complete N×N area in the bright map, delete the CP

352. . . If the motion in the M×M area is completely in the bright picture, delete the CP

354. . . Identify candidates for quadrilateral from the deleted CP

356. . . No convex surface (the total angle is 360°)

358. . . Any internal angle >110°

360. . . Video aspect ratio (4:3, 16:9)

362. . . Area ≧ image 1/25

364. . . 2 equal neighbors

365. . . Candidates for associating quadrilaterals based on depth inspection

366. . . Add 4 corners of the candidate to the main list

368. . . Remove any quads that occupy 80% of the other quad

370. . . Remove corners based on detection of the shape of the border and uniform color that characterizes the monitor/display

371. . . Decision ability

372. . . Test the bandwidth of the connection to the server

374. . . Determine device performance constraints

376. . . Access to user preferences or the cost of using bandwidth

378. . . Review the data table (LUT) to determine the image processing that the device is currently assigning to the server for media content recognition.

380. . . Local processing mode low connection device selection with digital signal processing capability

382. . . N×N based bar graph + edge detection & size invariant feature conversion

384. . . Send a feature vector to the server

386. . . Shared processing mode medium connection local and remote DSP selection

388. . . N×N based bar graph + edge detection

390. . . Send results to the server

392. . . Remote processing mode for fast connection to DSP-free device selection

394. . . Send the selected clip to the server

396. . . Receive a candidate list of media content matches

397. . . No match: grab more media content

398. . . Limited match (for example, 3): prompt the user to make a selection

399. . . A large number of matches: upload more clips

400. . . Method for performing image analysis using a communication device or device having a camera function

402. . . Save frames from low resolution mode of the camera viewfinder

404. . . High resolution snapshot or snapshot sequence selected automatically or manually

406. . . Extracting the display of the portion of the viewfinder image corresponding to the external display/monitor that is presenting the media content

408. . . Select the correct part of the full local, decentralized local and remote or all remote execution

410. . . Harris corner detector examines all corners in the ROI

412. . . Align all the corners of the view and combine until a set is created

414. . . (1) The internal content has an average brightness that is much higher than the average brightness of the overall image.

416. . . (2) The lines connecting the four corners are uniform and have substantially the same hue in the HSV space.

418. . . (3) Delete the area where the perspective of the 4 points is higher than the height of the eye

420. . . In the case where an accurate inflection point is found, optionally, if the perspective is greater than a certain threshold, the perspective is corrected

422. . . Create a bar graph of an image in a rectangular or square N×N grid

424. . . Calculate the number of primitives in each region by hard or soft edges

426. . . Combine N2 RGB values and hard edge values + soft edge values into the data payload and send

428. . . Media content is recognized and formatted for continued presentation on mobile devices

430. . . Explain the subscription of data rights

500. . . image

502. . . monitor

600. . . image

602. . . monitor

604. . . corner

606. . . Media content

700. . . Primitive collection

702. . . External point

704. . . Internal point

706. . . monitor

708. . . Image point

710. . . Media content

712. . . Irrelevant point

800. . . system

802. . . Logical group

804. . . Electronic component for receiving images from a camera of a mobile device

806. . . Electronic component for detecting quadrilateral contained in an image

808. . . Electronic component for capturing visual media content contained within a quad to identify the visual media content

820. . . Memory

902. . . Device

904. . . Membrane for receiving images from a camera of a mobile device

906. . . A member for detecting a quadrangle contained in an image

908. . . Metric for capturing visual media content contained within a quad to identify the visual media content

920. . . Memory

1000. . . Mobile device

1010. . . Digital signal processor (DSP)

1022. . . device

1026. . . Display controller

1028. . . monitor

1030. . . Input device

1032. . . Memory

1034. . . Transcoder

1036. . . speaker

1038. . . microphone

1040. . . Wireless interface

1042. . . Wireless antenna

1044. . . power supply

1046. . . Image data

1064. . . Bar graph, corner detector and SIFT generator

1068. . . Camera interface

1070. . . Camera

1100. . . MIMO system

1110. . . Wireless device

1112. . . source

1114. . . TX data processor

1120. . . TX MIMO processor

1122a. . . Transceiver

1122t. . . Transceiver

1124A. . . antenna

1124T. . . antenna

1130. . . processor

1132. . . Memory

1136. . . source

1138. . . TX data processor

1140. . . Demodulator

1142. . . RX data processor

1152A. . . antenna

1152R. . . antenna

1154a. . . Transceiver

1154r. . . Transceiver

1160. . . RX data processor

1170. . . processor

1172. . . Memory

1180. . . Data machine

1190. . . Interference control

1192. . . Interference control

1200. . . Computing environment

1212. . . computer

1214. . . Processing unit

1216. . . System memory

1218. . . Busbar

1220. . . Volatile memory

1222. . . Non-volatile memory

1224. . . Disk storage device

1226. . . interface

1228. . . working system

1230. . . application

1232. . . Module

1234. . . data

1236. . . Input device

1238. . . Interface埠

1240. . . Output device

1242. . . Output adapter

1244. . . Remote computer

1246. . . Memory storage device

1248. . . Network interface

1250. . . Communication connection

1 illustrates a schematic diagram of a mobile device that recognizes visual media content presented externally by a display for viewing by a user.

2 illustrates a flow diagram of a method or sequence of operations for identifying visual media content.

3A illustrates a flow diagram of a method or sequence of operations for capturing and recognizing visual media content within a detected external display imaged by a camera.

3B illustrates a flow diagram of a method or sequence of operations for viewing a quadrilateral image/video source within an image.

3C illustrates a flow diagram of a method or sequence of operations for identifying media content of a captured frame.

4 illustrates an exemplary flow chart of a method or sequence of operations for performing image analysis using a communication device having camera functionality.

Figure 5 illustrates a graphical illustration of a schematic low resolution image of a room containing a display viewed from the side.

Figure 6 illustrates a graphical illustration of an illustrative low resolution image of a region of interest (ROI) portion of a room with a plurality of detected corners as candidates for automatically defining the display.

Figure 7 illustrates a graphical illustration of a set of candidate clustered and deleted primitives that are exported from an image analysis.

Figure 8 illustrates a schematic diagram of a system for identifying visual media content.

Figure 9 illustrates a schematic diagram of an apparatus having means for identifying visual media content.

10 is a block diagram of a mobile device including a bar graph, a corner detector, and a scale invariant feature transform (SIFT) generator.

11 illustrates a schematic diagram of communication elements that may include one or more components that perform interference control operations.

Figure 12 illustrates a schematic diagram of an exemplary computing environment.

200. . . Method for identifying visual media content

202. . . Receiving images from the camera of the mobile device

204. . . Detecting the quadrilateral contained in the image

206. . . Capturing visual media content contained within the quad to identify the visual media content

Claims (79)

A method for recognizing visual media content, comprising: receiving an image from a camera of a mobile device; detecting a quadrilateral included in the image; and capturing visual media content included in the quadrilateral for identification The visual media content. The method of claim 1, wherein extracting the visual media content included in the quadrilateral to identify the visual media content further comprises: performing character recognition. The method of claim 1, wherein extracting the visual media content contained within the quadrilateral to identify the visual media content further comprises performing image recognition for static visual media content. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: establishing a region of interest map based on the object contrast, the region of interest map identifying important details in the image. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: receiving a user input with respect to a portion of a viewfinder scene of the image. The method of claim 1, wherein receiving the image from the camera of the mobile device further comprises: receiving a plurality of consecutive frames, and wherein the visual media content included in the quadrilateral is captured to identify the visual The media content further includes: performing video image recognition for the dynamic visual media content. The method of claim 6, wherein detecting the quadrilateral included in the image further comprises: establishing a motion map by determining a difference between the consecutive plurality of frames. The method of claim 7, further comprising: performing corner detection; and deleting a corner point in the motion map. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: creating a region of interest map; and cropping the image to include the region of interest map. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: establishing a bright map by detecting a portion having a brighter illumination. The method of claim 10, wherein detecting the quadrilateral included in the image further comprises: performing corner detection; clustering the corner points; and deleting a cluster of corner points within the bright map. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: establishing a depth map by detecting a depth of focus of the plurality of portions in the image. The method of claim 1, wherein detecting the quadrilateral included in the image further comprises: detecting a candidate quadrilateral shape satisfying an identification criterion of the selected four corner point clusters to identify a rectangular display A perspective view of the device. The method of claim 13, wherein the recognizing the perspective view of the rectangular display device further comprises: deleting any candidate quadrilateral shape occupying a majority of the area of the other quadrilateral by determining whether the quadrilateral shape is large enough to be Contains all other candidate quadrilateral shapes. The method of claim 13, wherein the identifying the perspective view of the rectangular display device further comprises: identifying a border of the rectangular display device. The method of claim 15, wherein the identifying the frame of the rectangular display device further comprises: detecting a common frame thickness of the portion of the frame on the opposite side. The method of claim 16, wherein detecting the common bezel thickness of the portion of the bezel on the opposite side comprises detecting a percentage of a thickness of the enclosing dimension of the rectangular display device, for example, about 10%. The method of claim 15, wherein the identifying the frame of the rectangular display device further comprises: detecting a common color of a common main portion of the frame on opposite sides. The method of claim 18, wherein detecting the common color of a common major portion of the frame on opposite sides further comprises detecting at least a percentage of the plurality of primitives having the common color. The method of claim 13, wherein the recognizing the perspective view of the rectangular display device further comprises: determining that the candidate quadrilateral shape satisfies more than one criterion, the criterion comprising non-convex surfaces, all of the internal angles being greater than 110 degrees And having an area occupying a major portion of the image, having an aspect ratio approximately equal to a standard video aspect ratio, and two adjacent angles having approximately the same angle. The method of claim 1, further comprising: performing a bar graph analysis, edge detection, and scale invariant feature conversion on a portion of the image within the selected quadrilateral to identify the corresponding media content. . The method of claim 1, further comprising: determining an image processing constraint; and, in response to the image processing constraint, allocating a portion of the image within the quadrilateral between the mobile device and a remote server Image processing. The method of claim 22, wherein the image processing constraint comprises: the ability of the mobile device to perform the image processing. The method of claim 22, wherein the image processing constraint comprises, at least in part, a data transmission cost transmitted via a transmission channel from the mobile device to the remote processing. The method of claim 22, wherein the image processing constraint comprises: a capability from the mobile device to a transmission channel of the remote processing. The method of claim 25, wherein the allocating the image processing of the portion of the image comprises: transmitting the image data comprising an image segment in response to determining a low capability of the transmission channel; in response to the determining a medium capability of the transmission channel to transmit the image data including the image segment after partial image processing; and in response to determining a high capability of the transmission channel, transmitting the image segment including the image segment after all image processing Image data. The method of claim 1, further comprising: transmitting, to a remote server, image data obtained from a portion of the image within the quadrilateral; and receiving information about the image from the remote server A matching report of any media content library. The method of claim 27, wherein receiving the report regarding the match from the remote server further comprises: determining that no match is recognized; and receiving an image from a camera of a mobile device, detecting A quadrilateral included in the image and a visual media content contained within the quadrilateral to identify the visual media content for repetition to obtain additional image material to be sent to the remote server. The method of claim 27, wherein receiving the report regarding the match from the remote server further comprises: determining that the number of matches obtained from the report is suitable for presentation on a user interface of the mobile device Size; and receiving a user selection for a media content in the list of media content derived from the report and presented on the user interface. The method of claim 27, wherein receiving the report regarding the match from the remote server further comprises: determining that the number of matches obtained from the report has a ratio greater than that suitable for presentation on a user interface of the mobile device One size larger than one size; and an image segment is sent in response to the remote server for additional image processing. The method of claim 1, further comprising: receiving the matched media content for presentation by the mobile device. The method of claim 31, further comprising: presenting the matched media content from a point identified by the captured visual media content. The method of claim 31, further comprising: reformatting the matched media content for a user of the mobile device. The method of claim 31, further comprising: transmitting an identifier of a user interface of the mobile device to prompt reformatting the matched media content. The method of claim 31, further comprising: negotiating data management rights to present the matched media content. The method of claim 1, further comprising: receiving audio captured by a microphone; and using a hash view function to assist in matching the image material with a library of media content. A processor for recognizing visual media content, comprising: a first module for receiving an image from a camera of a mobile device; and a second module for detecting inclusion in the image a quadrilateral; and a third module for capturing a visual media content included in the quadrilateral to identify the visual media content. A computer program product for recognizing visual media content, comprising: a non-transitory computer readable storage medium, comprising: a first set of codes for causing a computer to receive an image from a camera of a mobile device; a second set of codes for causing the computer to detect a quadrilateral included in the image; and a first program for causing the computer to capture a visual media content contained in the quadrilateral to identify the visual media content Three sets of code. An apparatus for recognizing visual media content, comprising: a component for receiving an image from a camera of a mobile device; a component for detecting a quadrilateral included in the image; and A visual media content contained within the quadrilateral is taken to identify a component of the visual media content. An apparatus for recognizing visual media content, comprising: a camera of a mobile device for generating an image; and a computing platform for detecting a quadrilateral included in the image received from the camera, And for capturing a visual media content included in the quadrilateral to identify the visual media content. The device as claimed in claim 40, wherein the computing platform is further configured to: capture the visual media content included in the quadrilateral to identify the visual media content by performing character recognition. The device as recited in claim 40, wherein the computing platform is further for: capturing the visual media content contained within the quadrilateral to identify the visual media content by performing image recognition for the static visual media content. The device as claimed in claim 40, wherein the computing platform is further configured to: detect the quadrilateral included in the image by establishing a region of interest map based on the object contrast, wherein the region of interest map is identified in the image Important details in the picture. The apparatus of claim 40, wherein the computing platform is further for detecting the quadrilateral included in the image via a user input that receives a portion of a viewfinder scene relative to the image. The apparatus of claim 40, wherein the computing platform is further configured to: receive the image from the camera of the mobile device by receiving a plurality of consecutive frames, and wherein the image included in the quadrilateral is captured Visual media content to identify the visual media content further includes performing video image recognition for the dynamic visual media content. The device as recited in claim 45, wherein the computing platform is further configured to: detect a motion picture included in the image by establishing a motion map by determining a difference between the plurality of consecutive frames The quadrangle. The apparatus of claim 46, wherein the computing platform is further configured to: perform corner detection; and delete a corner point within the motion map. The apparatus of claim 40, wherein the computing platform is further configured to detect the quadrilateral included in the image by establishing a region of interest map and cropping the image to include the region of interest map. The apparatus of claim 40, wherein the computing platform is further configured to detect the quadrilateral included in the image by establishing a bright map by detecting a portion having a brighter illumination. The apparatus of claim 49, wherein the computing platform is further configured to detect the quadrilateral included in the image by: performing corner detection; clustering corner points; and deleting in the bright picture A corner point cluster. The device of claim 40, wherein the computing platform is further configured to: detect the inclusion in the image by establishing a depth map by detecting a depth of focus of the plurality of portions in the image quadrilateral. The device as recited in claim 40, wherein the computing platform is further configured to: identify a candidate quadrilateral shape satisfying an identification criterion by detecting the selected four corner point clusters to identify a perspective view of a rectangular display device The quadrilateral contained in the image is detected. The apparatus of claim 52, wherein the computing platform is further for identifying the perspective view of the rectangular display device by deleting any candidate quadrilateral shape occupying a majority of the area of the other quadrilateral by: Whether the quadrilateral shape is large enough to include all other candidate quadrilateral shapes. The device as claimed in claim 52, wherein the computing platform is further configured to recognize the perspective view of the rectangular display device by recognizing a border of the rectangular display device. The device of claim 54, wherein the computing platform is further configured to: identify the border of the rectangular display device by detecting a common border thickness of the portion of the bezel on the opposite side. The device as recited in claim 55, wherein the computing platform is further configured to detect the border on the opposite side by detecting a percentage of a size of the enclosure of the rectangular display device, for example, about 10% Part of the common border thickness. The apparatus of claim 54, wherein the computing platform is further configured to: identify the border of the rectangular display device by detecting a common color of a common main portion of the bezel on opposite sides. The device of claim 57, wherein the computing platform is further configured to detect a common portion of the frame on the opposite side by detecting that at least a percentage of the plurality of primitives have the common color The common color. The apparatus of claim 52, wherein the computing platform is further configured to: identify the perspective view of the rectangular display device by determining that the candidate quadrilateral shape satisfies more than one criterion, the standard including non-convex surfaces, all of which are included The interior angles are all greater than 110 degrees, have a region that occupies a major portion of the image, have an aspect ratio that is approximately equal to a standard video aspect ratio, and have two adjacent angles that are approximately the same angle. The apparatus of claim 40, wherein the computing platform is further configured to: perform a bar graph analysis, edge detection, and scale invariant feature conversion on a portion of the image within the selected quadrilateral to Identify the corresponding media content. The apparatus of claim 40, wherein the computing platform is further configured to: determine an image processing constraint, and to allocate the image between the mobile device and the remote server in response to the image processing constraint Image processing of the portion within the quadrilateral. The apparatus of claim 61, wherein the image processing constraint comprises: an ability of the mobile device to perform the image processing. The apparatus of claim 61, wherein the image processing constraint comprises, at least in part, a data transmission cost transmitted via a transmission channel from the mobile device to the remote processing. The apparatus of claim 61, wherein the image processing constraint comprises: a capability from the mobile device to a transmission channel of the remote processing. The apparatus of claim 64, wherein the transmitter is further configured to: in response to determining a low capability of the transmission channel, transmitting the image data comprising an image segment; in response to determining a medium capability of the transmission channel Transmitting the image data including the image segment after partial image processing; and in response to determining a high capability of the transmission channel, transmitting the image data including the image segment after all image processing. The device as recited in claim 40, further comprising: a transmitter for transmitting image data obtained from a portion of the image within the quadrilateral to a remote server; and a receiver for The remote server receives a report of any match of the image material to a library of media content. The apparatus of claim 66, wherein the computing platform is further configured to receive, from the remote server, the report regarding any match via the following: the decision not identifying the match; and the pairing from a mobile device The camera receives an image, detects a quadrilateral included in the image, and captures a visual media content contained in the quad to identify the visual media content for repetition to obtain a message to be sent to the remote server. Additional image data. The apparatus of claim 66, wherein the computing platform is further for receiving, by the receiver, the report from the remote server for any match via the following: determining that the number of matches obtained from the report is suitable for the a size presented by a user interface of the mobile device; and receiving a user selection for a media content in the list of media content derived from the report and presented on the user interface. The apparatus of claim 67, wherein the computing platform is further for receiving, by the receiver, the receiver from the remote server for any matching of the report: determining that the number of matches obtained from the report has a ratio suitable for A larger size of a size presented by a user interface of the mobile device; and an image segment is sent in response to the remote server for additional image processing. The device as recited in claim 60, wherein the receiver is further configured to: receive the matched media content for presentation by the mobile device. The apparatus of claim 70, wherein the computing platform is further for presenting the matched media content from a point identified by the captured visual media content. The device as claimed in claim 70, wherein the computing platform is further configured to: reformat the matched media content for a user interface of the mobile device. The device of claim 70, wherein the transmitter is further configured to: send an identifier of a user interface of the mobile device to prompt reformatting the matched media content. The apparatus of claim 70, wherein the computing platform is further configured to: negotiate data management rights with the receiver via the transmitter to present the matched media content. The device as claimed in claim 60, further comprising: a microphone for capturing audio, wherein the computing platform is further configured to: use a hash view function to assist in matching the image material with a media content library. A method comprising: capturing a digital image using a camera of a mobile communication device; determining at least one of the mobile communication device, an empty intermediary from the mobile communication device to a remote network, and a network server a capability constraint; based on the capability constraint, allocating image processing of the digital image between the mobile communication device and the network server; and receiving a result of image recognition. The method of claim 76, wherein determining the capability constraint further comprises: determining a data transmission capability of the null intermediary. The method of claim 76, wherein determining the capability constraint further comprises determining an image processing capability of the mobile communication device. The method of claim 76, wherein the distributing the image processing of the digital image between the mobile communication device and the network server based on the capability constraint further comprises: selecting local processing, sharing processing, and remote processing One.