WO2018132145A1

WO2018132145A1 - Using machine learning to detect a constituent image within a composite image

Info

Publication number: WO2018132145A1
Application number: PCT/US2017/057668
Authority: WO
Inventors: Filip Pavetic; King Hong Thomas Leung; Dmitrii TOCHILKIN
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2017-01-13
Filing date: 2017-10-20
Publication date: 2018-07-19
Anticipated expiration: 2019-07-13
Also published as: GB201717849D0; US20180204065A1; US20200210709A1; CN108305299A; GB2558721B; DE102017125463A1; US11829854B2; US20210374418A1; GB2558721A; US10586111B2; US12217142B2; US11093751B2; US20240104435A1

Abstract

A system and methods are disclosed for training a machine learning model to identify constituent images within composite images. In one implementation, a composite image is generated, where the composite image comprises a first portion containing pixel data of a first constituent image, and a second portion containing pixel data of a second constituent image. A first training input comprising pixel data of the composite image and a first target output for the first training input are generated, where the first target output identifies a position of the first portion within the composite image. The training data is provided to train the machine learning model on (i) a set of training inputs comprising the first training input and (ii) a set of target outputs comprising the first target output.

Description

USING MACHINE LEARNING TO DETECT A CONSTITUENT IMAGE WITHIN A

COMPOSITE IMAGE

TECHNICAL FIELD

[001] Aspects and implementations of the present disclosure relate to image processing, and more specifically, to detecti ng a constituent i mage within a composite image, for example to detecting which part of a screen includes embedded frames of an uploaded video.

BACKGROUND

[002] Content sharing platforms enable users to upload, consume, search for, approve of ("like"), dislike, and/or comment on content such as videos, images, audio clips, news stories, etc. Images typical ly compri se a plurality of picture elements (e.g., pixels) organized in a particular arrangement, such as a two-dimensional array, and videos typically comprise sequences of images, referred to as frames.

SUMMARY

[003] The following presents a simpli fied summary of various aspects of thi s di sclosure in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contempl ated aspects, and is intended to neither identify key or critical elements nor delineate the scope of such aspects. Its purpose is to present some concepts of this disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[004] In an aspect of the present disclosure, a system and methods are disclosed for training a machine learning model (e.g., a neural network, a support vector machine [SVM], etc. ) and using the trained model to process images. In one implementation, a method for training a machine learning model to identify constituent images within composite images comprises generating training data for the machine learning model, wherein generating the training data comprises: generating a composite image compri sing a first portion containing pixel data of a first constituent image, and a second portion containing pixel data of a second constituent image, generating a first training input comprising pi xel data of the composite image, and generating a first target output for the first training input, where the first target output identifies a position of the first portion w ithin the composite image. The method further comprises providing the training data to train the machine learning model on (i) a set of training inputs compri sing the first training input, and (ii) a set of target outputs comprising the first target output.

[005] In some implementations, the second portion of the composite image surrounds the first portion of the composite image. In some implementations, the first constituent image is a frame of a first video and the second constituent image is a frame of a second video. In some implementations, the position of the first portion within the composite image comprises coordinates of an upper left corner of the first constituent image and coordinates of a lower right corner of the first constituent image. In some implementations, each training input in the set of training inputs is mapped to a target output in the set of target outputs. In some implementations, the machine learning model is configured to process a new image and generate one or more outputs indicating (i) a level of confidence that the new image is a composite image including a constituent image, and (ii) a spatial area in which the constituent image is located within the new image.

[006] In an aspect of the present disclosure, a method comprises receiving an input image, processing the input image using a trained machine learning model, and obtaining, based on the processing of the input image using the trained machine learning model, one or more outputs indicating (i) a level of confidence that the input image is a composite image including a constituent image, and (ii) a spatial area that includes the constituent image ithin the input image. In some implementations, the method further comprises determining that the level of confidence satisfies a threshold condition, and extracting the constituent image from the spatial area within the input image. (The machine learning model may have been trained according to the aspect described above. ) In further implementations the extracted constituent image may be used in an image comparison technique or in a content detection technique, for example by creating a fingerprint of the extracted constituent image, and comparing the created fingerpri nt with a fingerprint of at least one other image. In one example, comparing the created fingerprint with a fingerprint of at least one other image may comprise comparing the created fingerprint with stored fingerprints of a pl urality of images to determine whether the constituent image is similar to any of the plurality of images

[007] In some implementations, the input image comprises a second constituent image that surrounds the constituent image. In some implementations, the constituent image is a frame of a video. In some implementations, the first spatial area is one of a plurality of spatial areas of the input image. In some implementations, the first spatial area is one of a plurality of spatial areas of the input image, and a union of the plurality of spatial areas contains all pixels of the input image. In some implementations, the first spatial area i s one of a plurality of spatial areas of the input image, and the plurality of spatial areas are uniform in size and shape. In some implementations, the spatial area is one of a plurality of spatial areas of the input image, and at least two of the plurality of spatial areas have different sizes. In some implementations, the spatial area is one of a plurality of spatial areas of the input image, and at least two of the plurality of spatial areas have different shapes. In some implementations, the first spatial area is one of a plurality of spatial areas of the input image, and at least two of the plurality of spatial areas overlap. In some implementations, the first spatial area i s one of a plurality of spatial areas of the input image, and the plurality of spatial areas are non- overlapping.

[008] In an aspect of the present disclosure, a method comprises receiving an input image, and processing the input image using a trained model that is configured to generate, based on pixel data of the input image, one or more outputs indicating (i) a level of confidence that the input image is a composite image including a constituent image, and (ii) a spatial area that includes the constituent image within the input image. (Again, the model may have been trained according to the aspect described above. ) In some implementations, the method further compri es determining that the level of confidence satisfies a threshold condition, and extracting the constituent image from the spatial area within the input image. In further implementations the extracted constituent image may be used in an image comparison technique or in a content detection technique, for example by creating a fingerprint of the constituent image, and comparing the created fingerprint with a fingerprint of at least one other image. In one example, comparing the created fingerprint ith a fingerprint of at least one other image may comprise comparing the created fingerprint with stored fingerprints of a plurality of images to determine whether the constituent image is similar to any of the plurality of images.

[009] In an aspect of the present di sclosure, an apparatus comprises a memory to store a first (input) image, and a processing device, operatively coupled to the memory, to provide pixel data of a first image as input to a trained machine learning model, obtain one or more outputs from the trained machine learning model, and extract, from the one or more outputs, a level of confidence that the first image is a composite image including a constituent image, and at least a portion of the constituent image is in a particular spati al area of the first image. In some implementations, the processing device is further to determine that the level of confidence sati sfies a threshold condition, and extract the constituent image from the particular spatial of the first image. [0010] In some implementations, the first spatial area is one of a plurality of spatial areas of the first image. In some implementations, the first spatial area is one of a plurality of spatial areas of the first image, and the plurality of spatial areas are uniform in size and shape. In some implementations, the first spatial area is one of a plurality of spatial areas of the first image, and wherein at least two of the plurality of spatial areas have different sizes. In some implementations, the first spatial area is one of a plurality of spatial areas of the first image, and at least two of the plurality of spatial areas overlap. In some implementations, the first spatial area is one of a plurality of spatial areas of the first image, and the plurality of spatial areas are non-overlapping. In some implementations, the first spatial area is one of a plurality of spatial areas of the first image, and at least two of the plurality of spatial areas have different shapes. In some implementations, the spatial area is one of a plurality of spatial areas of the first image, and a union of the plurality of spatial areas contains all pixels of the first image. In some implementations, the trained machine learning model is trained with an input-output mapping compri sing an input and an output, the input based on pixel data of a composite image, the composite image comprising a first portion containing pixel data of a fourth image and a second portion containing pixel data of a fifth i mage, and the output identifying a position of the first portion within the composite image.

[0011] Further, computing devices for performing (or configured to perform ) the operations of any one of the above described methods and the various implementations described herein are disclosed. C om pu ter-readabl e media that store instructions for performing operations that comprise any one of the above described methods and the various implementations described herein are also di sclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects and implementations of the present disclosure will be understood more fully from the detai led description given below and from the accompanying drawings of various aspects and implementations of the disclosure, which, however, should not be taken to limit the di sclosure to the specific aspects or implementations, but are for explanation and understanding onl .

[0013] Figure 1 depicts an il lustrative computer system architecture, in accordance with one or more aspects of the present disclosure.

[0014] Figure 2 depicts a flow diagram of one example of a method for training a machine learning model, in accordance with one or more aspects of the present di sclosure. [0015] Figure 3 depicts an il lustrative composite image, in accordance with one or more aspects of the present disclosure.

[0016] Figure 4 depicts a flow diagram of one example of a method for processing images using a trained machine learning model, in accordance with one or more aspects of the present disclosure.

[0017] Figure 5 depicts a first set of spatial areas of an image, in accordance with one or more aspects of the present di sclosure.

[0018] Figure 6 depicts a second set of spatial areas of an image, in accordance ith one or more aspects of the present disclosure.

[0019] Figure 7 depicts a third set of spatial areas of an image, in accordance with one or more aspects of the present disclosure.

[0020] Figure 8 depicts a fourth set of spatial areas of an image, in accordance with one or more aspects of the present di sclosure.

[0021] Figure 9 depicts a block diagram of an illustrative computer system operating in accordance with one or more aspects of the disclosure.

DETAILED DESCRIPTION

[0022] There are situations in which it is desired to compare two images or image sequences ith one another. While it is in principle possible to compare the two images or image sequences directly this is likely to be computationally expensive and/or slow, and it is therefore usual to create a lower resolution version of an image or image sequence, sometimes referred to a "fingerprint" of the image or image sequence, for use in image comparison. As one example, in a content sharing platform users may upload content (e.g., videos, images, audio clips, etc. ) for inclusion in the platform, thereby enabling other users to consume (e.g., view, etc. ) the content. In some instances, it might be desirable to prohibit certain types of content from inclusion, such as content containing material of another entity, content containing classified material, content containing age-inappropriate material, and so forth. Accordingly, some content sharing platforms employ automated detection

technologies in an attempt to identify and reject such content. The automated detection technologies may create a fingerprint of uploaded content (e.g., by applying a function to frames of a video) to determine whether the created fingerprint is similar to any of the stored fingerprints. [0023] Some users have developed strategies to try to thwart automated detection technologies. For example, users may generate videos in which video frames occupy only a portion of the screen, which can "throw off" feature-based extraction/matching techniques that process the entire screen image. Conventional detection technologies are typically not able to discern that a video frame had been reduced in size and placed within a particular portion of the screen.

[0024] Disclosed herein are aspects and implementations of an automated system that is capable of detecting instances in which an image, such as a video frame, has been reduced in size and embedded within a larger image (e.g., a larger image of a screen or a window on the screen ). More particularly, in accordance ith one implementation, a set of example images is generated and used to train a machine learning model . In particular, each of the example images i s generated by taking two existing images (e.g., two frames of a movie, etc. ) and juxtaposing them to form a single composite image (e.g., adjusting sizes of the first and second images to use the first image as a smaller foreground image placed in a smaller portion of the composite image, and to use the second image as a larger background image placed in a larger portion of the composite image). In Figure 3, for example, the first image is placed within a smaller portion 301-1 of a full -si zed image, and the second image i s placed within a larger portion 30 1 -2 of the full -si zed image.

[0025] During training of the machine learning model, pixel data of a composite image is provided to the machine learning model as input, and one or more positions of the first image w ithin the composite image are provided to the machine learning model as one or more target outputs (e.g., coordinates of the upper left and lower right corners of the first image, etc. ). In thi s way, the machine learning model is trained to identify the position of a constituent image within a composite image.

[0026] After the machine learning model has been trained, a new image (e.g., an image w ith an embedded frame of an uploaded video, etc. ) may be identi ied for processing. In this case, the position of a constituent image (e.g., the embedded frame of the uploaded video) within this new image i s not known because the new image is provided in its entirety to the machine learning model, without any knowledge of how the new image was generated.

[0027] In one implementation, a plurality of spatial areas (e.g., rectangular areas, etc ) within the new image are defined (various examples are shown in Figures 5 through 8, and are discussed in detail below with respect to the method of Figure 4). Pixel data of the new image is presented as input to the trained machine learning model, which generates one or more outputs based on the input. In one implementation, confidence data is then extracted from the output(s) of the trained machine learning model, where the confidence data includes, for each of the plurality of spatial areas, a level of confidence that a constituent image within the new image is located in that particular spatial area. For example, if an image similar to the image 300 of Figure 3 is input to the trained machine learning model, and spatial areas 501-1 through 501 - 12 of Figure 5 are defined as the plurality of spatial areas, then the machine learning model of aspects of the present disclosure may indicate a much higher level of confidence that the embedded image is located in spatial area 501-6, versus being located in spatial area 501-12. Particular aspects concerning the training and usage of the machine learning model will be described in greater detail below.

[0028] Aspects of the present disclosure thus provide a mechanism by which embedded images within full-sized images can be identified and located. This mechanism allows image comparison techniques, and automated content detection techniques (e.g., automated techniques to detect material of another entity, classified material, etc.) to be "focused" on the rel evant portions of full-sized images, rather than operating on the entire images. This increases the reliability of image comparison/content detection techniques, and reduces or eliminates the need for manual checking of the results of such techniques. For example, if a user, in an attempt to circumvent the fingerprinting of a movie, reduces the size of each video frame of the movie and places each resulting video frame within a particular portion of a screen image, aspects of the present disclosure are able to detect the location of the image portion containing the video frames of the movie, and ensure that meaningful fingerprints of the movie are produced. Accordingly, when embodiments of the present disclosure are used in conjunction with automated content detection techniques, the accuracy of these techniques can be improved.

[0029] Figure I illustrates an illustrativ e system architecture 100, in accordance with one implementati on of the present discl osure. The system architecture 100 includes a server machine 120, one or more server machines 130 through 150, a content repository 110 and client machines 102A-102N connected to a network 104. Network 104 may be a public network (e.g., the internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.

[0030] The client machines 102A-102N may be personal computers (PCs), laptops, mobile phones, tablet computers, set top boxes, televisions, video game consoles, digital assistants or any other computing devices. The client machines 102A- 102 may run an operating system ( OS) that manages hardware and software of the client machines 102A- 102N. In one implementation, the client machines 102A-102N may upload videos to the web server for storage and/or processing.

[0031] Server machine 120 may be a rack mount server, a router computer, a personal computer, a portable digital assistant, a mobile phone, a laptop computer, a tablet computer, a camera, a video camera, a netbook, a desktop computer, a media center, or any combination of the above. Server machine 120 includes an upload server 125 that is capable of receiving content (e.g., videos, audio clips, images, etc.) uploaded by client machines 102A-102N (e.g., via a webpage, via an application, etc.).

[0032] Content repository 1 10 is a persistent storage that is capable of storing content items as well as data structures to tag, organize, and index the media items. Content repository 1 10 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage based disks, tapes or hard drives, Network Attached Storage (NAS), Storage Area Network (SAN), and so forth. In some implementations, content repository 1 10 may be a network-attached file server, while in other embodiments content repository 1 10 may be some other type of persistent storage such as an object-oriented database, a relational database, and so forth, that may be hosted by the server machine 120 or one or more different machines coupled to the server machine 120 vi a the network 104.

[0033] The content items stored in the content repository 1 10 may include user-generated media items that are uploaded by client machines, as well as media items from service providers such as news organizations, publishers, libraries and so forth. In some

implementations, content repository 1 10 may be provided by a third-party service, while in some other implementations content repository 1 10 may be maintained by the same entity maintaining serv er machine 20. In some examples, content repository 10 and serv er machine 120 may be part of a content sharing platform that allows users to upload, consume, search for, approv e of ("like^" ), dislike, and/or comment on media items.

[0034] The content sharing platform may include multiple channels. A channel can be data content av ailable from a common source or data content hav ing a common topic, theme, or substance. The data content can be digital content chosen by a user, digital content made av ailable by a user, digital content uploaded by a user, digital content chosen by a content prov ider, di ital content chosen by a broadcaster, etc. A channel can be associated with an owner, who is a user that can perform actions on the channel . Different activities can be associated with the channel based on the owner's actions, such as the owner making di ital content av ailable on the channel, the owner selecting (e.g., liking) digital content associated with another channel, the ow ner commenting on digital content associated with another channel, etc. The activities associated with the channel can be collected into an activity feed for the channel. Users, other than the owner of the channel, can subscribe to one or more channels in which they are interested. The concept of "subscribing" may also be referred to as "liking", "following", "friending", and so on.

[0035] Each channel may include one or more medi a items. Examples of media item s can include, and are not limited to, digital video, digital movies, digital photos, digital music, website content, social media updates, electronic books (ebooks), electronic magazines, digital newspapers, digital audio books, electronic journals, web blogs, real simple syndication (RSS) feeds, electronic comic books, software applications, etc. In some implementations, media items are also referred to as a video content item.

[0036] Media items may be consumed via media viewers executing on client machines 1 Q2A-102N. In one implementation, the media viewers may be applications that allow users to view content, such as images, videos (e.g., video content items), web pages, documents, etc. For example, the media viewers 1 1 2a-z may be a web browser that can access, retrieve, present, and/or navigate content (e.g., web pages such as Hyper Text Markup Language (HTML) pages, digital media items or content items, etc.) served by a web server. The media viewers may render, display, and/or present the content (e.g., a web page, a media viewer) to a user. The media viewers may also display an embedded media player (e.g., a Flash® player or an HT ML 5 player) that is embedded in a web page (e.g., a web page that may provide information about a product sold by an online merchant). In another example, the media viewers may be a standalone application (e.g., a mobile application) that allows users to view digital media content items (e.g., digital videos, digital images, electronic books, etc.).

[0037] The media viewers may be provided to the client devices 1 02 A through 102N by the server 120 and/or content sharing platform. For example, the media viewers may be embedded media players that are embedded in web pages provided by the content sharing platform. In another example, the media viewers may be applications that communicate with the server 1 20 and/or the content sharing platform.

[0038] Serv er machine 130 includes a training set generator 131 that is capable of generating training data (e.g., a set of training inputs and target outputs) to train a machine learning model. Some operations of training set generator 131 are described in detail below with respect to Figure 2.

[0039] Server machine 140 includes a training engine 14 1 that i s capable of training a machine learning model 160. The machine learning model 160 may refer to the model artifact that is created by the training engine 14 1 using the training data that includes training inputs and corresponding target outputs (correct answers for respective training inputs ). The training engine 14 1 may find patterns in the training data that map the training input to the target output (the answer to be predicted ), and provide the machine learning model 160 that captures these patterns. The machine learning model may be composed of, e.g., a single level of linear or non-linear operations (e g., a support vector machine [SVM] or may be a deep network, i .e., a machine learning model that is composed of multiple levels of non-linear operations. An example of a deep network i s a neural network with one or more hidden layers, and such machine learning model may be trained by, for example, adjusting weights of a neural network in accordance with a backpropagation learning algorithm or the like. For convenience, the remainder of this disclosure will refer to the implementation as a neural network, even though some implementations might employ an SVM or other type of learning machine instead of, or in addition to, a neural network. In one aspect, the training set is obtained from server machine 130.

[0040] Serv er machine 150 includes a spatial analysis engine 151 that is capable of providing pixel data of images as input to trained machine learning model 160 and running trained machine learning model 160 on the input to obtain one or more outputs. As described in detail below with respect to Figure 4, in one implementation spatial analysis engine 151 is al so capable of extracting confidence data from the output of the trained machine learning model 160 and using the confidence data to estimate the position of embedded images in input images.

[0041] It should be noted that in some other implementations, the functions of server machines 120, 130, 140, and 1 50 may be prov ided by a fewer number of machines. For example, in some implementations server machines 130 and 140 may be integrated into a single machine, while in some other implementations serv er machines 130, 140, and 1 50 may be integrated into a single machine. In addition, in some implementations one or more of server machines 120, 130, 140, and 1 50 may be integrated into the content sharing platform .

[0042] In general , functions described in one implementation as being performed by the content item sharing platform, serv er machine 120, serv er machine 130, serv er machine 140, and/or server machine 1 50 can also be performed on the client devices 102 A through 102N in other implementations, if appropriate. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together. The content sharing platform, serv er machine 120, server machine 130, serv er machine 140, and/or server machine 1 50 can also be accessed as a serv ice provided to other systems or devices through appropriate application programming interfaces, and thus is not limited to use in websites.

[0043] Figure 2 depicts a flow diagram of one example of a method 200 for training a machine learning model, in accordance with one or more aspects of the present disclosure. The method is performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination thereof. In one implementation, the method is performed by computer system 100 of Figure 1 , while in some other implementations, one or more blocks of Figure 2 may be performed by one or more other machines not depicted in the figures. In some aspects, one or more blocks of Figure 2 may be performed by traini ng set generator 131 of server machine 130.

[0044] For simplicity of explanation, methods are depicted and described as a series of acts. However, acts in accordance with thi s disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all il lustrated acts may be required to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, it should be appreciated that the methods disclosed in this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computing devices. The term article of manufacture, as used herein, is intended to encompass a computer program accessible from any computer- readable device or storage media.

[0045] Method 200 begins with generating training data for a machine learning model . In some implementations, at block 201, a training set T is initialized to an empty set. At block 202, a first image and a second image are selected as constituent images to generate a composite image, a described in detail below with respect to block 203. In one

implementation, the first and second images are randomly-selected video frames (e.g., two different frames from a particular video, a first frame from a first video and a second frame from a second video, etc. ). It should be noted that in some other implementations, the images may be selected in some other manner, and may not be part of a video (e.g., still photographs, etc.).

[0046] At block 203, a composite image is generated based on the first and second constituent images that were selected at block 202. In one implementation, the composite image compri ses two portions, one portion containing pixel data of the first constituent image, and the second portion containing pixel data of the second constituent image. Figure 3 depicts an illustrative composite image 300; as shown in the figure, composite image 300 comprises a first image in a portion 30 1 - 1 and a second image in a portion 30 1 -2. In one example, the composite image may be generated to mimic an attempt by a malicious user to "sneak" an improper image (e.g., a copyrighted image, an age-inappropriate image, etc.) through an automated detection system (e.g., a fingerprint-based detection system).

[0047] It should be noted that the relative sizes of portions may be different than that depicted in Fi ure 3 (e.g., the interior portion might occupy a larger proportion of the composite image than in Figure 3, such as 40% of the composite image, 70% of the composite image, etc. ). Similarly, the spatial arrangement of portions may be different than that depicted in Figure 3 (e.g., side-by-side horizontally, side-by-side vertical ly, etc.).

[0048] It should further be noted that in the case of one portion being surrounded by another portion, as in Figure 3, the composite image mi ht be generated by superimposing a ^•'foreground^" image on top of a portion of a "background^" image. In addition, pixels on the borders separating portions may be processed (e.g., anti -aliased, etc. ) in order to mimic an attempt by a malicious user to make edge detection more difficult.

[0049] At block 204, an input/output mapping is generated . The input/output mapping refers to the training input that includes or is based on pixel data of the composite image (e.g., red/green/blue intensity values of the pixels of the composite image, etc. ), and the target output for the training input, where the target output identifies one or more positions of the first portion (the portion that contains the pixel data of the first constituent image) within the composite image (e.g., coordinates of two or more points that define the first portion, such as coordinates of the upper left and lower right corners of the first portion, etc. ), and where the training input is associated with (or mapped to) the target output. At block 205, the input/output mapping generated at block 204 is added to training set T.

[0050] Block 206 branches based on whether training set T is sufficient for training machine learning model 160. If so, execution proceeds to block 207, otherwi se, execution continues back at block 202. It should be noted that in some implementations, the sufficiency of training set T may be determined based simply on the number of input/output mappings in the training set, while in some other implementations, the sufficiency of training set T may be determined based on one or more other criteria (e.g., a measure of diversity of the training examples, etc. ) in addition to, or instead of, the number of input/output mappings. [0051] At block 207, training set T is provided to train machine learning model 160. In one implementation, training set T is provided to training engine 141 of server machine 140 to perform the training. In the case of a neural network, for example, input values of a given input/output mapping (e.g., pixel values of a training image, etc.) are input to the neural network, and output values of the input/output mapping are stored in the output nodes of the neural network. The connection weights in the neural network are then adjusted in accordance with a learning algorithm (e.g., backpropagation, etc.), and the procedure is repeated for the other input/output mappings in training set T. After block 207, machine learning model 160 can then be used to process images (for example, in accordance with method 400 of Figure 4, described below).

[0052] Figure 4 depicts a flow diagram of one example of a method 400 for processing images using a trained machine learning model, in accordance with one or more aspects of the present disclosure. The method is performed by processing logic that may comprise hardware (circuitry, dedicated logic, etc.), software (such as is mn on a general purpose computer system or a dedicated machine), or a combination thereof. In one implementation, the method is performed using the server machine 150 and trained machine learning model 160 of Figure 1, while in some other implementations, one or more blocks of Figure 4 may be performed by one or more other machines not depicted in the figures.

[0053] Method 400 may include receiving an input image (e.g., from a user device or a server such as upload server 125) and processing the input image using a trained model such as trained machine learning model 160. The trained model may be configured to generate, based on pixel data of the input image, one or more outputs indicating (i) a level of confidence that the input image is a composite image including a constituent image, and (ii) a spatial area that includes the constituent image within the input image.

[0054] In some implementations, at block 401 , an image ( input image) may be identified for processing. In one example, the image is a frame of an uploaded video (e.g., a video uploaded to a content sharing platform).

[0055] At block 402, a set of spatial areas { Aj, A₂, ... , A„} of the image is determined. Various examples of spatial areas are depicted in Figures 5 through 8. In Figure 5, the spatial areas are equally-sized rectangles (uniform in size and shape) 501-1 through 501-12, while in Figure 6, the spatial areas 601-1 through 601-9 are rectangular but are not equally-sized (i .e., have different sizes). In Figure 7, the spatial areas 701 - 1 through 701 -6 are non-polygonal (e.g., bounded by parametric curves, etc.), while in Figure 8, the spatial areas 801-1 through 801-5 are overlapping non -un i form 1 y-si zed rectangles. It should be noted that other spatial area definitions are possible, such as other types of polygonal areas (e.g., triangles, hexagons, non-rectangular quadrilaterals, etc.), differently-shaped polygonal areas (e.g., some of the areas triangular and other areas rectangular, etc.), overlapping non-polygonal areas, non- overlapping polygonal or non-polygonal areas, and so forth. Moreover, although in the examples of Figures 5 through 8 the spatial areas cover the entire image (i.e., the union of the spatial areas contains all the pixels of the image), in some other examples this might not be the case.

[0056] It should further be noted that in some other implementations, images may be rendered on a non-planar surface (e.g., the inside of a hemisphere in a virtual reality application, etc. ). In such cases, the spatial areas may be three-dimensional surfaces, rather than two-dimensional.

[0057] At block 403, pixel data of the image is provided as input to the trained machine learning model, and at block 404, one or more outputs are obtained from the trained machine learning model . At block 405, confidence data is extracted from the output(s) obtained at block 404 as to whether the input image is a composite image that includes a constituent image, and to whether at least a portion of the constituent image is in a particular spatial area of the input image. In one implementation, the confidence data comprises, for each spatial area A_j of the image, a level of confidence that the image is a composite image including a constituent image ¾ (and in some implementations also at least one more other image I₂), with at least a portion of image 11 in spatial area A, (and in some implementations, also at least a portion of image I₂ in spatial area A,). In one example, the level of confidence is a real number between 0 and 1 inclusive. It should be noted that the lev el of confidence may not be a probability (for example, the sum of the confidence levels for all spatial areas may not equal 1).

[0058] At block 406, the confidence data is used to estimate one or more spatial areas containing image 11 (and in some implementations, also one or more spatial areas containing image I₂). In some implementations, if the level of confidence for a spatial area satisfies a threshold condition, then such a spatial area is identified as containing image

For example, the spatial area that has the highest level of confidence may be identified as containing image I_1; while in some other examples, one or more spatial areas may be identified as containing 11 if such spatial areas hav e a lev el of confidence abov e a particular threshold. [0059] At block 407, pixel data in the identified spatial area(s) is processed. In some implementations, the processing of pixel data may comprise a determination of whether the pixel data is inappropriate for upload to the content sharing platform (e.g., copyrighted material, age-inappropriate material, etc.). As described above, such processing is facilitated by honing in on where in the overall image the smaller image is located. In some

implementations, processing of the pixel data includes extracting the pixel data ( constituent image) from the identified spatial area, creating a fingerprint of the constituent image based on the extracted pixel data, and comparing the created fingerprint with stored fingerprints of multiple images to determine whether the constituent image is similar to any of the multiple images, it should be noted that if multiple constituent images are identified in the composite image using the trained model, the above operations (including operations of blocks 406 and 407) can be performed for each constituent image.

[0060] After block 407, execution continues back at block 401 for processing of another image (e.g., the next frame of the upl oaded video, etc.).

[0061] Figure 9 depicts a block diagram of an i llustrative computer system 900 operating in accordance with one or more aspects of the disclosure. In various illustrative examples, computer system 900 may correspond to a computing device within system architecture 100 of Figure 1. In certain implementations, computer system 900 may be connected (e.g., via a network 930, such as a Local Area Network (LAN), an intranet, an extranet, or the Internet ) to other computer systems. Computer system 900 may operate in the capacity of a server or a client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. Computer system 900 may be provided by a personal computer (PC), a tablet PC, a set-top box (STB ), a Personal Digital Assi stant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, the term "computer" shall include any col lection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods described herein.

[0062] In a further aspect, the computer system 900 may include a processing device 902, a volatile memory 904 (e.g., random access memory (R AM)), a non -volatile memory 906 (e.g., read-only memory ( ROM ) or electrically-erasable programmable ROM (EEPROM)), and a data storage device 916, which may communicate with each other via a bus 908. [0063] Processing device 902 may be provided by one or more processors such as a general purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a microprocessor implementing other types of instructi on sets, or a microprocessor implementing a combinati on of types of instruction sets) or a specialized processor (such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0064] Computer system 900 may further include a network interface device 922.

Computer system 900 also may include a video display unit 910 (e.g., an LCD), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and a signal generation device 920.

[0065] Data storage device 916 may include a non-transitory computer-readable storage medium 924 on which may store instructions 926 encoding any one or more of the methods or functions described herein, including instructions for implementing methods 200 and 400 of Figures 2 and 4, respectively.

[0066] Instructions 926 may also reside, completely or partially, within volatile memory 904 and/or within processing device 902 during execution thereof by computer system 900, hence, volatile memory 904 and processing device 902 may also constitute machine-readable storage media.

[0067] While computer-readable storage medium 924 is shown in the illustrative examples as a single medium, the term "computer-readable storage medium" shall include a single medium or multiple media (e.g., a centralized or di stributed database, and/or associated caches and servers) that store the one or more sets of executable instructions. The term "computer-readabl e storage medium" shall also include any tangible medium that is capable of storing or encoding a set of instructions for execution by a computer that cause the computer to perform any one or more of the methods described herein. The term "computer- readable storage medium" shall include, but not be limited to, solid-state memories, optical media, and magnetic media.

[0068] The methods, components, and features described herein may be implemented Indiscrete hardware components or may be integrated in the functionality of other hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the methods. components, and features may be implemented by component modules or functional circuitry within hardware devices. Further, the methods, components, and features may be

implemented in any combination of hardware dev ices and computer program components, or in computer programs.

[0069] Unless specifically stated otherwi se, terms such as "generating,^" "providing,^" "training,^" or the li ke, refer to actions and processes performed or implemented by computer systems that manipulates and transforms data represented as physical (electronic) quantities within the computer system registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. Al so, the terms "first," "second," "third," "fourth," etc. as used herein are meant as labels to distinguish among different elements and may not have an ordinal meaning according to thei r numerical designation.

[0070] Examples described herein al so relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed for performing the methods described herein, or it may comprise a general purpose computer system selectiv ely programmed by a computer program stored in the computer system . Such a computer program may be stored in a computer-readable tangible storage medium.

[0071] The methods and illustrativ e examples d escribed herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove conv enient to construct more specialized apparatus to perform methods 200 and 400 and/or each of thei r individual functions, routines, subroutines, or operations. Examples of the structure for a v ariety of these systems are set forth in the description abov e.

[0072] The abov e description is intended to be i llustrativ e, and not restrictiv e. Although the present disclosure has been described with references to specific illustrativ e examples and implementations, it will be recognized that the present disclosure is not limited to the examples and implementations described. For example, a feature described with reference to one aspect or implantation may be provided in any other aspect or implementation . The scope of the disclosure should be determined with reference to the following cl aims, along with the full scope of equiv alents to which the claims are entitled.

Claims

CLAIMS What is claimed is:

1. A method implemented by one or more processors for training a machine learning model to identify constituent images within composite images, the method comprising: generating training data for the machine learning model, wherein generating the training data comprises:

generating a composite image comprising a first portion containing pixel data of a first constituent image, and a second portion containing pixel data of a second constituent image;

generating a first training input comprising pixel data of the composite image; and

generating a first target output for the first training input, wherein the first target output identifies a position of the first portion within the composite image; and providing the training data to train the machine learning model on (i) a set of training inputs compri sing the first training input and (ii) a set of target outputs comprising the first target output.

2. The method of claim 1 wherein the second portion of the composite image surrounds the first portion of the composite image.

3. The method of claim I or 2 wherein the first constituent image is a frame of a first video and the second constituent image is a frame of a second video.

4. The method of claim 1 , 2 or 3 wherein the position of the first portion ithin the composite image comprises coordinates of an upper left corner of the first constituent image and coordinates of a lower right corner of the first constituent image.

5. The method of any one of claims 1 to 4 herein each training input in the set of training inputs is mapped to a target output in the set of target outputs.

6. The method of any one of claims 1 to 5 wherein the machine learning model is configured to process a new image and generate one or more outputs indicating (i) a level of confidence that the new image is a composite image including a constituent image, and (ii) a spatial area in which the constituent image is located within the new image.

7. An apparatus comprising:

a memory to store a first image; and

a processing device, operatively coupled to the memory, to:

provide pixel data of the first image as input to a trained machine learning model; obtain one or more outputs from the trained machine learning model; and extract, from the one or more outputs, a level of confidence that:

the first image is a composite image that includes a constituent image, and at least a portion of the constituent image i s in a particular spatial area of the first image.

8. The apparatus of cl aim 7, wherein the processing device is further to:

determine that the level of confidence satisfies a threshold condition; and

extract the constituent image from the particular spatial area of the first image.

9. The apparatus of cl aim 7 or 8 wherein the first spatial area is one of a plurality of spatial areas of the first image, and wherein the plurality of spatial areas are uniform in size and shape.

10. The apparatus of claim 7 or 8 wherein the first spatial area is one of a plurality of spatial areas of the first image, and wherein at least two of the plurality of spatial areas have different sizes.

1 1 . The apparatus of claim 7, 8, 9 or 10 wherein the first spatial area is one of a plurality of spatial areas of the first image, and wherein at least two of the plurality of spatial areas overlap.

12. The apparatus of claim 7, 8, 9 or 10 wherein the first spatial area is one of a plurality of spatial areas of the first image, and wherein the plurality of spatial areas are non- overlapping.

1 3. The apparatus of any one of claims 7 to 1 2 wherei n the first spatial area is one of a plurality of spatial areas of the first image, and wherein at least two of the plurality of spatial areas have different shapes.

14. The apparatus of any one of claims 7 to 13 wherein the trained machine learning model is trained with an input-output mapping comprising an input and an output, the input based on pixel data of a composite image, the composite image compri ing a first portion containing pixel data of a fourth image and a second portion containing pixel data of a fifth image, and the output identifying a position of the first portion within the composite image.

15. A method implemented by one or more processors, the method comprising:

receiving an input image;

processing the input image using a trained machine learning model; and

obtaining, based on the processing of the input image using the trained machine learning model, one or more outputs indicating (i) a level of confidence that the input image is a composite image including a constituent image, and (ii) a spatial area that includes the constituent image within the input image.

16. The method of claim 15 wherein the input image compri ses a second constituent image that surrounds the constituent image.

1 7. The method of claim 1 5 or 16 wherein the constituent image is a frame of a video.

18. The method of claim 15, 16 or 1 7 wherein the spatial area is one of a plurality of spatial areas of the input image, and wherein a union of the plurality of spatial areas contains all pixels of the input image.

19. The method of any one of claims 1 5 to 1 8 wherein the first spatial area is one of a plurality of spatial areas of the input image, and wherein the plurality of spatial areas are uniform in size and shape.

20. The method of any one of claims 1 to 19 wherein the spatial area is one of a plurality of spatial areas of the input image, and wherein at least two of the plurality of spatial areas have different sizes.

2 1 . The method of any one of claims 1 5 to 20 wherein the spatial area is one of a plurality of spatial areas of the input image, and wherein at least two of the plurality of spatial areas have different shapes.

22. A method implemented by one or more processors, the method comprising:

receiving an input image; and

processing the input image using a trained model that is configured to generate, based on pixel data of the input image, one or more outputs indicating (i) a level of confidence that the input image is a composite image including a constituent image, and (ii) a spatial area that includes the constituent image within the input image.

23. The method of any one of claims 15 to 22, further comprising:

determining that the level of confidence satisfies a threshold condition; and extracting the constituent image from the spatial area within the input image.

24. The method of claim 23, further comprising:

creating a fingerprint of the constituent image; and

comparing the created fi ngerprint with stored fingerprints of a plurality of images to determine whether the constituent image is similar to any of the plurality of images.