BRPI0605994B1 - PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING - Google Patents

Abstract

Progressive Randomization Process and Equipment for Multimedia Analysis and Reasoning The present invention relates to a process and equipment for extracting high level multimedia information (e.g., images, sounds, videos). More specifically, the present invention deals with a progressive process of disturbing multimedia content aiming at extracting features not visible in the natural state of the analyzed content. The process and equipment is able to aggregate information extracted from various media classes to solve known issues such as multimedia content categorization, hidden message detection, content tamper detection, robust watermarking, recognition of media originator media (eg, type of camera used in the capture process) among other applications.

Description

“PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING”

FIELD OF THE INVENTION

The present invention deals with a process and equipment for extracting high-level information in multimedia (e.g., images, sounds, videos). More specifically, the present invention deals with a progressive process of disturbance of multimedia content aiming to extract characteristics not visible in the natural state of the analyzed content. Given a set of multimedia objects to be analyzed, the 10 process performs a set of stages to extract intrinsic statistical characteristics from the analyzed objects.

The process and equipment of the present invention are able to aggregate information extracted from various classes of media to solve known problems such as the categorization of multimedia content, detection of hidden messages, detection of adulteration of content, creation of watermarks ( watermarking), recognition of the medium that originated the media (eg, type of camera used in the capture process) among other applications. The process can be loaded on a variety of devices to solve specific applications. For example, it can be embedded in routers in order to analyze the media that pass through the router and, in a non-invasive way, point out suspicious media that may harbor hidden content. On the other hand, the process can also be embedded in digital cameras in order to group the customer's photos into categories (types of images).

BACKGROUND OF THE INVENTION

Semantic knowledge about a particular medium allows us to develop intelligent techniques for processing these media based on their content (Szummer & Picard, 1998). Digital cameras or computer applications can automatically correct color and brightness taking into account the properties of the analyzed scene. In general, reasoning in multimedia is defined in the process of analyzing a set of multimedia data to extract high-level information or semantics about them. Given arbitrary multimedia information, one wants to find out what kind of information it brings. This information can be the automatic determination of the class of the analyzed multimedia object, the information about the presence or

Petition 870190049275, of 05/27/2019, p. 20/36

2/11 not of content hidden in that object, verification of authenticity, verification of possible adulteration of the same among others.

In general, multimedia processing techniques use intrinsic local properties of the analyzed media. For example, in the context of digital images, color histograms and analysis of shape and texture are used, among other techniques. However, this type of approach limits the process to solving only a certain type of problem directly linked to the media used.

The various types of approaches to extracting information and later solving problems using multimedia are described in the state of the art.

The Patent Document US 7075571 discloses an image falsification detection system. The system is able to tell whether a particular photo came from a digital camera or has been modified digitally. Other similar patents are the No US 6,516,078 and US 6,970,259 for detection of copyright infringement in printed documents. These approaches in general need the original means for comparison and subsequent identification of adulteration of content.

The patent document No US 7099510, US 6990239 describe systems and methods for detecting objects in digital images. The systems are not able to classify the analyzed image within a given semantic class, providing the user with only information regarding the absence / presence of a certain object in the analyzed image.

The patent document No. US 7,039,856 en US 7039239 describe methods for classification of digital documents from the content information. The first approach groups similar documents together, but is unable to tell which class they belong to. The second approach groups regions of images, but is not able to generalize saying to which class of images the analyzed image belongs.

The patent document US 7027065 does not disclose a method for image classification for the presence of textures. Patent No. US 6,985,628 describes a method for classification based images on edge information. However, this type of approach is extremely context-dependent. Images with little edge information generally pose problems with this type of classification approach. The patent document No. US 6,810,149 describes a method and system for classification of digital media. However, the approach

Petition 870190049275, of 05/27/2019, p. 21/36

3/11 described represents more a content-based retrieval system than a classification system itself.

The nonpatent document US 6,480,627 describes an evolutionary system able to learn the most important features in a digital media and then use this information in a classification system.

The patent document No. US 6,246,793 discloses a method and device for processing digital images for subsequent pattern recognition. The method and device uses information on textures, edges, shapes and other information. Other similar patents are the No US 5,995,651 for classification based on texture information, US No 5,787,201 based on extraction order fractal characteristics and the patent No. US 5,781,650 which provides a method for classification faces.

The patent documents n ^the US 2005140791, US 6862038, US 2005231611 and JP 2003330941 feature based image categorizers in color information, shape and texture.

The patent documents ^Nos EP 1107179, US 6504951 and DE 60008486 describe a system for the differentiation of outdoor images (billboards) of indoor images (indoors) using medium - level information (middle took information) such as presence / absence of grass, sky, snow, trees and water textures in the analyzed image.

The patent document No. US 6,535,636 describes a method for determining the overall quality of a digital image.

In general, the methods presented are extremely linked to the problem being solved. For images rating on indoors / outdoors, for example, only these two types of classes can be differentiated as in Patent No US 6,504,951.

The state of the art of detecting messages hidden in digital content and detecting adulteration of content is also extensively researched, constituting a very important field in the industry in general.

Patent No. US 6,831,991 describes a hidden message detection system in images using statistical properties of color images and grayscale. However, the system is not considered a “blind-detection” type since the system is not independent of the type of process used to mask the message. A dependent detection system implies

Petition 870190049275, of 05/27/2019, p. 22/36

4/11 that developers create a detection system against a certain masking algorithm and not against a general set of masking techniques. Thus, the detection process may fail on digital content whose masking process (insertion of the hidden message) is not known to it.

The patent document No. US 6,804,377 presents a detection system of hidden messages in digital images based on information changes in the color planes.

Patent documents Paragraphs US 6,064,764, US 7,039,215 and US 6,735,325 have systems to detect watermarks in digital images. The systems are also able to point out attempts to tamper with the inserted brand. They can also be used as techniques for creating more robust watermarks.

In general, the approaches are dependent on the context analyzed. This implies that a system capable of detecting tampering with watermarks, in general, is quite different from a system capable of detecting the presence of hidden messages in digital content.

Therefore, it would be interesting to develop a simple and unified system for detecting hidden messages as well as for detecting tampering with content and attempted counterfeiting in multimedia as well as for categorizing multimedia content. A system that requires only initial training where examples of typical content for each class of problem being analyzed are presented. A process that adapts to solve a different problem in a very simple way, usually implying only direct changes in the learning process and in the process by which the media is analyzed.

BRIEF DESCRIPTION OF THE INVENTION

The present invention deals with a process and equipment for extracting high-level multimedia information (e.g., images, sounds, videos) for later application in a classification system. Given a set of media to be analyzed, the present process consists of changing certain properties of these media in order to obtain high level information that these media present when disturbed by a process of progressive randomization of their content. Different types of media have different behaviors to the developed process and allow the use of these different behaviors in the resolution of some

Petition 870190049275, of 05/27/2019, p. 23/36

5/11 problems associated with the analyzed media.

More specifically, given a set of multimedia objects to be analyzed, the process performs a set of stages to extract intrinsic statistical characteristics from the analyzed objects. Then, a descriptor set is created that represents the analyzed objects. Each group of objects has a specific descriptor set. When you want to differentiate one group of objects from another, just compare the two sets of descriptors and analyze their differences. More specifically, consider the problem of detecting messages hidden in digital content. The distinction between groups of media with and without hidden messages is made by applying the process described here in a sample set of each of the groups. Any new example can be analyzed directly by simply comparing its descriptor set to the descriptor sets already processed by the procedure described here.

The present invention therefore presents a simple and unified system for categorizing images according to their semantic class. The process requires only an initial learning step to capture the statistical behavior of a new class. After learning, any new copy of media supplied to the system will be identified and the invention will be able to pinpoint the class to which this object belongs as long as the training has been effective for the classes of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows the progressive randomization procedure applied to a region (a) of an image. The indicator (b) shows the least significant bits (LSBs or least significant bits) of the image region in (a). The progressive randomization applied on (b) results in the bits disturbed and shown in (c-h). Statistical descriptors are then applied to each of the generated sets (b-h) for the construction of a set of identifying characteristics of the region presented in (a).

Figure 2 shows the selection of some characteristic regions considered important for the progressive randomization procedure. More specifically, eight regions were selected, four of them with overlap (1-4) and four without overlap (5-8).

Petition 870190049275, of 05/27/2019, p. 24/36

6/11

Figure 3 shows the result of a disturbance in four less significant bits of an image. The disturbance entered is represented by bits 1110. The original bits are 1000 and, after the disturbance, the resulting bits are 1110.

DETAILED DESCRIPTION OF THE INVENTION

Small disturbances in the most varied types of media are imperceptible to human beings (Wayner, 2002). However, these disturbances are statistically detectable and useful for inference about the type of media analyzed. The present invention presents PROGRESSIVE RANDOMIZATION as a useful process for reasoning and analysis in multimedia. This is an unprecedented process that captures the changing dynamics of statistical artifacts inserted during the process of disturbance in each semantic class of media of interest. Different types of media have different behaviors under the process presented here and hereinafter referred to as Progressive Randomization (PR).

The present invention has a very wide field of applications that can be used for classification as to the semantic class of media, detection of adulteration of content, detection of hidden messages (Steganalysis), recognition of the medium that originated the media (eg, type of camera used capture process) among other applications.

In general, the Progressive Randomization process has up to seven stages, namely: (1) Acquisition of the media to be analyzed; (2) Multi-scale decomposition; (3) Randomization; (4) Selection of regions; (5) Statistical descriptors; (6) Standardization; (7) Inference. Its prerequisite is the acquisition of the media to be analyzed and results in a set of characteristics that describe the 25 media analyzed. Not all problems require the completion of all stages presented.

ILLUSTRATIVE EXAMPLES:

Progressive Randomization - Procedure

Requirements:

- Set of percentages representing the disturbance intensities

P = {P1, P2, ..., Pn};

Petition 870190049275, of 05/27/2019, p. 25/36

7/11

Stage 1. Acquisition of the media to be analyzed. It consists of the formation of a sample space of M media with examples of media representative of the problem being solved. For each media m present in the media set M, apply the stages described in 2-7.

Stage 2. Multi-scale decomposition. It consists of the decomposition of the media m provided in j scales. For each scale generated, the procedures in stages 3-7 are repeated.

Stage 3. Randomization. It consists of disturbing the less significant statistical properties of the original media m.

Stage 4. Selection of regions. It consists of selecting regions of interest in the analyzed media. It can be understood as a process of sampling the media provided for analysis.

Stage 5. Statistical descriptors. It consists of calculating d statistical descriptors about each region generated by the procedure described in stage 3. The descriptors used are related to the specific application being developed.

Stage 6. Invariance. It consists of normalizing the descriptors in relation to the measures present in the original media before the progressive randomization procedure.

Stage 7. Inference. It consists of using some machine learning technique to differentiate the set of descriptors collected from the set of descriptors already learned by the procedure on other media present in the M media sample space provided for training.

Stage (1): consists of building the sample space of the media to be analyzed. The progressive randomization procedure is then applied to all examples of the sample space provided. If the problem is to detect hidden messages in images, then a set of images with and without hidden messages must be provided. If the problem is with image categorization, then examples of each of the desired classes in the classification should be provided.

Stage (2): the image can be decomposed into multi-levels. Decomposition is optional and depends on the problem being analyzed. The realization of only one scale is a particular case and the most used. The type of decomposition used is related to the problem and the time available to obtain a

Petition 870190049275, of 05/27/2019, p. 26/36

8/11 satisfactory system response.

Stage (3): randomization consists of receiving the original media m belonging to a set of media M and applying a progressive disturbance process over m resulting in T (m, 0), T (m, Pi), ..., T (m, P „).

The transformation T (m, Pi) represents disturbances with different intensities in the less significant properties of the analyzed media. In the case of digital sounds and images, these properties can be the least significant bits. The percentages used and the number of transformations required are related to the problem being solved.

The perturbation process can be defined as a sequence of samplings made from a random variable X of some statistical distribution, such as the Gaussian and Bernoulli distributions. Given a medium m, it is defined as a perturbation of percentage p to the transformation T (m, p), such that p percent of the bits of the medium m will be altered (disturbed) according to the statistical distribution of the random variable X. This The process is similar, but not equivalent, to the process of adding noise to the analyzed media.

Stage (4): Local properties of multimedia content do not appear under a global analysis (Wayner, 2002). Statistical descriptors in localized regions can be used to capture the changing dynamics of the statistical artifacts inserted in the process of progressive randomization.

Given a medium m, r sampling regions over m can be used in order to best represent the medium m. The type and number of regions selected are linked to the problem solved. In one problem, it may be required to select regions with a high level of detail while in another problem, it may be necessary to select regions with a low level of detail.

Stage (5): The disturbance inserted by progressive randomization alters a set of properties of less significance in the analyzed media, inducing local statistical changes. Thus, stage five consists of selecting a set of statistical descriptors capable of measuring these changes and providing a descriptor set of the analyzed media. Different types of media have different behaviors when submitted to Progressive Randomization. For example, if the evaluated media has a message that is somehow hidden in its properties then its behavior under Progressive Randomization is different from that of a media without a hidden content in its properties.

Petition 870190049275, of 05/27/2019, p. 27/36

9/11

Stage (6): In certain problems, a last stage may be necessary to promote the transformation of invariance in the set of collected descriptors. Since the procedure consists of progressively disturbing a set of properties of a given type of media, it may be necessary to provide only the relative behavior of these descriptors in relation to the media without the randomization process. In this way, the set of descriptors is normalized in relation to the original media provided for analysis.

Stage (7): It consists of using a machine learning system to separate (differentiate) the sets of characteristics created after using the previous stages. The classifier used depends on the problem being solved. In general, there are several equivalent possibilities.

The present invention is suitable for solving several problems related to digital and analog multimedia. Experiments have shown that the method described here can detect hidden messages in digital images with an accuracy of 96.3% exceeding techniques described in the art such as, for example, n US patent 2004 / 6,831,991 and US 2004/6804 .377 and (Westfeld, 1999), (Fridrich, 2000), (Fridrich, 2001), (Farid, 2002) and (Provos, 2001) without using the multi-scale stage.

The process described here is capable of differentiating the semantic class from digital images with great precision. The process distinguishes digital photographs from computer-created (photo-realistic) images with an average accuracy of 94.3% surpassing techniques such as those of (Athisos, 1997), (Oliveira, 2002) and (Farid, 2005). The process of distinguishing images outdoors (outdoors) images of internal environments (indoors) with an average accuracy of 93.9% exceeding techniques such as those described in EP Patent ^Nos 1107179, JP 1195591, US 6504951 and DE 60008486 and the work de (Picard, 1998), (Luo, 2001) and (Savakis, 2002).

In addition to the positive results that point to the large-scale applicability of the described procedure, this invention has a simplified and at the same time unified way to solve certain types of problems that were previously solved only on time.

The application steps in image classification are more specifically: (1) acquisition of a database of images that represent each class of image required in the final classification; (2) decomposition of the image in multi-scales; (3) randomization of the least significant bits representing a

Petition 870190049275, of 05/27/2019, p. 28/36

10/11 process of disturbance of these bits; (4) selection of regions with rich details in the analyzed image; (5) use of statistical descriptors that assess image sensitivity to the randomization process; (6) invariance or normalization of the collected sensitivity measures; (7) Inference using some machine learning technique.

The steps of the application in detecting messages hidden in images are more specifically: (1) acquisition of a database of images that represent images without hidden messages; (2) decomposition of the image in multi-scales; (3) randomization of the least significant bits representing a process of disturbance of these bits and also simulating the presence of hidden messages in the analyzed images; (4) selection of regions with rich details in the analyzed image; (5) use of statistical descriptors that assess image sensitivity to the randomization process; (6) invariance or normalization of the collected sensitivity measures; (7) Inference using some machine learning technique.

The described procedure can be adapted to different types of problems and can be embedded in different mechanisms such as routers, video cards, digital cameras, digital signal processors (DSPs), camcorders, microcontrollers, FPGA cards, and other digital mechanisms. It is possible to develop specific equipment containing the procedure described and applied to a given problem. As an example, one can have equipment aimed at analyzing suspects on the internet capable of creating logs and records about individuals who are communicating on the internet using hidden messages on digital media. This act may constitute a crime, for example, the dissemination and sale of images of 25 pedophilia hidden in media previously without an offensive appearance.

The above description of the present invention has been presented for the purpose of illustration and description. In addition, the description is not intended to limit the invention to the form disclosed herein. As a result, variations and modifications compatible with the above teachings, and the skill or knowledge of the relevant technique, are within the scope of the present invention.

Accordingly, the modalities described above are intended to better explain the known methods for practicing the invention and to allow those skilled in the art to use the invention in such or other modalities and with various modifications necessary for the specific applications or uses of the present invention. It is the intention that

Petition 870190049275, of 05/27/2019, p. 29/36

11/11 the present invention includes all modifications and variations thereof, within the scope described in the report and the attached claims.

Claims (4)

1) “PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING”, capable of extracting high-level information in multimedia through a progressive process of content disturbance, consisting of subsets of the steps: - Acquisition of the media to be analyzed, in which the referred acquisition consists, more specifically, in the formation of a sample space of M media with examples of media representative of the problem being solved; - Multi-scale decomposition, in which the aforementioned decomposition consists of the decomposition of the media m provided in j scales, the realization of only one scale being a particular case and the most used; - Randomization, which consists of the disturbance of the less significant statistical properties of the original media m; - Selection of regions, in which the referred step consists in the selection of r regions of interest in the analyzed media, which can be understood as a process of sampling the media provided for analysis; - Analysis of statistical descriptors, in which the said analysis consists of calculating a certain set of statistical descriptors about each generated region being analyzed, where the descriptors to be used are related to the specific application being developed; characterized by also understanding the steps of: - Invariance, which consists of normalizing the descriptors in relation to the measures present in the original media before the progressive randomization procedure; - Inference, which consists of using a machine learning technique to differentiate a set of descriptors collected according to criteria related to the problem solved. 2/2 works of art. 2) "PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING", according to claim 1, characterized by being able to be applied in multimedia content categorization, detection of hidden messages, detection of content tampering, creation of watermarks (watermarking), recognition of the medium that originated the media (eg, type of camera used in the capture process) and identification of falsification of Petition 870190049275, of 05/27/2019, p. 7/36 3) “PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING”, according to claims 1 and 2, characterized by the application stages in image classification to be more specifically: (1) acquisition of a database of images that represent each class of image needed in the final classification; (2) decomposition of the image in multi-scales; (3) randomization of the least significant bits representing a process of disturbance of these bits; (4) selection of regions with rich details in the analyzed image; (5) use of statistical descriptors that assess image sensitivity to the randomization process; (6) invariance or normalization of the collected sensitivity measures; (7) Inference using some machine learning technique. 4) "PROGRESSIVE RANDOMIZATION PROCESS FOR MULTIMEDIA ANALYSIS AND REASONING", according to claims 1 and 2, characterized by the steps of the application in detecting hidden messages in images to be more specifically: (1) acquisition of a database of images that represent images without hidden messages; (2) decomposition of the image in multi-scales; (3) randomization of the least significant bits representing a process of disturbance of these bits and also simulating the presence of hidden messages in the analyzed images; (4) selection of regions with rich details in the analyzed image; (5) use of statistical descriptors that assess image sensitivity to the randomization process; (6) invariance or normalization of the collected sensitivity measures; (7) Inference using some machine learning technique.