KR20090121024A - A watermarking method of the mobile terminal for multimedia forensic - Google Patents

Abstract

PURPOSE: A watermarking method for a mobile terminal is provided to enhance an image processing speed for watermarking in the mobile terminal and minimize the image degradation after the watermark inserting by adjusting the insertion strength according to the frequency characteristic of the location. CONSTITUTION: The electronic magazine watermark image consisting of the photographing information including the telephone number of the mobile terminal, and the generic number and shooting date is generated(S10). The binary bit string of the watermark image which is generated in the watermark video generating step is inserted repetitively into the luminance component of the photographic image(S30). The binary bit string of the watermark image which is generated in the watermark video generating step is inserted repetitively into the chrominance component of the photographic image(S40). The final image is compressed and stored(S60).

Description

A watermarking method of the mobile terminal for multimedia forensic}

The present invention relates to a watermarking method of a mobile terminal, and more particularly, to generate a watermark image through a combination of images for each character of shooting information from shooting information including a phone number, a terminal unique number, a shooting date, and the like. The binary bit string of the generated watermark image is repeatedly inserted into the luminance component and the chrominance component of the photographed image, and the insertion intensity is adjusted according to the frequency characteristic of the position at which the watermark is inserted, resulting in watermarking on the mobile terminal. The present invention relates to a watermarking method that can reduce image processing time delay and deterioration of image quality after embedding a watermark and at the same time ensure robustness against image processing and intentional attack.

In the past, embedded systems were only devices that were optimized to perform specific functions, and most of the features for intercommunication and compatibility were largely excluded. However, in recent years, the integration of semiconductor processes and the innovative development of mobile multimedia media and increasing user demand have developed into a paradigm of digital convergence, extending the performance and compatibility of embedded systems to personal portable PCs.

In particular, the era of 2G to 3G in the mobile terminal market has shifted the main functions of mobile terminals for voice communication from simple communication means to high-resolution digital cameras / camcorders with hardware for high-performance image and video processing. have.

As such, the mobile terminal equipped with the image capturing means is bringing a lot of civilization as the performance and resolution of the built-in camera are improved day by day, and it is expected to have more useful value in the future.

On the other hand, due to the miniaturization and high performance of these mobile terminals, users can take pictures of people from a distance or in a small space without permission to take pictures without permission. There have been many reports of serious personal privacy infringement and moral problems by dissemination, and illegally photographing and leaking documents that require national confidentiality or documents and equipment related to vital core technologies that require the vitality of the company. It is often reported in the media that serious side effects can result in significant economic losses for a country or a company.

To prevent such illegal leaks, bringing in a mobile terminal equipped with a camera function when entering or leaving a place where actual state confidentiality is required, or where important corporate documents or equipment are located, or where important meetings such as business strategy are held. This situation is forbidden.

In addition, in order to prevent social side effects caused by the unrestricted use of the camera phone, the mobile device equipped with the camera function is mandated by the terminal manufacturer to forcibly generate a sound when taking a picture or video. It makes people aware of the shooting.

However, this method is easy to be neutralized by replacing a silent ringtone with the default recording sound embedded in a mobile terminal, and the effect is not so great in a noisy and crowded place. There is an urgent need to develop forensic techniques that can obtain information from fraudulent photographers and illegal distributors and track the first photographers and reservations, ie, multimedia forensic.

The present invention solves the problems of the prior art described above. That is, an object of the present invention is to generate a watermark image through a combination of images for each character of the shooting information from the shooting information including the telephone number, the terminal unique number and the shooting date, and the binary bit string of the generated watermark image. Is repeatedly inserted into the luminance component and the chrominance component of the captured image, and the insertion intensity is adjusted according to the frequency characteristic of the position at which the watermark is inserted, thereby improving the image processing speed for watermarking on the mobile terminal, This is to minimize image quality deterioration after insertion and to secure robustness against image processing and intentional attack.

According to an aspect of the present invention, there is provided a watermark image generation step of generating a binary watermark image composed of photographing information including a phone number, a unique number, and a photographing date of a mobile terminal; A luminance watermarking step of repeatedly inserting a binary bit string of the watermark image generated in the watermark image generation step into a luminance component of a captured image; A color difference watermarking step of repeatedly inserting a binary bit string of the watermark image generated in the watermark image generation step into a color difference component of a captured image; Summing a luminance component with a watermark and a chrominance component through the luminance watermarking step and the color difference watermarking step to generate a final image having a watermark embedded therein, and compressing and storing the generated final image; It provides a watermarking method of a mobile terminal, characterized in that comprising a.

The watermarking method of a mobile terminal according to the present invention greatly improves the image processing speed required for watermarking by simplifying and optimizing an image processing algorithm for watermarking photographing information, and at the same time, invisibility and robustness of watermarks. By securing it, it is possible to extract suspects quickly and accurately by applying it to the multimedia forensics for extracting fraudulent photographers and unauthorized distributors.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically illustrating an image processing related configuration of a mobile terminal to which a watermarking method according to the present invention is applied.

As shown in FIG. 1, the mobile terminal to which the watermarking method according to the present invention is applied includes a camera module 10 for capturing a still image or a video, a camera controller 15 for controlling the camera module 10, and The memory 30 stores the image captured by the camera module 10, the memory controller 35 controlling the memory 30, and inserts a watermark into the captured image stored in the memory 30. And a main controller 20 connected to each of the controllers 15 and 35 and the image processor 50 to control the entire apparatus. In addition, it includes a photographing button 60 for receiving a photographing command from the user, and an interface manager 65 for receiving the input signal of the photographing button 60 and transmitting it to the main controller.

The image processing unit 50 includes a watermark inserting unit 51 for inserting and hiding a watermark in the captured image, and an image compression for compressing the watermark embedded image by the watermark inserting unit 51 in JPEG or MPEG format. The unit 52 is comprised.

The watermark inserter 51 does not have an original image because the first photographed image is automatically watermarked and compressed and stored due to the characteristics of the mobile terminal, and since the storage space is narrow, additional information necessary for future detection is required. Considering that it is difficult to store the watermark, a blind method of watermarking capable of detecting a watermark without an original image is performed.

The image compression unit 52 typically compresses each divided region by dividing the target image into a predetermined number in order to improve image processing speed. For example, when compressing the target image in the JPEG format, the image compressing the target image is 8 × 8. The block is divided into blocks, and a discrete cosine transform (DCT) is performed on each divided block.

Since the original image photographed by the mobile terminal configured as described above is not left and only the watermarked image is stored through JPEG or MPEG compression and image processing for image quality improvement, the watermarked information is stored in various image processing methods. It must be able to satisfy the toughness. In addition, since the CPU core mounted on the camera phone has a lower performance in terms of processing speed and data throughput than the CPU core of a general computer, the watermarking algorithm should be as simple as possible, and watermarking should be performed at the same time as the video recording. Real-time should be satisfied as much as possible.

Therefore, in the present invention, watermarking is performed on an integer-based 5/3 Tab Discrete Wavelet Transform (DWT) domain with respect to the luminance signal of the captured image in consideration of the robustness of the watermark and the real-time characteristics of the embedded environment. By adopting a method of performing watermarking in the pixel domain by changing a range of chroma difference values (CDV) for the color difference signal, it is possible to secure robustness for image processing and to minimize time delay due to watermarking. .

2 is a flowchart illustrating a watermarking process according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a detailed procedure of the watermarking process shown in FIG. 2.

Hereinafter, a watermarking method according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3.

First, a binary watermark image including information related to photographing, that is, photographing information such as a phone number and a unique number of a mobile terminal and a photographing date is generated (S10). That is, one binary image is selected by selecting binary images that match the shooting information at the time of shooting from among a predefined set of binary images for each letter and number that may be included in the shooting information, and arranging and combining the selected binary images in a predetermined rule. Create a watermark image.

Subsequently, raw data for the captured image including the luminance component and the chrominance component is received from a camera module such as a CMOS imaging sensor (S20). The raw data of the captured image is data obtained by converting Bayer data consisting of RGB components of each pixel of the captured image to the YCbCr color space, and corresponding to Y value and luminance component corresponding to luminance component. It consists of Cb and Cr.

Subsequently, the binary bit string of the watermark image generated in the step S10 is repeatedly inserted into the luminance component of the row data transmitted in the step S20 (S30). That is, 5/3 Tab DWT conversion of luminance components of the raw data of the captured image (S31), and a binary bit string of the generated watermark image is repeatedly inserted into the DWT transform coefficients of the converted 5/3 Tab DWT domain. After that (S32), color space data of the watermark-embedded image is obtained by inverse DWT conversion (5/3 Tab IDWT) of the watermark-embedded DWT domain (S33). Here, after regal 5/3 DWT conversion of a luminance component of a captured image using a lifting scheme for fast image processing, a watermark is adaptively inserted into a low frequency region to secure robustness. A detailed process of inserting watermark information into the luminance component of the image will be described later in detail.

The coefficients corresponding to the high frequency region among the DWT transform coefficients acquired during the watermark embedding process for the luminance component in step S30 are stored separately for reference in a color difference component watermark embedding step to be performed later.

Thereafter, the binary bit string of the watermark image generated in the step S10 is repeatedly inserted into the color difference component of the raw data transmitted in the step S20 (S40). That is, the watermark is inserted in a manner of changing the range of chroma difference values (CVV) of the color difference component for each pixel among the raw data of the captured image according to corresponding bit values in the binary bit string of the generated watermark image. . Here, the CDV corresponds to the difference between the Cb value and the Cr value in each pixel of the captured image. A detailed process of inserting watermark information into the color difference component will be described in detail later.

When the step of inserting the watermark for the chrominance component (S40) is completed, the final component including the watermark embedded by adding the luminance component and the chrominance component including the watermark is generated (S50), and the resulting final image is JPEG Or it is compressed and stored in the MPEG format (S60).

Meanwhile, in the watermark embedding step (S30) for the luminance component and the watermark embedding step (S40) for the color difference component, the watermark image generated in the watermark generation step (S10) is first performed before performing each step. May be enlarged at a predetermined magnification in consideration of the size of the captured image, and watermark information may be inserted using a binary bit string of the enlarged watermark image.

FIG. 4 is a diagram illustrating an example of watermark generation in the watermark image generation step shown in FIG. 2.

As described above, in the watermark image generation step (S10), in order to consider a multimedia forensic type, photographing information such as a phone number, a unique number, and a photographing date of the mobile terminal where the picture is taken are used as watermark information.

For this purpose, as shown in FIG. 3, a total of 36 binary image sets including 10 × 20 binary images representing ASCII characters corresponding to uppercase letters of the alphabet and numbers 0 to 9 and A to Z are generated. Pre-stored in the memory of the mobile terminal, in the shooting mode of the pre-recorded binary image of the 36 pre-stored binary images that match each letter or number of the current shooting information sequentially selected and arranged by combining in a certain way to shoot Generate a binary watermark image including information. For example, if a total of 24 characters are required as the shooting information, a total of 24 binary images corresponding to each character are extracted, and these are combined into 8 × 3 width and width, which is a predetermined combination rule, to form a binary having a total size of 80 × 60. The watermark image is generated.

Here, in order to more securely protect user information included in the watermark image, the binary bit string of the watermark image is encrypted with a binary bit string unknown to the user using an encryption key of a certification authority, and then the encrypted binary watermark. It is desirable to reconstruct the watermark image into a bit string.

In addition, by extracting pixel values located at the edge of each character region from the generated combined image to form a pattern image having the same size as the combined image, by interpolating the pixel values of the remaining important pixels except the extracted pixel to the pattern image By generating a watermark image, even if a photograph is distributed to the Internet and processed by a third party such as moving, cutting, and scaling, the watermark pattern of the pattern image is applied when forensic is applied. It is possible to find and easily detect important user information.

FIG. 5 is a diagram illustrating a DWT conversion method of a watermark embedding step for the luminance component illustrated in FIG. 2.

Hereinafter, a method of inserting a binary bit string of a watermark image in the watermark embedding step S30 of the luminance component will be described in detail with reference to FIG. 5.

The watermark embedding of the luminance component according to the present embodiment is based on Re Gall 5/5 adopted by the JPEG2000 standard in order to reduce the computational complexity and the amount of calculations and minimize the memory usage so that the mobile system can be driven smoothly in the embedded system of the mobile terminal. 3 Tab Uses the integer DWT and the lifting scheme.

The two-channel filter bank of Re Gall's 5/3 Tab DWT is represented by Equation 1 below.

Here, X _e (z) and X _o (z) represent even-numbered and odd-numbered pixel values, respectively, and S (z) and D (z) represent low-frequency and high-frequency DWT coefficients, respectively.

In order to quickly perform the DWT conversion of Equation 1 on the entire image, DWT is performed by applying a lifting scheme as shown in FIG. 5. Prediction Stage (P (z)) and Update Stage (U (z)) By quickly calculating the DWT coefficients S (z) and D (z) can be calculated.

When the DWT conversion is completed as described above, watermark information is inserted into the LL3 region of the low-level three-level DWT transform coefficient. First, the watermark image is enlarged to the same size as the size (M x N) of the LL3 region. The binary bit string of the enlarged watermark image is inserted into all DWT transform coefficients of the LL3 region by sequentially concealing one bit to each DWT transform coefficient of the LL3 region.

At this time, the sum of all DWT transform coefficients C _{(m, n)} in the high frequency region as shown in Equation 2 below. Calculate a threshold value (Th) representing a high frequency region characteristic at the DWT level of the captured image from the average, compare the frequency characteristic and the threshold value (Th) at each position of the LL3 region, and insert the watermark inserted into the corresponding DWT conversion coefficient. Determine the strength

Here, M and N represent the vertical and horizontal size of the frequency domain of Level 3 in the DWT conversion domain.

는 비가시성 및 강인성을 위해 삽입 강도를 조절할 수 있게 하는 조절인자로 사용된다.Subsequently, the frequency characteristic value g _{(m, n)} for the DWT transform coefficient value C _{LL3 (m, n)} for each position of the LL3 region is determined as shown in Equation 3 below, and then the Th determined in Equation 2 is used. Then, the watermark embedding intensity mag _{(m, n)} is determined as in Equation 4 below. here

Is used as a modulator to control the insertion strength for invisibility and toughness.

Finally, the watermark is concealed on the bit plane of the LL3 coefficients of the integer-based DWT transform region as shown in Equation 5 below.

Where C ^* _{LL3 (m, n)} is the DWT conversion coefficient of the LL3 region where the watermark is inserted, C _{LL3 (m, n)} is the DWT conversion coefficient of the LL3 region before the watermark embedding, w is a watermark bit, &, |, << means a bitwise operator.

Hereinafter, a method of inserting a binary bit string of a watermark image in the watermark embedding step S40 for the color difference component will be described in detail.

In the present invention, by using the chrominance difference value (CDV) of each color difference component (Cb, Cr) of the photographed image, it is possible to improve the additional robustness to zero scratches such as cutting, rotation, and scaling of the photographed image. Perform watermarking with less computation. This is based on the fact that most JPEG compression uses the same quantization matrix for Cb and Cr signals, and that the color information of each pixel has little change for image compression, cutting, rotational scaling, etc. By using unique CDV, it is possible to realize the robust watermarking for various geometric image processing.

The watermarking of the color difference component is performed by changing the CDV range of each pixel of the captured image to an arbitrary value within a range that satisfies Equation 6 below according to the corresponding watermark bit value and the frequency characteristic of the corresponding position. . That is, after checking whether the CDV of the current chrominance component is located in one of the categories representing the watermark '0' or '1' Bit, when the watermark bit to be inserted is '1', the CDV is closest to the current CDV. The color difference components Cb and Cr are changed to move to the position of '1'. On the contrary, even when the watermark bit to be inserted is '0', the CDV is moved to the same '0' position as the watermark bit by changing the color difference component Cb and Cr values in the same manner. If the current CDV belongs to the same category as the watermark bit to be inserted, the robustness against attack is improved by changing the chrominance components Cb and Cr so that the CDV value is located at the center of the category.

Here, CDV _{i, j} and CDV ^* _{i, j} represent the difference between the original color difference component (Cb _{i, j} -Cr _{i, j} ) and the color difference component changed due to the watermark, respectively, m is a step variable, w Is a binary watermark bit value.

In the YCbCr 4: 2: 2 and 4: 2: 0 formats, since the change of the chrominance signal is likely to cause contour noise for the color of the entire image, in order to take this into consideration, in the step of inserting a watermark for the luminance component (S30) Calculation of the obtained equation (3) Using the result, the contour noise of the color difference signal due to the watermark insertion is varied by varying the step variable m according to the following Equation 7 according to the frequency characteristic g _{L1 (i / 2, j / 2)} of the corresponding pixel region. It can be minimized.

Where R and β are constant factors, respectively.

As described above, in the watermarking method according to the present invention, the binary watermark bit string of the watermark image composed of the photographing information is repeatedly concealed in the luminance component and the chrominance component of the photographed image, and the position at which the watermark is inserted. By adjusting the insertion strength in accordance with the frequency characteristic of, it is possible to minimize image quality deterioration after watermark insertion and to secure robustness to image processing.

Hereinafter, a method of reversely detecting a watermark inserted by the aforementioned watermarking method will be briefly described.

In case of watermarking the luminance component, user information hidden in the image statistically by pattern matching the watermark image detected by inverse DWT conversion of the luminance component of the watermark-embedded image with the prestored binary image of each character. Detect.

Meanwhile, in the case of watermarking a color difference signal, a CDV value of a color difference component of a watermark-embedded image is obtained to find a pattern image, and various image processing parameters are determined according to the degree of deformation of the watermark pattern in the distribution process. After the calculation, the watermark information is detected through post-processing according to the calculated parameters. In this way, the result of detecting the watermarking of the color difference signal can be used in addition to the watermark detected from the luminance signal, and to detect accurate user information together with the watermark of the luminance signal for extracting the negative photograph of the mobile phone. Can be used for

In order to evaluate the performance of the watermarking method according to the present invention, a mobile phone equipped with a 200M Pixel CMOS camera module and the MSM-6550 chipset was used, and four resolutions supported by the mobile phone (UVGA, 4VGA, VGA, QVGA) were used. Experiments were performed at all resolution levels except low quality QVGA resolution, which is not suitable for fraud and video information preservation, and all possible JPEG compression ratios (High: 95%, Middle: 80%, Low: 50) in the test mobile phone. Toughness was evaluated.

As shown in Table 1 below, the experimental results verifying the robustness of the watermarking of the 24 characters of the luminance component according to each resolution and compression ratio.

Resolution Quality Statistical ratio CER UVGA High (90%) 92.15% 0% Middle (80%) 91.67% 0% Low (75%) 91.44% 0% 4VGA High 86.53% 0% Middle 84.76% 0% Low 83.19% 0% VGA High 74.32% 0% Middle 73.25% 7.4% Low 72.76% 14.8%

Table 1 shows the statistical detection rate and the error rate of the detected character information according to each resolution and compression rate. From the experimental results shown in Table 1, the first 24 hidden characters regardless of the compression rate in the VGA or higher resolution It was confirmed that it was detected.

FIG. 6 is a graph showing a CDV histogram of JPEG compression results of color difference component watermarking according to the present invention. For convenience of experiment, m value applied to Equation 6 is fixed to 20. FIG.

As can be seen in FIG. 6, as a result of applying the color difference component watermarking applied to the watermarking for the luminance component and the watermark robustness for scaling and rotation of the negatively photographed image, most histogram values are obtained after the JPEG compression. It can be seen that it is distributed within the detectable range.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

1 is a block diagram schematically illustrating a configuration related to image processing of a mobile terminal to which a watermarking method according to the present invention is applied.

2 is a flow chart showing a watermarking process according to an embodiment of the present invention.

3 is a view showing a detailed procedure of the watermarking process shown in FIG.

4 is a diagram illustrating a watermark generation example of the watermark image generation step shown in FIG. 2;

FIG. 5 is a diagram showing a DWT conversion method of a watermark embedding step for the luminance component shown in FIG. 2; FIG.

6 is a graph showing a CDV histogram for JPEG compression results of chrominance component watermarking according to the present invention.

<Explanation of symbols for the main parts of the drawings>

10: camera module 15: camera control unit

20: subject control 30: memory

35: memory controller 50: image processor

65: interface management unit

Claims (7)

A watermark image generation step of generating a binary watermark image composed of photographing information including a phone number, a unique number, and a photographing date of the mobile terminal; A luminance watermarking step of repeatedly inserting a binary bit string of the watermark image generated in the watermark image generation step into a luminance component of a captured image; A color difference watermarking step of repeatedly inserting a binary bit string of the watermark image generated in the watermark image generation step into a color difference component of a captured image; Summing a luminance component with a watermark and a chrominance component through the luminance watermarking step and the color difference watermarking step to generate a final image having a watermark embedded therein, and compressing and storing the generated final image; Watermarking method of a mobile terminal, characterized in that comprises a. The method of claim 1, The watermark image generation step, By selecting binary images that match the shooting information at the time of shooting from among a predefined set of binary images for each letter and number that may be included in the shooting information, by arranging and combining the selected binary images in a predetermined predetermined rule Watermarking method of a mobile terminal, characterized in that for generating one binary watermark image. The method of claim 1, The brightness watermarking step, Binary bits of the watermark image generated in the watermark image generation step in the DWT coefficients obtained by the discrete wavelet transform (DWT) of the luminance data of the raw data for the captured image and the discrete wavelet transform Watermarking method of a mobile terminal, characterized in that to insert a row repeatedly. The method of claim 1, The brightness watermarking step, The luminance component of the raw data of the captured image is converted into a Re Gall 5/3 DWT using a lifting scheme, and the three-level DWT conversion coefficient obtained through the Re Gall 5/3 DWT conversion is used. Insert watermark bits sequentially in the LL3 region, but agree on all DWT transform coefficients in the high frequency region. Calculating a threshold indicating a high frequency region characteristic at a DWT level of the captured image from the average, and comparing the calculated frequency value with the frequency characteristic at each position of the LL3 region into which the watermark bit is inserted to determine the watermark embedding intensity. Watermarking method of a mobile terminal characterized in that. The method of claim 1, The color difference watermarking step, Chrominance Difference Values (CVVs) of the color difference components of the raw data for the captured image are shifted to a range close to the corresponding bit values in the binary bit string of the watermark image generated in step a). And changing the watermark to insert watermark information into the color difference component. The method of claim 5, CDV of the color difference component of the raw data, Watermarking method of a mobile terminal, characterized in that it is changed to any value within the range satisfying the following formula. (Equation)

(CDV _{i, j} : Difference value of chrominance component of original image, CDV ^* _{i, j} : Difference value of chrominance component changed due to watermark, w _{i, j} : Binary watermark bit, m: Step variable) The method of claim 6, The step variable (m) of the equation is a watermarking method of a mobile terminal, characterized in that the variable according to the frequency characteristic value of the corresponding pixel area.

Images