KR101220420B1 - Hybrid video forensic watermarking system - Google Patents

Abstract

,

)을 구하고, 최종 추정 패턴과 참조패턴 사이의 상관 연산을 수행하여 robust 마크 패턴 및 fragile 마크 패턴을 추출하는 포렌식 마크 추출기; 를 포함한다.The present invention relates to a complex forensic watermarking system, which simultaneously inserts a robust watermark and a fragile watermark, stably extracts the inserted information even after deformation, and determines whether the corresponding content is deformed. Its purpose is to provide a system that can.
In order to achieve the above object, the present invention generates an insertion pattern to be inserted into an original image by using a robust forensic mark pattern, a fragile forensic mark pattern, and a weight value considering invisibility, and inserts the generated insertion pattern into the original image. A forensic mark inserter; And extracted and accumulated final forensic mark patterns (

,

A forensic mark extractor for extracting the robust mark pattern and the fragile mark pattern by performing a correlation operation between the final estimated pattern and the reference pattern; .

Description

Hybrid Forensic Watermarking System {HYBRID VIDEO FORENSIC WATERMARKING SYSTEM}

The present invention relates to a complex forensic watermarking system, which simultaneously inserts a robust watermark and a fragile watermark, stably extracts the inserted information even after deformation, and determines whether the corresponding content is deformed. It's about a system that can do that.

Forensic marking technique can be divided into forensic mark insertion process and extraction process. Insertion process is the process of inserting the forensic mark, which is copyright information or buyer's own information, invisible to the original data. Extraction process is whether copyright information or forensic mark, which is buyer's own information, exists in the data obtained through distribution process or It is the process of extracting the inserted information. At this time, the forensic mark must be able to withstand various attacks that can be received in the distribution process of the data in which the forensic mark is inserted.

In general, forensic mark insertion is performed by inputting original data, information to be inserted (forensic mark), and keys. The key is used to prevent forensic mark recovery and extraction by unauthorized users.

Based on a general forensic marking technique, as shown in FIG. 1A, a technique of inserting a robust forensic mark and a semi-fragile forensic mark into a single medium at the same time is called a complex forensic marking technique. Robust forensic mark inserts CCL and copyright information into the content so that it can be detected even after modification, and semi-fragile forensic mark is used to determine whether the content has been modified after insertion.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and inserts a robust watermark and a fragile watermark at the same time, stably extracts the inserted information even after deformation, and whether the corresponding content is deformed. The purpose is to provide a system that can determine the.

)을 생성하는 robust 포렌식 마크 패턴 생성부와, 입력 영상의 바탕으로 기본패턴 블록을 생성하고, 각 기본패턴 블록에 대한 비밀키를 생성하며, 상기 비밀키를 이용하여 최종패턴(

)을 생성하는 fragile 포렌식 마크 패턴 생성부와, 포렌식 마크 패턴 삽입 위치에 대한 패턴의 세기를 결정하는 복합적 마스킹 값을 구하는 마스킹부, 및 상기 robust 포렌식 마크 패턴 생성부를 통해 생성된 최종패턴(

), fragile 포렌식 마크 패턴 생성부를 통해 생성된 최종패턴(

) 및 상기 마스킹부를 통해 계산된 각각의 가중치(

)를 이용하여, 최종적으로 원본영상에 삽입할 삽입패턴을 생성하고, 생성된 삽입패턴을 원본영상에 삽입하는 포렌식 마크 패턴 삽입부를 포함하는 포렌식 마크 삽입기; 및 검사 대상 영상으로부터 프레임 단위로 입력을 받아 포렌식 마크 패턴을 추정하는 포렌식 마크 패턴 추정부와, 상기 포렌식 마크 패턴 추정부를 통해 각 프레임에서 추정된 포렌식 마크 패턴을 이용하여, robust 포렌식 마크 삽입, semi-fragile 포렌식 마크 삽입에 사용했던 프레임 그룹 구간동안 누적하여 최종 포렌식 마크 패턴(

,

) 을 구하는 포렌식 마크 패턴 축적부와, 비밀키 정보를 이용하여 참조 기본패턴(

)을 생성하고, 최종 추정 패턴(

)과, 참조 기본패턴(

) 사이의 상호 상관 연산을 수행하여 마크 추출 여부를 결정하는 robust 포렌식 마크 패턴 추출부, 및 시간적/공간적 인덱스 및 프레임당 할당 시간을 바탕으로 키 및 참조패턴을 생성하고, 영상의 마지막 프레임의 존재여부를 검사함으로써, 변형 여부를 판단하는 fragile 포렌식 마크 패턴 추출부를 포함하는 포렌식 마크 추출기; 를 포함한다. The present invention for achieving the above technical problem is related to a complex watermarking system, generating a basic pattern using a secret key and a message, and extends and repeats the basic pattern vertically and horizontally to the final pattern (

A robust forensic mark pattern generation unit for generating a basic pattern block based on an input image, a secret key for each basic pattern block, and a final pattern

Fragile forensic mark pattern generation unit for generating a), a masking unit for obtaining a complex masking value for determining the intensity of the pattern with respect to the forensic mark pattern insertion position, and the final pattern generated by the robust forensic mark pattern generation unit (

), the final pattern generated by the fragile forensic mark pattern generator (

) And the respective weights calculated by the masking unit (

A forensic mark inserter including a forensic mark pattern inserter which finally generates an insertion pattern to be inserted into the original image and inserts the generated insertion pattern into the original image; And a forensic mark pattern estimator for estimating a forensic mark pattern by receiving an input from the inspection target image in units of frames and a forensic mark pattern estimated in each frame through the forensic mark pattern estimator. The final forensic mark pattern (accumulated over the frame group period used to insert the fragile forensic mark)

,

), And a reference basic pattern (

) And the final estimated pattern (

), And the reference base pattern (

Robust forensic mark pattern extraction unit that determines whether to extract the mark by performing cross-correlation between the two), and generates the key and reference pattern based on the temporal / spatial index and the allocation time per frame, and whether the last frame of the image exists Forensic mark extractor comprising a fragile forensic mark pattern extraction unit for determining whether the deformation by checking; .

According to the present invention as described above, there is an effect that can be applied to all video formats by inserting a watermark in the pixel region of the image, in the case of a robust watermark, by inserting the information divided into all frames, recompression and It also has a robust effect on time-base cutting.

According to the present invention, CCL is technically used in a situation where a movement to protect the rights of authors is started in the situation where a large number of videos are being distributed on the Internet due to the spread of video capture equipment, high-speed Internet environment, and generalization of video manipulation tools. By proposing complex video watermarking technology that can be protected and made efficient, it also has the effect of laying the foundation for content protection and proper sharing environment.

Figure 1a is an exemplary view showing a conventional general composite forensic marking.
2 is an overall configuration diagram conceptually showing a complex forensic watermarking system according to the present invention.
3 is a detailed block diagram of a robust forensic mark pattern generation unit according to the present invention;
Figure 4 is an example representation of 2bit information in one pattern according to the present invention.
5 is an exemplary view showing the insertion position of the robust forensic mark according to the present invention.
Figure 6 is a detailed configuration of the fragile forensic mark pattern generation unit according to the present invention.
7 is an exemplary view showing a secret key generation process for a fragile forensic mark pattern block according to the present invention.
8 is a detailed configuration of the masking unit according to the present invention.
9 is a graph showing the allowable amount of change according to the brightness value according to the present invention.
10 is an exemplary view illustrating a process of generating a final estimation pattern by folding a final forensic mark pattern according to the present invention.
11 is a detailed block diagram of a robust forensic mark pattern extraction unit according to the present invention.
12 is a detailed block diagram of a fragile forensic mark pattern extraction unit according to the present invention;
13 is an exemplary view showing a forensic mark inserted original image and a reduced image according to the present invention.
14 is an exemplary view illustrating a procedure for determining whether to deform in accordance with the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. It is to be noted that the detailed description of known functions and constructions related to the present invention is omitted when it is determined that the gist of the present invention may be unnecessarily blurred.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings.

The complex forensic watermarking system according to the present invention will be described with reference to FIGS. 2 to 14.

FIG. 2 is an overall configuration diagram conceptually illustrating the complex forensic watermarking system S according to the present invention, and includes a forensic mark inserter 100 and a forensic mark extractor 200 as shown.

The forensic mark inserter 100 generates an insertion pattern to be inserted into an original image by using a robust forensic mark pattern, a fragile forensic mark pattern, and a weight value in consideration of invisibility, and generates the inserted pattern. As shown in FIG. 2, a robust forensic mark pattern generator 110, a fragile forensic mark pattern generator 120, a masking unit 130, and a forensic mark pattern inserter may be inserted into the original image. 140).

)을 생성하는 기능을 수행하는 바, 도 3 에 도시된 바와 같이 robust 기본패턴 생성모듈(111), 페이로드 패턴 생성모듈(112), 패턴 확장모듈(113), 최종패턴 생성모듈(114), 패턴위치 설정모듈(115) 및 최종패턴 삽입모듈(116)을 포함한다. Specifically, the robust forensic mark pattern generator 110 generates a robust forensic mark pattern by spread spectrum watermarking, generates a basic pattern using a secret key and a message, and transverses the basic pattern. Expand and repeat vertically to final pattern (

3) the robust basic pattern generation module 111, the payload pattern generation module 112, the pattern expansion module 113, the final pattern generation module 114, and the like. The pattern position setting module 115 and the final pattern insertion module 116 are included.

Robust basic pattern generation module 111 generates 80 * 60 sized gaussian distribution with a mean of 0 and a variance of 1 using a secret key and a message to generate a forensic mark pattern to be inserted in one frame. Create a basic pattern.

Here, the basic pattern generated by the first basic pattern generation module 111 contains 1 bit of information according to the presence or absence of the pattern.

The payload pattern generation module 112 includes bit information, and generates the basic pattern and the basic pattern in the form of vertical axis inversion, horizontal axis inversion, and horizontal and vertical axis inversion. That is, the pattern is divided into four types, and each pattern contains 2 bits of information.

In this embodiment, the generated basic pattern is mirrored on the horizontal axis, the vertical axis, and the horizontal / vertical axis, respectively, to represent 2 bits of information as shown in FIG. 4.

The pattern expansion module 113 extends the basic pattern of 80 * 60 size horizontally and vertically (height / 180 of the input image).

The reason for extending the block is to enhance the low frequency characteristics of the pattern to increase the robustness against various attacks, especially the reduction transformation. In addition, by using a value obtained by dividing the height of the image by a constant constant as an expansion coefficient, it is easy to estimate the size of the forensic mark even when the image is enlarged or reduced.

)을 생성한다.The final pattern generation module 114 repeats the basic pattern extended through the pattern expansion module 113 horizontally and vertically twice, respectively, to obtain the final pattern (

).

The repeated pattern increases the amount of the forensic mark estimation pattern extracted at the same time upon detection by four times, thereby calculating a high reliability correlation value. Therefore, the effect of improving the invisibility can also be exhibited by making the intensity relatively weaker than the method of inserting one forensic mark.

The pattern position setting module 115 sets the insertion position of the final pattern.

Locating robust forensic marks plays an important role in detection, even in unattacked or under attack situations.

In the present invention, since the commonly used 16: 9 and 4: 3 images are considered, the position of the inserted forensic mark can be known by inserting the forensic mark at a predetermined position.

In particular, when a 16: 9 image is cut into a 4: 3 image, the insertion position should be adjusted so that the forensic mark can be stably detected based on the horizontal length of the two images. The insertion position for each image ratio is shown in FIG. 5.

The 16: 9 image was inserted into 1/4 of the width and length, and the 4: 3 image was inserted into 1/6 and 1/4 of the width, so that the position of the robust forensic mark could be determined according to the aspect ratio of the incoming image.

)을 전체 프레임을 일정 시간 구간을 가지는 프레임으로 분할하고, 분할된 각각의 프레임에 삽입한다.The final pattern insertion module 116 is the final pattern (

) Is divided into frames having a predetermined time interval, and inserted into each divided frame.

That is, the information to be inserted into one frame group is 2 bits, and the entire frame is divided into frame groups having a predetermined time interval in order to insert a total payload of 40 bits.

Next, we use 20 different patterns to insert 40-bit messages and 20 secret keys to generate them.

This is to increase the robustness against multiple averaging attacks. Twenty different secret keys are generated by combining the information of the forensic mark itself to be inserted and the forensic mark bit information divided into twenty for the purpose of extracting the forensic mark without additional information. In order to prevent the forensic mark pattern generated by the same secret key for different forensic marks, the key is changed by using forensic mark information in 8-bit units, and the exponent power of 2, which represents the forensic mark pattern divided by 2 bits in 8 bits, is used. By accumulating and adding constants, a secret key was created to avoid overlapping as much as possible.

When generating a pattern for the current character (8bit), it generates a different secret key value depending on the character inserted immediately before. A secret key value is generated by dividing 2 groups into 4 groups by 2 bits within one character (8 bits). To generate unique key values within a character, unique key values are generated by adding unique constants according to the order and value of each two bits. For example, when inserting a 40-bit message with the ASCII values 114, 45, 54, 52, and 49, the first 2 bits of the key value for 114 will add up all ASCII values and divide by 16, the remainder of the promised initial value of 100. In addition, it is 110 generated.

[Table 1] is a table of constants used for secret key generation. Based on 110, refer to [Table 1] to generate key value by adding appropriate values for the next three pairs of 2 bits.

For the next 8 bits, generate the key for the first 2 bits by adding the promised key value 100 to the number 16 used to obtain the first key value when extracting the forensic mark and the decimal value of the immediately preceding 8 bits (114 here). Refer to -1 to generate key value for the next 6 bits. Similarly, key values are also generated for the next three 8-bit information.

2 bit
value 1st and 2nd bit 3rd and 4th bit 5th, 6th bit 7,8th bit 00 2 ⁰ 2 ⁴ 2 ⁸ 2 ¹² 01 2 in ¹ 2 ⁵ 2 ⁹ 2 ¹³ 10 2 ² 2 ⁶ 2 ¹⁰ 2 ¹⁴ 11 2 ³ 2 ⁷ 2 ¹¹ 2 ¹⁵

In addition, the fragile forensic mark pattern generator 120 generates a basic pattern block based on an input image, generates a secret key for each basic pattern block, and generates a final pattern using the secret key. As shown in FIG. 6, the frame includes a fragile basic pattern block generation module 121, a secret key generation module 122, and a final pattern generation module 123.

Basically, the size of the semi-fragile forensic mark basic pattern is the same as the size of the input image.

Therefore, the fragile basic pattern block generation module 121 divides the input image into blocks. That is, the basic pattern, which is the size of the input image, is divided into blocks having a size of 80 * 60 for images of 1280 * 720 or more, and divided into blocks having a size of 80 * 45 for images of less than that.

For example, the number of semi-fragile forensic mark pattern blocks inserted into each frame for a 1920 * 1080 image is (1920/80) * (1080/60), that is, 432.

The secret key generation module 122 generates a secret key for each block generated through the basic pattern block generation module 121.

, 공간적 위치 인덱스 값

, 프레임당 평균 할당 시간

의 조합으로 이루어진다.The secret key is a temporal position index value of the block as shown in [Equation 1].

Spatial position index value

Average allocation time per frame

Is made up of a combination.

[Equation 1]

값에 일정한 숫자를 곱하고 나누어 줌으로써 비밀키값이 가질 수 있는 숫자의 범위 이내에서 최대한 비밀 키가 겹치지 않도록 설계하였다. 일반적으로 비밀키는 최소 0부터 최대 2³²-1의 범위를 가진다. As shown in [Formula 1],

By multiplying and dividing a value by a certain number, it is designed so that the secret key does not overlap as much as possible within the range of numbers that a secret key value can have. In general, secret keys range from a minimum of zero to a maximum of 2 ³² -1.

는 일정 수의 프레임을 하나로 묶어 순차적으로 프레임 그룹에 부여된 인덱스를 의미한다. 본 발명에서는 프레임들을 8초 단위로 묶어 인덱싱(indexing)한다. 또한, 영상의 끝부분을 알리기 위해 마지막 프레임 그룹에 대해서는

를 상수

로 고정한다. Temporal position index

Denotes an index assigned to a frame group sequentially by grouping a certain number of frames into one. In the present invention, the frames are indexed by grouping them in units of 8 seconds. Also, to inform the end of the image, the last frame group is

Constant

.

는 패턴 블록의 두 개의 열(column)을 기준으로 부여한다. 통상적인 개념으로는 1, 2번째 열은 0, 다음 3, 4번째 열은 1과 같은 식으로 인덱싱을 하지만, 공간축 절삭공격 등에 민감성을 높이기 위해 인덱싱을 쉬프트(shift)하여 진행한다. 즉 3, 4번째 열의

값을 0으로, 다음 5, 6번째 열의

값을 1로, 1, 2번째 열의

값을 마지막 인덱스로 정한다.Spatial location index

Is given based on two columns of the pattern block. Conventionally, the first and second columns are indexed as 0, the next 3 and 4 columns are 1, but the indexing is shifted in order to increase sensitivity to a space cutting attack. That is, the third and fourth columns

The value to 0, then in the fifth and sixth columns

The value to 1, the first and second columns

Set the value to the last index.

값을 사용함으로써 프레임률 변환 공격에 대한 민감성을 높인다.Finally, average allocation time per frame

Using the value increases the susceptibility to frame rate conversion attacks.

7 is a schematic diagram of all the above processes.

The final pattern generation module 123 uses the secret key for each block generated by the secret key generation module 122, similarly to robust forensic mark pattern generation, 80 * 60 or 80 * 45 size along the Gaussian distribution. Create a pattern block for.

) 을 생성한다.
In addition, after the pattern is generated for all the pattern blocks corresponding to one frame size, the final pattern (

)

In order to strongly insert forensic marks while satisfying invisibility, an understanding of the human cognitive system is required. Therefore, when the forensic mark is inserted into the input image, the intensity of the pattern should be determined based on the Human Visual System (HVS).

Accordingly, the masking unit 130 performs a function of obtaining a complex masking value that determines the strength of the pattern with respect to the forensic mark pattern insertion position. As shown in FIG. 8, the masking module 131 is NVF (Noise Visibility Fuction) masking module 131. , A brightness masking module 132 and a motion masking module 133.

The Noise Visibility Fuction (NVF) masking module 131 calculates a function indicating the degree of noise (NVF) when noise is added to an image.

는 지역별 분산(local variance)을,

는 영상에서의 최대 지역별 분산(maximum local variance)을, D∈[50, 150] 는 실험적 조정 값을 나타내고 있다.That is, the NVF masking module 131 calculates an NVF value through the following [Equation 2]. here,

Is the local variance,

Represents the maximum local variance in the image, and D∈ [50, 150] represents the experimental adjustment value.

[Equation 2]

In the region where there is no change in the image, the NVF value increases because noise insertion is more noticeable than in the region where the change in image is relatively large. Thus, the NVF value represents the degree of noise visible between 0 and 1 with the highest value 1 and the smallest value 0 for a given image.

The noise inserted by this method is suitable for the forensic mark insertion scheme proposed in the present invention because the noise inserted by the Laplacian filter, which plays a role similar to the NVF value, survives the noise removal attack and is also excellent in visibility.

) 을 [수식 3] 을 통해 계산한다.In addition, the NVF masking module 131 is a local weight value (inserted into the actual forensic mark based on the NVF value calculated through Equation 2) (

) Is calculated using Equation 3.

[Equation 3]

은 영상에서 상대적으로 변화가 적은 영역에서의 삽입 세기 하한치를 의미하고,

은 상대적으로 변화가 많은 영역에서의 삽입 세기 상한치를 의미한다.here,

Means the lower limit of the insertion intensity in the relatively small area of the image,

Is the upper limit of insertion strength in a relatively variable area.

The weight derived from Equation 3 above is multiplied by the other two masking weights described below to become the weight of the final forensic mark insertion.

을 다양한 고려 사항을 염두에 두어 선택해야 한다. In particular, the NVF value has a large effect on the overall weight.

Should be chosen with various considerations in mind.

값은 PSNR(Peak Signal to Noise Ratio), 포렌식 마크 가시성에 대한 주관적 고려, 포렌식 마크의 용도에 따른 세기를 바탕으로 결정할 수 있다.

The value may be determined based on the peak signal to noise ratio (PSNR), subjective consideration of forensic mark visibility, and intensity depending on the use of the forensic mark.

However, the process of obtaining the weight through NVF operation is a high time complexity. Therefore, the NVF weight was obtained by reducing the image to 1/2 each of the width and length, and then again using the original size to reduce the computation time to utilize the system configuration corresponding to real time.

Human visual perception systems are generally insensitive to changes in brightness in dark or bright areas.

9 shows the allowable change amount according to the brightness value. As shown in FIG. 9, the brightness value is sensitive to the amount of change in the vicinity of 75 and tends to be insensitive to the brightness value in the light or dark areas. Thus, by increasing the forensic mark intensity in relatively bright and dark areas, both invisibility and toughness can be satisfied at the same time.

) 을 실험적 결과를 통해 다음의 [수식 4] 와 같이 계산할 수 있다. Accordingly, the luminance masking module 132 has a brightness weighting value having a weight for each section based on this principle.

) Can be calculated from Equation 4 through experimental results.

[Equation 4]

는 영상의

좌표의 픽셀 값을 의미한다.here,

Of the footage

The pixel value of the coordinate.

According to the human visual recognition system for moving pictures, the human eye is less sensitive to noise in areas of high motion than in areas of low motion.

Therefore, the forensic mark can be strongly inserted while satisfying the invisibility by increasing the intensity of the forensic mark for a portion having a large difference by comparing pixel values of the front and rear frames.

) 을 실험적 결과를 통해 [수식 5] 와 같이 계산할 수 있다. Accordingly, based on this principle, the motion masking module 133 divides the difference value between two consecutive pixels into intervals to determine a motion weight value (

) Can be calculated as shown in [Equation 5] through experimental results.

[Equation 5]

는 전 프레임의 위치

에서의 픽셀 값을 나타낸다.
here,

Is the position of the previous frame

Represents a pixel value in.

), fragile 포렌식 마크 패턴 생성부(120)를 통해 생성된 최종패턴(

) 및 상기 마스킹부(130)를 통해 계산된 각각의 가중치(

)를 이용하여, 최종적으로 원본영상에 삽입할 삽입패턴을 생성하고, 생성된 삽입패턴을 원본영상에 삽입한다. The forensic mark pattern insertion unit 140 may include a final pattern (generated through the robust forensic mark pattern generation unit 110).

), the final pattern generated through the fragile forensic mark pattern generation unit 120 (

) And the respective weights calculated by the masking unit 130 (

Finally, an insertion pattern to be finally inserted into the original image is generated, and the generated insertion pattern is inserted into the original image.

를 서로 곱한 값이 최종적 가중치가 되며, 이 가중치는 포렌식 마크가 삽입된 영상의 품질을 좌우하기 때문에 이러한 값들을 실험적 결과로 조절하여 PSNR기준 및 주관적 비가시성을 충분히 만족시켜야 한다. here,

Are multiplied by the final weight, and this weight affects the quality of the image in which the forensic mark is inserted. Therefore, these values should be adjusted experimentally to satisfy the PSNR criterion and subjective invisibility.

) 는 다음의 [수식 6] 과 같이 나타낼 수 있다.Therefore, the image in which the final insertion pattern as described above is inserted in the original image (X)

) Can be expressed as Equation 6 below.

[Equation 6]

Forensic mark extraction can be largely divided into an extraction method that requires an original and an extraction method that does not require an original. The extraction method that requires the original shows a high extraction performance by extracting the forensic mark by using the difference between the forensic mark inserted image and the original image, but the practical value is low in that the original is required. On the other hand, the method that does not require the original requires a consideration of the forensic mark extraction method, and the extraction performance is relatively low, but it is a very useful method in practical terms. Therefore, the forensic mark extraction system proposed by the present invention uses an extraction method that does not require an original image.

,

)을 구하고, 최종 추정 패턴과 참조패턴 사이의 상관 연산을 수행하여 robust 포렌식 마크 패턴 및 fragile 포렌식 마크 패턴을 추출하는 기능을 수행하는 바, 상기 도 2 에 도시된 바와 같이 포렌식 마크 패턴 추정부(210), 포렌식 마크 패턴 축적부(220), robust 포렌식 마크 패턴 추출부(230) 및 fragile 포렌식 마크 패턴 추출부(240)를 포함한다. Meanwhile, the forensic mark extractor 200 may extract and accumulate the final forensic mark pattern (

,

) And extracting a robust forensic mark pattern and a fragile forensic mark pattern by performing a correlation operation between the final estimated pattern and the reference pattern, as shown in FIG. 2. ), A forensic mark pattern accumulator 220, a robust forensic mark pattern extractor 230, and a fragile forensic mark pattern extractor 240.

The forensic mark pattern estimator 210 estimates the forensic mark pattern by receiving an input in units of frames from the inspection target image.

In this case, since the forensic mark is noise for the original image, the forensic mark may be extracted through an image processing filter that removes noise or extracts only noise. In the proposed system, the noise is removed from the image using an adaptive wiener filter. The equation for the adaptive Wiener filter is shown in Equation 7 below.

[Equation 7]

는 잡음이 제거된 영상,

와

는 각각 Y 의 지역 평균, 지역 분산을 의미한다.

는 잡음의 분산을 의미하지만 현실적으로 구하기 힘든 점이 있어 본 발명에서는 검출 대상 영상 Y 의 지역 분산들의 평균값으로 근사하였다.Where Y is a detection target image,

Is the noise-free image,

Wow

Are the local mean and regional variance of Y, respectively.

Denotes a variance of noise, but it is difficult to obtain in reality, and thus, the present invention approximates an average value of local variances of the detection target image Y.

) 의 차이를 계산하고, n번째 프레임 포렌식 마크 추정영상(

) 을 [수식 8] 을 통해 계산한다.That is, the forensic mark pattern estimator 210 detects the forensic mark original image Y and an image from which the noise obtained in [Equation 7] is removed (

) And calculate the nth frame forensic mark estimation image (

) Is calculated using Equation 8.

[Equation 8]

,

) 를 구한다.The forensic mark pattern accumulator 220 accumulates during the frame group period used for robust forensic mark insertion and semi-fragile forensic mark insertion by using the forensic mark estimated in each frame through the forensic mark pattern estimator 210. Final forensic mark pattern (

,

)

을 구하기 위해 삽입 시와 마찬가지로, 검사대상 영상이 16:9영상의 경우 가로 방향 1/4지점, 세로 방향 1/4지점을 좌상단 지점으로 하여 세로 길이(입력영상의 높이/180)*60*2, 가로 길이(입력영상의 높이)*80*2 크기의 사각형으로 얻어진다. At this time,

As in the case of the insertion, the vertical length (the height of the input image / 180) * 60 * 2 is used for the 16: 9 image. , Rectangles are obtained with the width (the height of the input image) * 80 * 2.

Similarly, when the inspection target image is a 4: 3 image, the horizontal 1/6 point and the vertical 1/4 point become the upper left end. In this way, robust forensic marks can be accurately estimated even when the image is scaled down or enlarged by a certain ratio.

) 을 입력영상의 높이/180배 만큼 축소한 후, 삽입 시와 반대로 도 10 과 같이 폴딩(folding)하여 80*60크기의 최종 추정 패턴(

) 을 구한다. The forensic mark pattern accumulating unit 220 obtains the final forensic mark pattern (

) Is reduced by the height / 180 times of the input image, and then folded as shown in FIG.

)

As such, the detection rate can be increased compared to the still image due to the accumulation of forensic marks repeatedly inserted in the frame group.

Forensic mark extraction from the extracted and accumulated estimated forensic marks is performed by obtaining a correlation between the reference forensic mark and the forensic mark estimated image.

Before we can correlate it, we need to find out the secret key information and use that information to generate a forensic mark pattern in order.

)을 생성하고, 최종 추정 패턴(

)과, 참조 기본패턴(

) 사이의 상호 상관 연산을 수행하여 마크 추출 여부를 결정하는 기능을 수행하는 바, 도 11 에 도시된 바와 같이 마크 패턴 생성모듈(231) 및 마크 삽입 판단모듈(232)을 포함한다. The robust forensic mark pattern extracting unit 230 uses the reference basic pattern (see FIG.

) And the final estimated pattern (

), And the reference base pattern (

And a mark pattern generation module 231 and a mark insertion determination module 232 as shown in FIG. 11.

The mark pattern generation module 231 sequentially generates 16 patterns from the initial secret key value input from the user to extract the first 2 bits. Then, the basic pattern, the vertical axis inversion pattern, the horizontal axis inversion pattern, and the two axis inversion pattern, which represent the 2-bit values 00, 01, 10, and 11, are additionally generated from each of the sixteen patterns.

For example, if the initial key value is 100, a pattern having a key value of 100 to 115 is generated, and four inverted patterns are generated to generate a total of 64 patterns.

Compute the forensic mark pattern estimated to find the first 2 bits, and the correlation between these 64 patterns, and the key value of the pattern with the highest maximum value and the inversion axis for each of the first character (8 bits). Key value, and the first 2 bits can be found from the index of the rotation pattern.

The secret key value for the forensic mark pattern to be generated to detect the next forensic mark 2 bits is determined by adding the value by finding the value in the constant table of the forensic mark insertion.

In other words, the first character key value in the constant table and the value indicated by the index according to the order and value of the 2 bit forensic mark just detected are added. The pattern generated with the key values thus obtained is additionally generated four rotation patterns and used for the next forensic mark extraction.

When forensic mark extraction is completed in character units (8 bits), a secret key value of the next character (8 bits) is generated. The secret key value for a character (8 bits) can be found only when the previous 8 bits are fully detected.

The secret key value for the characters except the first character is set to the initial key value plus 16, the number after the 16 patterns to know the first key value. All.

)과, 약속된 비밀키를 이용하여 생성한 참조 기본패턴(

) 사이의 정규화 상호 상관 연산(normalized cross correlation)을 수행한다. The mark insertion determining module 232 obtains the obtained final estimation pattern (

) And a reference base pattern created using the promised secret key (

Perform a normalized cross correlation between

Equation 9 represents a normal normalized cross correlation operation for a pattern of size n * m.

[Equation 9]

However, the above equations have very large computational complexity and are not suitable for systems that frequently use cross-correlation operations. Therefore, the computational complexity is greatly reduced by using FFT (Fast Fourier Transform) as shown in Equation 10 below.

In the formula, * means complex conjugate.

[Equation 10]

) 을 이용하여 가장 큰 값을 갖는 최종 상관도 값(

)을 다음의 [수식 11] 에서 구해진 임계치 (

) 와 비교하여 마크 삽입 여부를 판단하고, 추출 여부를 결정한다. Also, the mark insertion determination module 232 calculates the calculated correlation value (

) With the highest correlation value ()

) Is obtained from the following formula (11)

), It is determined whether or not to insert a mark, and whether or not to extract the mark.

[Equation 11]

는

의 평균,

는

의 표준편차를 나타내고,

는 긍정적 오류(false positive error)에 따라 결정된다.
here,

The

Mean,

The

Represents the standard deviation of

Is determined according to a false positive error.

)와, 참조패턴(

) 와의 상관관계를 구함으로써 이루어진다.
Semi-fragile forensic mark pattern extraction is similar to that of robust forensic marks.

), And the reference pattern (

) By correlating

The fragile forensic mark pattern extractor 240 generates a key and a reference pattern based on a temporal / spatial index and an allocation time per frame, and checks whether or not there is a last frame of the image, thereby determining whether to deform or not. 12, a reference pattern generation module 241 and a deformation determination module 242 are included.

)을 소정 블록 사이즈로 분할하며, 각 블록에 대해 그 블록의 시/공간적 위치 인덱스 및 전체 영상의 프레임당 할당 시간에 기초하여 키를 생성한 후, 그 키를 이용하여 참조패턴(

)을 생성한다. Reference pattern generation module 241 is a semi-fragile final forensic mark pattern obtained by accumulating for a predetermined period (

) Is divided into predetermined block sizes, and a key is generated for each block based on the spatio-temporal position index of the block and the allocation time per frame of the entire image, and then the reference pattern (

).

)은 블록단위로 위에 설명된 정규화 상호 상관 연산을 이용하여 검출 여부를 판별하게 된다.
These reference patterns (

) Is determined on a block-by-block basis using the normalized cross correlation operation described above.

In general, for all blocks in one frame group, it is determined whether to deform using semi-fragile forensic mark extraction results.

When the ratio of the number of unextracted blocks to the total number of blocks in the frame group is defined as the non-detection rate, the non-detection rate generally has a value close to 1 in size conversion, frame rate conversion, and time / space axis cutting attack.

값의 동기화가 어긋나 낮은 검출률을 보이게 된다. The reason for showing such a high detection rate is that in case of size deformation, as shown in (b) of FIG. 13, the synchronization between blocks is reduced due to the reduction, and thus the detection rate is low, and in the cutting attack from 16: 9 to 4: 3, Spatial index as the left and right sides of the image are cut off

The value is out of synchronization, resulting in a low detection rate.

값이 달라져 불검출률이 높아진고, 시간축 절삭은 시간적 인덱스

값의 동기화가 어긋나 불검출률이 높아지게 된다. 그러나, 이러한 과정만으로는 영상의 마지막 부분이 잘려져 나갔을 경우 변형 여부를 판단할 수 없다. Also, because frame rate changes change the average allocation time per frame

Different values increase the detection rate, and time-base cutting is the temporal index.

The value is out of synchronization, resulting in a high detection rate. However, this process alone cannot determine whether the image is deformed when the last part of the image is cut off.

를 정해진 상수로 고정하여, 영상의 마지막 프레임의 존재여부를 검사함으로써, 변형 여부를 판단한다.Accordingly, the deformation determination module 242 determines the temporal index during the last frame period.

Is fixed to a predetermined constant and the presence or absence of the last frame of the image is checked to determine whether or not deformation.

Therefore, if the non-detection rate is high in the last frame group of the image, it means that the end portion of the image is cut out, and thus it is possible to determine whether or not the deformation is performed.

Finally, the logo insertion is weaker in the area where the logo is inserted, resulting in a relatively low detection rate compared to other blocks, and thus it is possible to determine whether or not to change the format of the logo. By comparing with the format information, it is possible to determine whether or not the modification.

Experimentally, it was found that the non-deformation image was formed around 0.1 instead of 0 on average due to attenuation of the semi-fragile forensic mark signal generated during the initial compression process. Based on these facts, the deformation detection procedure in the present invention is as shown in FIG. 14.

Specifically, as shown in FIG. 14, the deformation determining module 242 first determines whether to convert the resolution, the time / space axis cutting, and the frame rate conversion. To judge.

If there is no problem in this process, finally, the process of 3 detects whether there is a partial deformation. In the process of 3, assuming that there are more unmodified blocks than the deformed block, the block having a predetermined degree of correlation with respect to the average of all blocks is estimated as the deformed block. Thresholds used in each process used Th1 = 0.8 and Th2 = 0.9 by experimental statistics.

While the present invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It will be appreciated by those skilled in the art that numerous changes and modifications may be made without departing from the invention. Accordingly, all such appropriate modifications and changes, and equivalents thereof, should be regarded as within the scope of the present invention.

S: complex forensic watermarking system
100: forensic mark inserter 110: robust forensic mark pattern generator
111: robust basic pattern generation module 112: payload pattern generation module
113: pattern expansion module 114: final pattern generation module
115: pattern position setting module 116: final pattern insertion module
120: fragile forensic mark pattern generation unit 121: fragile basic pattern block generation module
122: secret key generation module 123: final pattern generation module
130: masking unit 140: forensic mark pattern insertion unit
200: forensic mark extractor 210: forensic mark pattern estimation unit
220: forensic mark pattern accumulating unit 230: robust forensic mark pattern extracting unit
231: mark pattern generation module 232: mark insertion determination module
240: fragile mark pattern extraction unit 241: reference pattern generation module
242: Deformation determination module

Claims (17)

In a complex forensic watermarking system,
Create a basic pattern using a secret key and a message, and expand and repeat the basic pattern horizontally and vertically to obtain the final pattern (

Robust forensic mark pattern generation unit 110 for generating a basic pattern block based on an input image, a secret key for each basic pattern block, and a final pattern using the secret key.

Fragile forensic mark pattern generation unit 120 for generating a), a masking unit 130 for obtaining a complex masking value for determining the intensity of the pattern with respect to the forensic mark pattern insertion position, and the robust forensic mark pattern generation unit 110 The final pattern generated by

), the final pattern generated through the fragile forensic mark pattern generation unit 120 (

) And the respective weights calculated by the masking unit 130 (

A forensic mark inserter 100 including a forensic mark pattern inserting unit 140 for finally generating an insertion pattern to be inserted into the original image and inserting the generated insertion pattern into the original image using a); And
By using a forensic mark pattern estimator 210 for receiving a forensic mark pattern from the inspection target image in units of frames and a forensic mark pattern estimated in each frame through the forensic mark pattern estimator 210, The final forensic mark pattern (accumulated during the frame group period used for forensic mark insertion and semi-fragile forensic mark insertion) is accumulated.

,

Forensic mark pattern accumulator 220 for obtaining the reference pattern and reference base pattern (

) And the final estimated pattern (

), And the reference base pattern (

Robust forensic mark pattern extraction unit 230 that determines whether to extract a mark by performing cross-correlation operation between the first and second frames, and generates a key and a reference pattern based on a temporal / spatial index and an allocation time per frame. Forensic mark extractor 200 including a fragile forensic mark pattern extraction unit 240 to determine whether the deformation by checking the presence of; Complex forensic watermarking system comprising a.
delete The method of claim 1,
The robust forensic mark pattern generation unit 110,
A robust basic pattern generation module 111 for generating a basic pattern according to a Gaussian distribution having a mean of 0 and a variance of 1 using a secret key and a message;
A payload pattern generation module (112) for containing bit information, and generating the basic pattern and the basic pattern in the form of vertical axis inversion, horizontal axis inversion, and horizontal / vertical axis inversion;
A pattern expansion module 113 for extending the basic pattern horizontally and vertically, respectively;
Repeating the basic pattern extended through the pattern expansion module 113 in the horizontal, vertical, respectively, the final pattern (

A final pattern generation module 114 for generating a);
A pattern position setting module 115 for setting an insertion position of the final pattern; And
The final pattern (

) Is a final pattern insertion module 116 for dividing the entire frame into a frame having a predetermined time interval, and inserting each frame into each divided frame; Complex forensic watermarking system comprising a.
The method of claim 1,
The fragile forensic mark pattern generation unit 120,
A fragile basic pattern block generation module 121 for dividing an input image into blocks;
A secret key generation module 122 for generating a secret key for each block generated by the basic pattern block generation module 121; And
After generating a pattern block according to a Gaussian distribution using a secret key for each block generated by the secret key generation module 122, and generating a pattern for all pattern blocks corresponding to one frame size, Final pattern (

A final pattern generation module 123 for generating); Complex forensic watermarking system comprising a.
The method of claim 4, wherein
The secret key,
Equation 1 shows the temporal position index value of the block.

Spatial position index value

Average allocation time per frame

Complex forensic watermarking system, characterized in that consisting of.
[Equation 1]

The method of claim 1,
The masking unit 130,
Calculates how visible noise (NVF) is when adding noise to the image, and based on the calculated NVF value, the local weight value (inserted into the actual forensic mark)

NVF masking module 131 that calculates a);
Brightness weight value weighted by interval (

Luminance masking module 132 for calculating (); And
Motion weighting value by dividing the difference value between two consecutive pixels into intervals

Motion masking module 133 to calculate a; Complex forensic watermarking system comprising a.
The method according to claim 6,
The NVF masking module 131 calculates an NVF value through Equation 2 below.
[Equation 2]

here,

Is the local variance,

Is the maximum local variance in the image, and D∈ [50, 150] is the experimental adjustment value.
The method according to claim 6,
The NVF masking module 131,
Local weight value inserted in the actual forensic mark based on the calculated NVF value (

) Is calculated by [Equation 3] complex forensic watermarking system.
[Equation 3]

here,

Is the lower limit of insertion strength in the relatively small region of the image,

Is the upper limit of insertion strength in a relatively variable area.
The method according to claim 6,
The luminance masking module 132,
Brightness weighting value having weights for each section through [Equation 4]

Complex forensic watermarking system,
[Equation 4]

here,

Of the footage

Pixel value of the coordinate.
The method according to claim 6,
The motion masking module 133,
Equation 5 below divides the difference value between two consecutive pixels into intervals to determine the motion weight value (

Complex forensic watermarking system,
[Equation 5]

here,

Is the position of the previous frame

The pixel value at.
The method of claim 1,
The forensic mark pattern insertion unit 140,
The image where the final insertion pattern is inserted in the original image (X)

) Is a complex forensic watermarking system characterized in that the following formula [6].
[Equation 6]

delete The method of claim 1,
The forensic mark pattern estimation unit 210,
In order to estimate the forensic mark, the original image to be detected (Y) and the image from which the noise is removed (

) And calculate the nth frame forensic mark estimation image (

) Is calculated by the following [Equation 8].
Equation 8

Where Y is a detection target image,

Is the noise-free image,

Wow

Are the local mean, regional variance,

Is the variance of the noise (average of regional variances in image Y).
The method of claim 1,
The forensic mark pattern accumulator 220,
Final forensic mark pattern (

), And then fold and collapse the final estimated pattern (

Complex forensic watermarking system, characterized in that
The method of claim 1,
The robust forensic mark pattern extraction unit 230,
A predetermined number of patterns are generated sequentially from the initial secret key value input from the user, and the basic pattern representing the 2-bit values 00, 01, 10, and 11 from each pattern, the vertical axis inversion pattern, the horizontal axis inversion pattern, and the two axis inversion pattern A mark pattern generation module 231 for additionally generating; And
The final estimate pattern

) And a reference base pattern created using the promised secret key (

A mark insertion determining module 232 for performing a normalized cross correlation between the < RTI ID = 0.0 > Complex forensic watermarking system comprising a.
The method of claim 15,
The mark insertion determination module 232,
Calculated correlation value (

) With the highest correlation value ()

) Is the threshold value obtained from Equation 11 below.

Comprehensive forensic watermarking system characterized in that it determines whether the mark is inserted, and whether to extract it compared to).
[Equation 11]

here,

The

Mean,

The

Standard deviation of,

Is a value determined by a false positive error.
The method of claim 1,
The fragile forensic mark pattern extraction unit 240,
The semi-fragile final forensic mark pattern obtained by accumulating for a certain period (

) Is divided into predetermined block sizes, and a key is generated for each block based on the spatio-temporal position index of the block and the allocation time per frame of the entire image, and then the reference pattern (

A reference pattern generation module 241 for generating (); And
Temporal index over the last frame interval

Deformation determination module 242 to determine whether the deformation by fixing the constant to a predetermined constant, the presence of the last frame of the image; Complex forensic watermarking system comprising a.

Images