JP2000067210A - Device and method for processing image and providing medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a watermark pattern without the reduction of detection accuracy, prolongation of detection time, degradation of picture quality and increase of detection throughput. SOLUTION: This image processor is provided with a detecting means (step S54) for detecting the number of pixels to be correlated with a watermark symbol included in pixels within the range of encoding and a correcting means (step S55) for correcting additional information corresponding to the detected number of pixel values to be correlated with the watermark symbol.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a providing medium, and more particularly, to an image processing apparatus, an image processing method, and a providing medium for processing an image having additional information embedded in the image. About.

[0002]

2. Description of the Related Art There is a technique in which information to be added to specific image data (still image or moving image sequence) is added to the image data, and the additional information is detected and used during reproduction. A typical example of this is the addition of copyright information.

When an unspecified user can use specific image data, a person who has a copyright to the image must first copy the copyright information to the image in order to claim the right. It must be added to the data. By adding the copyright information, if the copyright information that should render the image data undisplayable is detected in the processing procedure of the image playback device or playback method, the image data is displayed. It is possible to take countermeasures such as the absence.

[0004] The addition or detection of the above-mentioned copyright information is often used at present to prevent illegal copying of video tapes. Recently, there are many shops that rent video tapes, but if many users enjoy illegally copying video tapes borrowed from the store at low prices, those who have copyrights of the video tapes and videos Damage to stores that rent tapes is enormous.

[0005] Since image data is recorded on a video tape in an analog manner, image quality is degraded when copying is performed. Therefore, when copying is repeated a plurality of times, it is very difficult to maintain the originally held image quality.

[0006] On the other hand, in a device or the like that digitally records and reproduces image data which has recently become widespread,
The damage from unauthorized copying is even greater. This is because, in a device that handles image data digitally, the image quality is not deteriorated by copying in principle.
Therefore, prevention of unauthorized copying in a device that performs digital processing is much more important than in the case of an analog device.

There are two main methods for adding information to be added to image data to the image data.

The first method is a method of adding image data to an auxiliary part. For example, in a video tape, as shown in FIG. 11, the auxiliary information of the image data is displayed on the upper part (auxiliary part) 2 of the screen which is not substantially displayed on the display screen 1.
It is recorded in. It is possible to add additional information using a part of this area.

The second method is a method of adding the image data to a main part (part that is substantially displayed). This is because, as shown in FIG. 12, a certain pattern (watermark pattern) (in FIG. 12, the image WM of the personal computer) is not visually perceivable, but the entire image or one of Part. As a specific example, there is a spread spectrum for adding or detecting information by using a key pattern generated by using a random number, an M sequence, or the like.

An example of adding or detecting additional information to a main part of image data when a watermark pattern is used will be described below. This specific example is shown in FIGS.

Assume that for each pixel, a watermark pattern corresponding to 4n × 4n pixels is used, as shown in FIG. 13, which takes either one of two symbols, plus or minus. It is desirable that the watermark pattern take one of the two symbols at random, and the shape and size of the region may be arbitrary.

When the additional information is added, an area having the same size as the area of the watermark pattern is set on the image to be added. The set area and the watermark pattern are overlapped and illuminated, and a value a is added to a pixel corresponding to a plus symbol, and a value b is subtracted from a pixel corresponding to a minus symbol. Both a and b may be arbitrary values, but should be constant throughout the watermark pattern.

In the examples of FIGS. 14 to 16, a = 1 and b =
As shown in FIG. 14, when the pixel values in the area to be added are all 100, the embedding operation corresponding to the watermark pattern causes the pixel value to be 101 in the plus area, as shown in FIG. And the pixel value in the minus area.

When detecting the additional information, an area having the same size as the area of the watermark pattern is set on the image to be detected. The value obtained by summing all the pixel values in this area is used as the evaluation value. When all the pixel values are summed, the set area and the watermark pattern are superimposed and illuminated, and addition is performed for a pixel corresponding to a plus symbol and subtraction is performed for a pixel corresponding to a minus symbol. In the example of FIG. 15, the pixel having the pixel value 101 is added, and the value 101 × 2n × 4n
Is obtained, and a pixel having a pixel value of 99 is subtracted to obtain a value (−99).
.Times.2n.times.4n) are obtained, and their operation results are added to obtain a value of 2.times.2n.times.4n. At this time, unless the same pattern as the watermark pattern used when adding the additional information is used, the additional information cannot be correctly detected. By such a detection operation, for example, as shown in FIG. 15, the evaluation value when additional information is added becomes (4n) ＾ 2 (the same as the number of pixels included in the area), and FIG. As shown, the evaluation value when no additional information is added is 0 (= 100 × 4n × 2n−100 × 4).
n × 2n).

When the area of the watermark pattern is sufficiently large and the watermark pattern is sufficiently random, the evaluation value when no additional information is added is almost zero. Therefore, when the evaluation value exceeds a certain threshold value, it can be determined that additional information has been added. According to the above procedure, 1-bit additional information can be added. When it is desired to add more information, 2 ＾ k kinds (k bits) of information can be added by dividing the entire image into k regions and performing the above-described operations. .

As the watermark pattern, for example, a pattern generated using an M-sequence can be used. The M-sequence (longest code sequence) is a sequence of binary symbols of 0 and 1, where the statistical distribution of 0 and 1 is constant, and the code correlation is 1 at the origin, and in other cases it is inversely proportional to the code length. is there.
Of course, the watermark pattern may be generated by a method other than the M-sequence.

By the way, an image is recorded as a digital signal,
When playing back, the amount of information is very large as it is,
Images are typically compressed. As a method for compressing an image, JPEG (Joi
nt Photographic Experts Group) or MPEG (Moving Picture Experts Grou
High-efficiency coding schemes such as p) have been standardized internationally and have been put to practical use. Data compressed by these high-efficiency coding methods is referred to as bit stream data or simply bit stream. In contrast, an image before compression and an image after decoding are referred to as a baseband image or simply a baseband.

In either case, the baseband image to which the watermark pattern is added is distributed as it is at the baseband or as a bit stream after high-efficiency encoding such as JPEG or MPEG. However, it would be advantageous if the watermark pattern could be easily detected.

As a method of realizing this, the present applicant has
In Japanese Patent Application No. 9-252819, a watermark pattern is converted to DC (Direct Current) in a bit stream.
A method is proposed in which components are added so that they can be detected from the components. In this method, 8 × 8 pixel DCT (Discrete Cosine)
A watermark pattern is added to the baseband image so as to be reflected on the DC component of a coding block such as a Transformation (discrete cosine transform) block. By replacing “pixel”, which is an additional unit in the above description of adding a watermark pattern, with “DC component of the encoded block”, the encoded block of the watermark pattern can be replaced.
Addition to the DC component becomes possible.

An image to which such a watermark pattern is added can be distributed as a baseband image.
Even if the watermark is distributed as a bit stream that has been highly coded by MPEG or the like, the watermark pattern can be easily detected. That is, in the case of detection from a baseband image, an average value of pixel values is obtained for each region corresponding to the coding block, and a watermark pattern is detected by correlating the obtained average value with the watermark pattern. You. In the case of detection from a bit stream, a watermark pattern is detected by scanning the bit stream, decoding the DC component value of each encoded block, and correlating the DC component value with the watermark pattern.

In an encoding system such as MPEG2, two modes of DCT, a frame mode and a field mode, are used.
Is used adaptively. FIG. 17 shows a block for performing frame DCT. When performing frame DCT, FIG.
A macroblock 11 of 16 × 16 pixels shown in FIG.
Is divided into 8 × 8 pixel frame mode blocks 12 to 15 in which top fields and bottom fields shown in FIG. 17B are alternately arranged. On the other hand, when field DCT is performed, 1 shown in FIG.
The macroblock 11 of 6 × 16 pixels is divided into field mode blocks 21 and 22 composed of a top field and field mode blocks 23 and 24 of 8 × 8 pixels composed of a bottom field shown in FIG. Divided.

As shown in FIGS. 17 and 18, the range of pixels occupied by the frame mode blocks 12 to 15 or the field mode blocks 21 to 24 is different. Therefore, 8 × 16, which is a combination of two DCT blocks vertically
A watermark pattern is added to the DC component using a pixel area as one unit. With this, regardless of whether the DCT is performed in either the frame mode or the field mode,
By decoding and averaging the DC components of the two blocks, the DC component of the area of 8 × 16 pixels is calculated, and the watermark pattern can be detected.

By the way, as shown in FIG. 19, the aspect ratio of the image (the ratio of the width to the length of the image) is usually set to the NTSC (National Television Systems Committee) video image 26 (FIG. 19 ( A)) is 4: 3, but an image 27 such as a movie (FIG. 19B) is 1
6: 9. As described above, since the aspect ratio differs between a video and a movie, the following two methods are mainly used as a method for recording an image of a movie on a video.

In a first method, an image 27 such as a movie shown in FIG. 20A is converted into an image 27 as shown in FIG.
This is a method in which the image 27 is displayed in a 16: 9 area in the center portion, leaving the top and bottom of the video image. The portions that are left above and below are usually filled with black. This display method is called a letter box display, and the image is called a letter box image 29.

In the second method, as shown in FIG. 20B, the image 27 is reduced to 3/4 times in the horizontal direction and displayed over the entire video image (4: 3 area). It is. The video image 27 is shrunk in the horizontal direction, and this display method is called a squeeze display, and the image is called a squeeze image 28.

The conversion from the squeeze image 28 to the letterbox image 29 is called letterbox conversion. The letterbox conversion is a conversion for reducing the squeeze image 28 in the vertical direction by a factor of 3/4 as shown in FIGS. 20 (A) and 20 (B). The squeeze image 28 is contracted in the horizontal direction, and the object represented on the image is represented finer than it actually is. Therefore, when the image is viewed, the squeeze image 28 is subjected to the letterbox conversion. The letterbox image 29 has an advantage that it can be viewed if it is displayed as it is. On the other hand, the squeeze image 28 has an advantage that the image quality is good because the vertical resolution of the video image frame is effectively used. On the other hand, the letterbox image 29 has inferior pixels in the upper and lower parts, so that the image quality is inferior to the squeeze image 28.

When a movie image is recorded on a video, it is often recorded as a squeeze image 28 of good image quality. MPEG
For example, a squeeze image 28 created from a movie image may be encoded as it is as a squeeze image 28, or may be subjected to letterbox conversion so that it can be displayed and viewed as it is after decoding. May be encoded as

Here, the squeeze image 28 recorded with the watermark pattern added thereto is the squeeze image 2
Whether the encoding is performed as 8 or after the letterbox conversion is performed and then the encoding is performed as the letterbox image 29, if the watermark pattern can be detected from the bit stream, the handling is easy.

Conventionally, an encoding device 31 and a decoding device 32 using a watermark pattern shown in FIG.
Has been realized. The encoding device 31 receives a baseband image as the letterbox image 29. When there is no need to add a watermark pattern, the input baseband image is supplied to the encoding unit 44 via the switching unit 43.
When it is necessary to add a watermark pattern, the baseband image is added to the watermark pattern adding unit 41.
And a watermark pattern is added. When the letter-box conversion is not performed, the baseband image to which the watermark pattern is added is sent to the encoding unit 44 via the switching unit 43 as it is. When letter-box conversion is performed, the baseband image to which the watermark pattern is added is letter-box converted by a letter-box conversion unit 42, which is a vertical reduction filter, and supplied to an encoding unit 44 via a switching unit 43. Is done. The encoding unit 44 encodes the input baseband image and outputs the obtained bit stream. The bit stream output from the encoding device 31 is supplied to the decoding device 32 via a predetermined transmission path.

The watermark pattern detecting section 52 of the decoding device 32 receives the bit stream output from the encoding device 31, detects information from the watermark pattern added to the image of the bit stream, and detects the detected information. Is output to the output control unit 51. The output control unit 51 sends the encoding device 3 to the decoding unit 53 based on the signal from the watermark pattern detection unit 52.
1 controls the supply of the bit stream supplied from 1.
The decoding unit 53 decodes the input bit stream,
Output a baseband image.

In the following, the processing performed in the unauthorized use prevention system including the encoding device 31 and the decoding device 32 using the watermark pattern shown in FIG. 21 will be described in more detail.

As shown in FIG. 22A, the additional blocks 61-1 to 61-n in which the watermark pattern is added to the DC coefficient in the squeeze image 28, as shown in FIG. , In the letterbox image 29 after the letterbox conversion,
Areas 62-1 to 62 -n. for that reason,
The additional blocks of the letterbox image 29 are shifted from the additional blocks 61-1 to 61-n of the squeeze image 28, and the additional blocks 61-1 to 61-n of the squeeze image 28 are shifted.
The watermark pattern added to the DC coefficient is not directly reflected on the DC coefficient of the additional block of the letterbox image 29. Therefore, when encoding is performed before performing the letterbox conversion, a watermark pattern for the squeeze image 28 that can detect a watermark pattern from the bit stream, and when encoding is performed after performing the letterbox conversion, It has been conceived to add both a watermark pattern for a letterbox image from which a watermark pattern can be detected to a baseband image.

The additional blocks 61-1 to 61-n of the watermark pattern for the squeeze image 28 are shown in FIG.
As shown in (A), this is an area of 8 × 16 pixels which is the same as the additional block of the normal watermark pattern. On the other hand, as shown in FIG. 23A, the additional blocks 71-1 to 71-n of the watermark pattern for the letterbox image 29 are 8 × 21.333.
/ 3) Consists of pixels. Additional blocks 71-1 to 7
1-n after letterbox conversion is performed as shown in FIG.
8 × 16 pixel blocks 72-1 to 72-n.

The process performed by the conventional unauthorized use prevention system of FIG. 21 is a process of adding these two watermark patterns, and is a process in which both processes of FIGS. 24 and 24 are sequentially performed. In addition, the order of the additional processing may be either one.

FIG. 24 is a flow chart for explaining a process for adding a normal (for the squeeze image 28) watermark pattern. First, in step S11, the watermark pattern adding unit 41 sets a watermark pattern and sets additional amounts a and b. In step S12, the watermark pattern adding unit 41 reads one block of 8 × 16 pixels corresponding to a normal watermark pattern adding block. In step S13, the watermark pattern adding unit 41 determines whether the block is a block to which a watermark pattern is to be added, and determines that the block is a block to which a watermark pattern is to be added. The process proceeds to step S14.

In step S14, the watermark pattern adding section 41 determines whether or not the read block is a block to be added. If it is determined that the read block is a block to be added, the process proceeds to step S14.
The process proceeds to step S15, where a is added to the DC value of the block, and then proceeds to step S17. If it is determined in step S14 that the read block is a block from which the watermark is to be subtracted, the process proceeds to step S16, where the watermark pattern adding unit 41 subtracts b from the DC value of the read block, and proceeds to step S17. move on. Step S
At 17, the watermark pattern adding unit 41
It is determined whether or not processing of all blocks has been completed. If it is determined that processing of all blocks has not been completed, the process returns to step S12 to continue the processing.

If it is determined in step S13 that the block is not a block to which a watermark pattern is to be added, the process proceeds to step S17.

If it is determined in step S17 that processing of all blocks has been completed, the watermark pattern adding unit 41 ends the processing.

FIG. 25 is a flowchart for explaining a process for adding a watermark pattern for the letterbox image 29. In step S21, the watermark pattern adding unit 41 sets a watermark pattern for a letter box and sets the added amounts a and b.
Make the settings for In step S22, the watermark pattern adding unit 41 sets the 8 × 21.33 corresponding to the watermark pattern adding block for the letter box.
3. Read one block of pixels. The following step S
The procedure from step S23 to step S27 is the same as that from step S13 to step S17, and a description thereof will be omitted.

In the procedures shown in FIGS. 24 and 25, the following two methods are used to select a block to which a normal watermark pattern is to be added and a block to which a watermark pattern for letterbox is to be added. is there.

The first method is to add two watermark patterns spatially or temporally to the DC component of another encoded block. For example, as a method of spatially adding a DC component of another encoded block, a watermark pattern for a squeeze image 28 is added to the right half of the image, and a watermark pattern for a letterbox image 29 is added to the left half of the image. There is a way to do that. As a method of temporally adding the DC component of another encoded block, a watermark pattern for the squeeze image 28 is provided in the even-numbered frames, and a letterbox image 29 is provided in the odd-numbered frames.
There is a method of adding a watermark pattern for each.

The second method is to add two watermark patterns to the same DC component both spatially and temporally. In this method, both watermark patterns are added to the whole area of the image, but it is necessary to use different watermark patterns for the squeeze image 28 and the letterbox image 29.

FIG. 26 is a flowchart for explaining the detection process. First, in step S31, the watermark pattern detection unit 52 sets a watermark pattern for the squeeze image 28 in W [x, y], and sets a watermark pattern for the letterbox image 29 in WL [x, y]. The evaluation value sum for setting the watermark pattern detection of the squeeze image 28 and the letterbox image 2
Evaluation value sumL for detecting watermark pattern No. 9
Is initialized to 0, the threshold th for detecting the watermark pattern of the squeeze image 28, and the threshold thL for detecting the watermark pattern of the letterbox image 29.
Set. In step S32, the watermark pattern detection unit 52 simply decodes the bit stream and substitutes the DC component of the block of 8 × 16 pixels into I [x, y]. Here, I [x, y] is the DC component of the y-th block from the top and the x-th block from the left in the image. Step S33
In the watermark pattern detection unit 52,
And add I [x, y] × W [x, y] to sumL and add I [x, y] × WL [x, y] to sumL
Is added. In step S34, the watermark pattern detection unit 52 determines whether or not processing has been performed for all I [x, y], and determines that processing has not been performed for all I [x, y]. If so, the procedure proceeds to step S
Returning to step 33, the process is repeated.

If it is determined in step S34 that the processing has been executed for all I [x, y], step S3
5, the watermark pattern detection unit 52 determines that the sum
Is determined to be greater than th, and if it is determined that sum is not greater than th, the process proceeds to step S36. In step S36, the watermark pattern detection unit 52
It is determined whether or not sumL is greater than thL. If it is determined that sumL is not greater than thL, the procedure proceeds to step S37, where the watermark pattern detection unit 52 determines whether the watermark pattern has not been added. Execute the process and end the process.

In step S35, when it is determined that sum is greater than th, and in step S36,
If it is determined that sumL is greater than thL, the watermark pattern detection unit 52 executes the processing when the watermark pattern is added in step S38, and ends the processing.

Since it is not known which watermark pattern is included in each block before detecting the watermark pattern, it is necessary to correlate both watermark patterns with all blocks.

[0047]

The first conventional method is as follows.
An encoding block for extracting a DC component used for detecting a watermark pattern is divided into two uses: an encoding block for detection from the letterbox image 29 and an encoding block for detection from the squeeze image 28. Therefore, in order to make the detection time the same as the detection time for only the letterbox image 29 (or the squeeze image 28), the number of DC components usable for the detection is reduced by half, and the detection accuracy is deteriorated. Conversely, in order to make the detection accuracy the same as that for only the letterbox image 29 (or the squeeze image 28), the number of DC components that can be used for detection in the letterbox image 29 and the squeeze image 28 is separately secured. Requires twice the detection time.

In the second conventional method, since signals of two watermark patterns are added to an image so as to be superimposed on each other, the pattern of the watermark pattern can be easily seen as noise added to the original image, and the image quality can be reduced. to degrade.

The present invention has been made in view of such a situation, and it is an object of the present invention to detect a watermark pattern without lowering detection accuracy, increasing detection time, deteriorating image quality, or increasing detection processing amount. Aim.

[0050]

An image processing apparatus according to a first aspect of the present invention includes a detecting means for detecting a watermark symbol included in at least a part of an encoded block, a detected watermark symbol, and a code. Calculating means for calculating the evaluation value of the water block according to the coefficient of the conversion block.

According to a fifth aspect of the present invention, there is provided an image processing method comprising the steps of: detecting a watermark symbol included in at least a part of an encoded block; and detecting the watermark symbol and a coefficient of the encoded block. hand,
A calculating step of calculating an evaluation value of the water block.

According to a sixth aspect of the present invention, there is provided the providing medium, comprising: a detecting step of detecting a watermark symbol included in at least a part of the encoded block; and the detected watermark symbol and a coefficient of the encoded block. And a computer readable program for executing a process including a calculating step of calculating an evaluation value of the water block.

The image processing apparatus according to claim 1,
In the image processing method described in (1) and the providing medium described in (6), a watermark symbol included in at least a part of the coding block is detected, and the detected watermark symbol and a coefficient of the coding block are detected. , The evaluation value of the water block is calculated.

[0054]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below. In order to clarify the correspondence between each means of the invention described in the claims and the following embodiments, each means is described. When the features of the present invention are described by adding the corresponding embodiment (however, an example) in parentheses after the parentheses, the result is as follows. However, of course, this description does not mean that each means is limited to those described.

That is, the image processing apparatus according to the first aspect of the present invention provides a detecting means for detecting a watermark symbol included in at least a part of an encoded block (for example, FIG. 6).
(Step S54 of the processing of FIG. 6) and calculating means (for example, step S55 of the processing of FIG. 6) for calculating the evaluation value of the water block according to the detected watermark symbol and the coefficient of the coding block. It is characterized by.

Hereinafter, assuming MPEG2 which adaptively uses DCT of a frame and a field, description will be made using a block of 8 × 16 pixels as a block for adding or detecting a watermark pattern.

The configuration of the unauthorized use prevention system of the present invention is as follows.
This is the same as the configuration of the unauthorized use prevention system shown in FIG.

FIG. 1 is a diagram showing a case where the squeeze image 28 is reduced in the vertical direction during the letterbox conversion, and the upper part of the screen is filled so that only the lower part is left. FIG. 1 (A)
Represents an additional block 81 of 8 × 16 pixels (x in the horizontal direction) for adding a watermark pattern in the squeeze image 28.
FIG. 21 is a diagram showing a (th, y-th block in the vertical direction).
When the squeeze image 28 including the additional block 81 is subjected to the letterbox conversion (by 3/4 in the vertical direction), the additional block 81 is an additional block 82 of 8 × 12 pixels on the letterbox image 29 shown in FIG. Becomes FIG.
(C) shows encoding blocks 83-1 to 83- of 8 × 16 pixels when encoding the letterbox image 29.
FIG. Note that y ′ in the drawing represents an integer.

FIG. 2 is a diagram showing a case where the squeeze image 28 is reduced in the vertical direction and the margins are set at the top and bottom of the screen during letterbox conversion. FIG. 2A is a diagram showing an additional block 81 of 8 × 16 pixels for adding a watermark pattern in the squeeze image 28. When the squeeze image 28 including the block 81 is subjected to letterbox conversion, the additional block 81 becomes an additional block 82 of 8 × 12 pixels on the letterbox image 29 shown in FIG. 2B. FIG. 2C is a diagram showing encoding blocks 83-1 to 83-3 of 8 × 16 pixels when the letterbox image 29 is encoded.

Hereinafter, a method of detecting a watermark pattern in the case of FIG. 1 will be described with reference to FIG. FIG.
5 shows a row of one vertical additional block 81 extracted from the image shown in FIG. FIG.
(A) shows the additional blocks 81-1 to 81-4 to which the watermark pattern is added in the squeeze image 28, and the additional blocks 81-1 divided every 8 × 16 pixels.
Symbols of a watermark pattern are assigned to the DC components 81 through 81-4, respectively. W [x, y] is the coordinates x, y of the watermark pattern (the coordinates of the block, the x coordinate is an integer part when divided by the x coordinate of the pixel, and the y coordinate is the y value of the pixel The integer part when the coordinates are divided by 16 is represented by a symbol). The coordinates have the origin at the upper left of the screen and start from 0 in both the vertical and horizontal directions. For example, the symbol of the watermark pattern corresponding to the block of 8 × 16 pixels at the upper left of the screen is W
[0,0], and the symbol of the watermark pattern corresponding to the block below it is W [0,1].

When the squeeze image 28 is subjected to letterbox conversion, the vertical resolution becomes ３. Therefore, FIG.
2 (A) or the watermark addition block 81 in the squeeze image 28 in FIG. 2 (A) is the watermark addition block 8 of 8 × 12 pixels shown in FIG. 3 (B).
The conversion is made to 2-1 to 82-4 (the y coordinate of the block is an integer part when the y coordinate of the pixel is divided by 12).

When the letterbox image 29 is encoded,
The DCT in the frame mode is performed twice or the DCT in the field mode is performed twice every 8 × 16 pixels of the letter box image 29.
(The DCT is performed for each 8.times.8 pixel). Now, the average value of the DC coefficients of the two DCTs of the coding blocks 83-1 to 83-3 shown in FIG. 3C is represented by I [x, y]. That is, I [x, y] is
It is assumed that the value represents the DC value of the DCT coefficient of the portion at the coordinates x, y of each block of 8 × 16 pixels.

In this example, between the additional blocks 82-1 to 82-4 shown in FIG. 3B and the coding blocks 83-1 to 83-3 shown in FIG. The sum of the parts corresponding to each other is calculated, and the watermark pattern is detected. That is, the component of the watermark pattern is calculated by the following equation (1).

Sum = Σ (I [x, 0] × W [x, 0]) + Σ (I [x, 0] × W [x, 1]) + Σ (I [x, 1] × W [x, 1]) + Σ (I [x, 1] × W [x, 2]) + Σ (I [x, 2] × W [x, 2]) + Σ (I [x, 2] × W [x, 3] ) (1) In the above equation, Σ means a product-sum operation for all x.

Of the pixels included in the encoding block 83-1, 12/16 of the total pixels are watermark symbols W
As a result of being affected by [x, 0] and a total of 4/16 pixels being affected by the watermark symbol W [x, 1],
Assuming that the DC component value I0 [x, y] of the squeeze image 28 has changed to I [x, y], I [x, 0] becomes I [x, 0] = I0 [x, 0] +12 / 16 × W [x, 0] + 4/16 × W [x,
1]. Therefore, the first term of the equation (1) is expressed by the equation (2)
Can be transformed as follows.

Σ (I [x, 0] × W [x, 0]) = Σ ((I0 [x, 0] + 12/16 × W [x, 0] + 4/16 × W [x, 1]) × W [x, 0]) Σ ((12/16 × W [x, 0] + 4/16 × W [x, 1]) × W [x, 0]) Σ (12/16 × W [x, 0] × W [x, 0]) = 3/4 Σ (W [x, 0] × W [x, 0]) (2)

Σ ((12/16 × W [x, 0] + 4/16 × W [x, 1]) × W [x,
0]) is that ΣI0 [x, 0] × W [x, 0] 0 holds,
That is, the DC component of the image and the watermark are generally based on a lack of correlation. Also, Σ (12/16 × W
The transformation to [x, 0] × W [x, 0]) is ΣW [x, 1] × W [x, 0] 0,
That is, the watermark is based on the fact that there is no correlation with the watermark shifted in phase.

Similarly, when the other terms of equation (1) are transformed, equation (3) is obtained.

[0069]

As described above, the evaluation value of the added watermark pattern is calculated for each of the coding blocks 83-1 to 83-3.
-3, its DC component and its coding block 83
-1 to 83-3 are obtained by taking the product with the watermark pattern symbol added to at least a part of at least one of them and performing a summation (by performing a product-sum operation). The reason why the evaluation value of the watermark is reduced to 3/4 is that the evaluation value is reduced to 3/4 vertically by the letterbox conversion.

Next, in the case of FIG. 2, that is, the calculation of the evaluation value of the watermark in the case where the image is reduced to 3/4 in the vertical direction at the time of the letterbox conversion, and the margins are formed above and below the screen will be described. . As shown in FIG.
It is assumed that the number of pixels in the margin at the top of the screen is y0. FIG. 4 (A)
Is a block 82 for adding a watermark in the letterbox image 29. FIG. 4B is a diagram showing a pixel 91 in one vertical column of the additional block 82 extracted.

The y-coordinate of the pixels 91 in one vertical column included in one additional block 82 of the letterbox image 29 is y0
The range is from + 12y to y0 + 12y + 11. FIG. 4C is a diagram illustrating pixels in the encoding blocks 83-1 to 83-3 when encoding as the letterbox image 29. In the pixels 91 in one vertical column, the encoding blocks 83-1 to 83-3 are composed of 8 × 16 pixels.
3 included. Therefore, for example, the y coordinate (pixel coordinate)
Is a pixel whose y coordinate (block coordinate) is (y0 + 12y).
12y) // 16. Further, the pixel whose y coordinate is y0 + 12y + 11 has a y coordinate of (y0 + 12y + 11) //
It is included in 16 coding blocks 83. here,
Represents division that rounds down the decimal point.

The size of the coding block 83 is 1 in the vertical direction.
Since the additional block to which the watermark symbol W [x, y] having the same value is added has only 12 pixels in the vertical direction while having a spread of 6 pixels, y
The pixels of the additional block whose coordinates are between y0 + 12y and y0 + 12y + 11 are the coding blocks 83 whose y coordinate is (y0 + 12y) // 16 or the y coordinate is (y0 + 12y + 11) / / 16 encoding block 83. Therefore, the DC component of the coding block 83 to be correlated with W [x, y] is (y0 + 12y) // 16 = (y0 + 12y + 11) //
If 16 (one additional block corresponds to one coded block), then only I [x, (y0 + 12y) // 16],
If (y0 + 12y) // 16 ≠ (y0 + 12y + 11) // 16 (one additional block corresponds to two encoded blocks), then I
[x, (y0 + 12y) // 16] and I [x, (y0 + 12y + 11) // 16].

Next, the process of adding a watermark pattern and the process of detection will be described. 24 and 25
In the conventional example shown in the flowchart of FIG. 2, two different watermarks for the squeeze image 28 and the letterbox image 29 are added, whereas the present invention is simplified in that only one watermark is added. Also, it is simplified that there is no need to determine which watermark is to be added.

FIG. 5 is a flowchart for explaining the process of adding a watermark pattern. Step S41
In, the watermark pattern adding section 41 sets the watermark patterns and sets the additional amounts a and b. In step S42, the watermark pattern adding unit 41 assigns one 8 × 16 pixel block corresponding to the normal watermark pattern adding block to one.
Read. In step S43, the watermark pattern adding unit 41 determines whether the read block is a block to which a watermark is to be added, and if it is determined that the read block is a block to which a watermark is to be added, The process proceeds to S44, a is added to the DC value of the block, and the process proceeds to Step S46.

If it is determined in step S43 that the read block is a block from which the watermark is to be subtracted, the process proceeds to step S45, where the watermark pattern adding unit 41 sets the DC value of the read block to b.
, And the process proceeds to step S46. In step S46, the watermark pattern adding unit 41 determines whether or not processing of all blocks has been completed. If it is determined that processing of all blocks has not been completed, the process returns to step S42, and the processing returns to step S42. To continue.

If it is determined in step S46 that all blocks have been processed, the watermark pattern adding unit 41 ends the processing.

FIG. 6 is a flowchart for explaining the process of detecting a watermark. First, in step S51, the watermark pattern detection unit 52 sets the watermark pattern to W [x, y], evaluates the evaluation value sum for detecting the watermark pattern of the squeeze image 28, and the While initializing the evaluation value sumL for detecting the watermark pattern to 0,
The threshold th for detecting the watermark pattern of the squeeze image 28 and the threshold thL for detecting the watermark pattern of the letterbox image 29 are set. Step S52
In, the watermark pattern detection unit 52 simply decodes the bit stream and outputs a block of 8 × 16 pixels.
Substitute the DC component into I [x, y]. In step S53, the watermark pattern detection unit 52 sets the first (x,
(0,0) is selected as a set of (y), I [x, y] × W [x, y] is added to sum, and I [x, (y0 + 12y) // 16] × W is added to sumL. Add [x, y].
Now, assuming that y0 = 0, this results in I [x, 0] in FIG.
× W [x, 0] is calculated.

In step S54, the watermark pattern detection unit 52 determines that (y0 + 12y) // 16 is (y0 + 12y + 11) // 1.
Judge whether it is equal to 6 and (y0 + 12y) // 16 is (y0 + 12y + 11)
If it is determined that it is not equal to // 16 (that is, if one additional block 82 corresponds to two encoded blocks), the process proceeds to step S55, and I [x, (y0 + 12y + 11) is added to sumL. ) /
/ 16] × W [x, y], and the process proceeds to step S56.

In step S54, (y0 + 12y) // 16 is
If it is determined that it is equal to (y0 + 12y + 11) // 16 (that is, if one additional block 82 corresponds to one encoded block), the process proceeds to step S56.

In step S56, the watermark pattern detecting section 52 determines whether or not processing has been executed for all sets. If it is determined that there is a set that has not been processed, the procedure proceeds to step S53. And the subsequent processing is repeated. In the case of the example of FIG. 3, step S5
In step 4, since the values of the left and right terms of the equation are both 0, a determination of YES is made, and in step S56, y = 1 is set (the process of incrementing x is omitted here).
In step S53, I [x, 0] × W [x, 1] is added to sumL (the processing of sum is also omitted here). In step S54, the value of the left term of the equation is 0 and the value of the right term is 1, so a NO determination is made. In step S55, I [x, 1]
× W [x, 1] is added to sumL

If it is determined in step S56 that the processing has been executed for all (x, y) pairs, the process proceeds to step S57, where the watermark pattern detection unit 52
It is determined whether sum is greater than th. If it is determined that sum is not greater than th, the process proceeds to step S58. In step S58, the watermark pattern detection unit 52
Determines whether sumL is greater than thL. If it is determined that sumL is not greater than thL, the procedure proceeds to step S5.
Proceeding to step S9, the watermark pattern detection unit 52 executes a process when no watermark pattern is added, and ends the process.

If it is determined in step S57 that sum is greater than th, and in step S58,
If it is determined that sumL is greater than thL, the watermark pattern detection unit 52 executes the processing when the watermark pattern is added, and ends the processing.

As described above, in the present invention, a watermark can be detected from a squeeze image and a letterbox image only by adding one watermark to the squeeze image 28. Therefore, the two watermarks are spatially or temporally separated by the DC of another coding block.
There is no need to add to the components. Further, in the present invention, the encoding block 83 for extracting the DC component used for detecting the watermark is defined as an encoding block 83 for detection from the letterbox image 29 and an encoding block for detection from the squeeze image 28. There is no need to divide the data into 83, and there is almost no decrease in detection accuracy and almost no increase in detection time. Further, since it is not necessary to add two watermarks to the same DC component, the image quality does not deteriorate.

The processing can be further simplified by adding the parameter k to the algorithm shown in FIG. An algorithm using the parameter k will be described with reference to FIG.

The steps of the processing in FIG.
Except for step S63, step S64, and step S65, the steps in the flowchart of FIG. 6 are the same as the steps having the same last digit of the same number, and the description thereof is omitted. Will be described only. In step S61, the watermark pattern detection unit 52 adds k in addition to the processing in step S51.
Set parameters. In step S63, the watermark pattern detection unit 52 adds I [x, y] × W [x,
y], and I [x, (y0 + 12y + k) // 16] × W [x, y] is added to sumL.

In step S64, the watermark pattern detection unit 52 determines that (y0 + 12y + k) // 16 is (y0 + 12y + 11-
j) is determined as equal to (y0 + 12y + k) // 16 is (y0 +
12y + 11-k) // 16 (that is, one additional block 82 is divided into two encoded blocks 8
3), the process proceeds to step S65, and I is added to sumL.
[x, (y0 + 12y + 11-k) // 16] × W [x, y] are added, and step S
Proceed to 66.

In step S64, (y0 + 12y + k) // 16
Is determined to be equal to (y0 + 12y + 11-k) // 16 (ie, when one additional block 82 corresponds to one encoded block), the process proceeds to step S66.

In this algorithm, for example, when k = 2 is set, the coding block 83 to which the pixel to be compared belongs belongs.
Is calculated by Expressions (4) and (5). (y0 + 12y + k) // 16 = (16i + 15) // 16 = i (4) (y0 + 12y + 11-k) // 16 = 16 (i + 1) +6) // 16 = i + 1 (5)

In this case, y shown in FIG.
In order to correlate with the additional block whose coordinate is y, the DC component of the coding block 83 whose y coordinate is i indicated by the obliquely upwardly sloping line in FIG. The y coordinate indicated by the diagonally downward slanted line in C) is the DC component of the i + 1 encoding block 83.

If k = 3, the coding block 83 to which the pixel to be compared belongs is calculated by the equations (6) and (7). (y0 + 12y + k) // 16 = 16 (i + 1) // 16 = i + 1 (6) (y0 + 12y + 11-k) // 16 = (16 (i + 1) +5) // 16 = i + 1 (7)

That is, in this case, the correlation used with the additional block whose y coordinate is y is shown in FIG.
Is the only DC component of the coding block of i + 1, and the
The component is not used for calculating the correlation value.

FIG. 9 shows additional blocks 82-1 to 82-4 and coding block 83-1 when k = 3 is set.
It is a figure explaining product-sum operation between 83-4. In this example, coding blocks 83-1 to 83-4 are:
One additional pixel is shifted upward from the additional blocks 82-1 to 82-4. Product-sum operation I [x, 0] × W [x, for a portion where the additional block 82-2 and the encoding block 83-1 overlap.
1] is omitted. Similarly, a product-sum operation is performed on a portion where the additional block 82-4 and the encoding block 83-4 overlap.
I [x, 3] × W [x, 3] is also omitted.

Similarly, when k is any other value, an additional block 82 (y additional block in FIG. 8B) whose y coordinate is y and an encoding block 83 (FIG. 8B) whose y coordinate is i.
The correlation with the DC component of the coding block to which the pixels 16i + 12 to 16i + 15 (C) belong is obtained when k is 2 or less. In the example of FIG. 8, the additional block 8 whose y coordinate is y
The number of pixels corresponding (overlapping) to 2 in the encoding block 83 whose y coordinate is i is 3 pixels (16i + 13, 16i + 14, 16i + 1)
5). In other words, when the number of pixels corresponding to the pixels of the additional block 82 and the pixels of the coding block 83 is equal to or smaller than the value of the parameter k, the pixels are determined to be non-corresponding (non-overlapping), and the correlation is calculated. It is made not to be used.

The watermark component is included only in pixels overlapping the additional block in the coding block 83. When the k parameter is used, k /
When only 16 or less pixels have a watermark component, by not performing correlation calculation, it is possible to omit the process of calculating the correlation for a portion where the watermark component is unlikely to be detected. Thus, the amount of processing can be reduced without substantially changing the calculated correlation value.

For example, if k = 6, the correlation is calculated only when at least half of the additional block 82 corresponds to the coding block 83.

As described above, the process using the k parameter eliminates the process of calculating the correlation for the portion where the watermark component is unlikely to be detected, and the processing amount is hardly changed without substantially changing the calculated correlation value. Can be reduced.

FIG. 10 is a flowchart for explaining a process of detecting a watermark component obtained by adding a normalization process to the process shown in the flowchart of FIG. This processing is to multiply the detection value by k / 16 when the watermark component is included only in the pixels at the rate of k / 16 in the encoding block 83. The steps of the process in FIG. 10 are the same as those in the flowchart of FIG. 6 except for steps S73 and S75, where the last digit of the number is the same, and the description thereof is omitted.
Hereinafter, only the parts different from the processing in FIG. 6 will be described.

In step S73, the watermark pattern detecting section 52 adds I [x, y] × W [x, y] to sum and I [x, (y0 + 12y) // 16] × sumL. W [x, y] × NOTMAX (12,16-
Add (y0 + 12y)% 16) / 16. In step S75, the watermark pattern detection unit 52 adds I to sumL.
[x, (y0 + 12y + 11) // 16] × W [x, y] × ((y0 + 12y + 11)% 16 + 1) / 1
Add 6. Note that NOTMAX (A, B) represents the smaller one of A and B, and% represents the remainder after division.

As described above, the processing in FIG. 10 is to correct the watermark evaluation value obtained by calculating the correlation between the watermark and the DC component by the ratio of the overlap of the coding block 83 with the additional block. Thereby, detection with higher reliability is enabled. Also, letterbox image 2
Each value added when obtaining the evaluation value sumL for 9 must be 1
Since it is divided by 6, if the threshold value is set to 16 times in advance, the operation of dividing by 16 is unnecessary.

As is apparent from FIG. 4, the ratio of the pixels of the coding block 83 corresponding to a certain additional block 82 is a value between 1/16 and 12/16 of the coding block 83. When the ratio of the pixels of the coding block 83 corresponding to the additional block 82 is 1/16 and 12/16 of the coding block 83, the watermark component that can be extracted when the correlation is obtained. Are 12 times different. Therefore, the ratio of the pixels of the coding block 83 corresponding to the additional block 82 is
When the ratio is 1/16, the influence of noise increases, making determination difficult. Therefore, in this processing example, the additional block 82
Is multiplied by a value of the pixel ratio of the coding block 83 corresponding to the value obtained by correlation with the coding block 83.
This makes it possible to correct that the reliability of the watermark component differs depending on the ratio of the pixels of the coding block corresponding to the additional block.

In the above description, the size of the coded block and the size of the additional block 83 in the squeeze image are set to 8 × 16 pixels. However, other numbers of pixels may be used.
For example, to make the block size 4 × 16 pixels,
When a watermark is added to an image, the addition process is performed for each area of 4 × 16 pixels instead of 8 × 16 pixels. When a watermark of an image is detected, in addition to DC, the minimum order is added only in the horizontal direction. AC (Alternating Cu
rrent: AC coefficient (coefficient on the right side of DC component, 8 ×
(Corresponding to the difference between the average value of the right half and the left half of the 16-pixel block), the watermark can be detected.

In order to make the block size 8 × 8 pixels, a watermark is added for each 8 × 8 pixel area. At the time of detection, for the DCT block in the field mode, in addition to DC, the lowest AC coefficient only in the vertical direction (the coefficient located below the DC coefficient, the average of the upper half and lower half of the block) (Corresponding to the difference between the values) to obtain a DC value in a range of 8 × 8 pixels. For DCT block in frame mode, 8
Since there is a DC value for each area of × 8 pixels, no special processing is required.

The unit for adding a watermark (the size of an additional block) is determined in such a manner that the watermark can be detected even when the conventional letterbox conversion is performed.
This is the same unit as when a watermark for the squeeze image 28 is added. Therefore, an image to which a watermark has been added in such a manner that a watermark can be detected even when conventional letterbox conversion is performed can be detected by the image processing apparatus and method of the present invention, and is compatible.

Therefore, the letter box conversion is performed by the conventional method, and the image to which the watermark is added is
There is an advantage that it is not necessary to add a watermark again.

The watermark symbols are +1 and -1.
Any other symbols may be used. Further, three or more types of symbols may be used instead of two types. For example, using three types of symbols of +1, 0, and -1 and performing matching with a watermark pattern, a pixel having a symbol of 0 does not affect the evaluation value sum (the pixel value is evaluated). Any meaning can be given to each symbol, such as adding or subtracting from the value sum).

The region where the watermark pattern is added on the image may have any shape and range. Further, as long as the matching with the added watermark pattern is achieved, the shape and range of the area for which the evaluation value is obtained at the time of detection may be arbitrary. Further, addition or detection may be performed using information in a wide range in space or time. That is, the correlation with the front / rear or left / right in the frame may be used, or the temporally previous / next frame may be used.

In this specification, a system is
It is assumed that the device as a whole is constituted by a plurality of devices.

Further, as a providing medium for providing a user with a computer program for performing the above processing,
In addition to recording media such as magnetic disks, CD-ROMs, and solid-state memories, communication media such as networks and satellites can be used.

[0110]

According to the image processing apparatus described in claim 1, the image processing method described in claim 5, and the providing medium described in claim 6, the water included in at least a part of the encoded block. Since the mark symbol is detected and the evaluation value of the water block is calculated according to the detected watermark symbol and the coefficient of the coding block, the detection accuracy decreases, the detection time increases, the image quality deteriorates,
Alternatively, the watermark pattern can be detected without increasing the amount of detection processing.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a letter-box converted image.

FIG. 2 is a diagram illustrating a letter-box converted image.

FIG. 3 is a diagram illustrating a relationship between an additional block and an encoded block.

FIG. 4 is a diagram illustrating a relationship between pixels of an additional block and a coding block.

FIG. 5 is a flowchart illustrating a process of adding a watermark pattern.

FIG. 6 is a flowchart illustrating a process of detecting a watermark.

FIG. 7 is a flowchart illustrating a process of detecting a watermark.

FIG. 8 is a diagram illustrating a relationship between pixels of an additional block and a coding block.

FIG. 9 is a diagram illustrating a relationship between an additional block and an encoded block.

FIG. 10 is a flowchart illustrating a process of detecting a watermark.

FIG. 11 is a diagram illustrating additional information to an image.

FIG. 12 is a diagram illustrating additional information to an image.

FIG. 13 is a diagram illustrating a watermark pattern.

FIG. 14 is a diagram illustrating the addition of a watermark pattern.

FIG. 15 is a diagram illustrating detection of a watermark pattern.

FIG. 16 is a diagram illustrating detection of a watermark pattern.

FIG. 17 is a diagram illustrating a range of image encoding.

FIG. 18 is a diagram illustrating a range of image encoding.

FIG. 19 is a diagram illustrating an aspect ratio of an image.

FIG. 20 is a diagram illustrating conversion of an aspect ratio of an image.

FIG. 21 is a diagram illustrating a configuration example of a conventional unauthorized use prevention system.

FIG. 22 is a diagram illustrating an additional block of a watermark pattern.

FIG. 23 is a diagram illustrating an additional block of a watermark pattern.

FIG. 24 is a flowchart illustrating processing for adding a conventional watermark pattern.

FIG. 25 is a flowchart illustrating a conventional watermark pattern detection process.

FIG. 26 is a flowchart illustrating a conventional watermark pattern detection process.

[Explanation of symbols]

28 squeeze image, 29 letterbox image,
41 watermark pattern adding section, 52 watermark pattern detecting section, 81 additional block, 8
2 additional blocks, 83 encoded blocks

Continued on the front page (72) Inventor Yoichi Yagasaki 6-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5B050 BA06 CA04 EA04 EA10 EA12 EA19 GA07 5B057 CD05 CE08 CG02 CG05 CG07 CH09 DA08 DA17 5C063 AC10 CA01 CA40 5C076 AA14 AA40 BA06

Claims (6)

[Claims] 1. An image processing apparatus for processing an image having additional information embedded as a watermark, comprising: detecting means for detecting a watermark symbol included in at least a part of an encoded block; An image processing apparatus comprising: a mark symbol; and a calculating unit that calculates an evaluation value of the water block according to a coefficient of the encoded block. 2. The method according to claim 1, wherein the calculating unit omits the calculation of the evaluation value for the pixels of the coding block in which the deviation of the watermark symbol from the pixel is equal to or less than a predetermined number of pixels. 2. The image processing device according to 1. 3. The image processing apparatus according to claim 1, wherein the calculation unit multiplies the evaluation value by a ratio of pixels included in the coded block affected by the watermark symbol component. apparatus. 4. The image processing apparatus according to claim 1, wherein the image is a letter-box converted image. 5. An image processing method for processing an image having additional information embedded as a watermark, a detecting step of detecting a watermark symbol included in at least a part of an encoded block, and detecting the detected watermark. An image processing method, comprising: a mark symbol; and a calculating step of calculating an evaluation value of the water block according to a coefficient of the coded block. 6. An image processing apparatus for processing an image having additional information embedded as a watermark, a detecting step of detecting a watermark symbol included in at least a part of an encoded block, and a step of detecting the watermark detected. A providing medium for providing a computer-readable program for executing a process including a mark symbol and a calculating step of calculating an evaluation value of the water block according to a coefficient of the coding block.