JP2017073653A - Image processor, control method and program of image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor, a control method and a program of the image processor capable of solving a problem in a conventional method such that a plurality of analysis results of pick-up images is merged without considering stationary noises, and capable of solving degradation due to stationary noises and additional information can be correctly acquired.SOLUTION: The image processor performs a series of processing including: setting shooting conditions of an imaging apparatus; acquiring a piece of image data corresponding to a plurality of images which is obtained by shooting printing item plural times on the basis of the shooting conditions; extracting a piece of additional information from the image data to evaluate the same; determining a piece of additional information to be adopted on the basis of the evaluation result; and acquiring a piece of additional information which is extracted from entire images by merging the adopted additional information.SELECTED DRAWING: Figure 15

Description

  The present invention relates to an image processing apparatus and an image processing method. In particular, prints embedded in the image information so as to be visually inconspicuous as additional information such as information other than the image information, for example, completely different image information such as audio information, text document information, and various information about the image The present invention relates to image processing for reading the additional information from an object using an imaging medium.

In recent years, a printed matter in which additional information is multiplexed is photographed multiple times with a portable terminal. Then, by merging the analysis results of the plurality of photographed images, the influence of random noise such as camera shake of the mobile terminal is suppressed, and it is finally possible to accurately acquire additional information on the printed matter (for example, Patent Document 1).

JP 2006-50043 A

  In the conventional method, since random noise such as camera shake is present at different locations randomly for each captured image, the influence of random noise can be suppressed by merging the analysis results of a plurality of captured images. For example, if a part of the additional information multiplexed from any one of a plurality of sheets can be acquired, the additional information obtained by demultiplexing from the plurality of images can be merged to obtain the entire additional information. . However, stationary noise caused by reflection of light of the mobile terminal, lens aberration, and the like continues to exist without changing the location for each captured image. For this reason, only the conventional merging of a plurality of photographed images results in noise occurring at the same position in the merged image, and there is a possibility that additional information unintended by the user may be acquired.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus, an image processing method, and a program in consideration of stationary noise.

In order to achieve the above object, an image processing apparatus of the present invention includes a photographing unit that photographs an image in which additional information is multiplexed under a plurality of photographing conditions, and records each image data.
Extraction means for extracting additional information multiplexed from the image data of each of the plurality of shooting conditions;
Evaluation means for evaluating additional information extracted from the image data of each of the plurality of imaging conditions;
And combining means for combining the additional information of evaluation exceeding a threshold value with each other based on the evaluation by the evaluation means.

  According to the present invention, it is possible to provide a technique capable of acquiring accurate additional information by merging analysis results of a plurality of images.

1 is an overall external view of an additional information multiplexing apparatus and an additional information separating apparatus according to an embodiment. It is a principal part block diagram which shows the additional information multiplexing apparatus of FIG. It is a block diagram which shows the detail of a quantization means. It is a flowchart which shows the operation | movement procedure of printed matter preparation. It is a figure explaining the electronic watermark superimposition area | region on paper. It is a figure explaining the period of a change of a quantization threshold value. It is a principal part block diagram which shows the additional information multiplexing apparatus. It is a figure explaining the inclination of the image in the image | photographed image. It is a figure which shows the example of a spatial filter. It is a figure explaining the frequency domain. It is a flowchart which shows the operation | movement procedure of digital watermark decoding. It is a figure which shows the example of a pixel identification method. It is a figure which shows the example of a pixel identification method. It is a figure which shows the example of the stationary noise by imaging | photography of a camera. It is a flowchart which shows suppressing stationary noise. It is a figure which shows the specific example which suppresses stationary noise by setting imaging | photography condition. It is a flowchart which shows suppressing stationary noise. It is a figure which shows the specific example which suppresses stationary noise. It is a flowchart which shows suppressing stationary noise. It is a figure which shows the specific example which suppresses stationary noise.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the drawings. The image processing system according to the embodiment includes an additional information multiplexing device 102 that embeds additional information in a printed matter, a printer 103, and a camera-equipped portable terminal 104 that includes an additional information separation device that reads the additional information from the printed matter. The In the present embodiment, such an imaging device is called a camera-equipped mobile terminal. In the embodiment and the like, the term “print” or “printed material” is used, but the printed object is not limited to characters. Other objects such as photographs and line drawings are also referred to as “printing” or “printed material”, which can be referred to as “printing”, “printed material”, “output”, or “output material”.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration of an image processing system according to the first embodiment. Multi-tone image information is input from the input terminal 100, and additional information to be embedded in the image information is input from the input terminal 101. This additional information is information different from the image information, and examples thereof include audio information, moving image information, and text document information. The additional information multiplexing apparatus 102 embeds additional information in the image information so that it is difficult to visually discriminate. The additional information multiplexing apparatus 102 also multiplexes input multi-tone image information together with multiplexing of additional information.
The printer 103 visualizes the information created by the additional information multiplexing apparatus 102 on a medium by a printer engine, and outputs it as a printed matter. The printer 103 is assumed to be a printer that realizes gradation expression by using pseudo gradation processing, such as an inkjet printer or a laser printer.

  The output image on the printed matter is read using the imaging sensor 105 of the camera-equipped mobile terminal 104, and the additional information (multiplexed data) embedded in the image printed on the printed matter by the additional information separating device 106. Are output to the output terminal 107. The output terminal 107 is an interface for outputting the acquired additional information. For example, audio information is output to the speaker 108, and image information is output to the display 109. The output terminal 107 may be an interface that outputs data to an external device. When the camera-equipped mobile terminal 104 has a plurality of image sensors, the second image sensor 111 may capture the printed matter. The camera-equipped mobile terminal 104 includes a control unit 120. For example, the control unit 120 adjusts the focal position of the optical system of the camera, the light for photographing, and the like according to a user operation or a preset setting.

<Configuration of Additional Information Multiplexer 102>
FIG. 2 is a block diagram showing a configuration of the additional information multiplexing apparatus 102 in FIG.
The error diffusion unit 200 converts the image information input from the input terminal 100 into a quantization level smaller than the number of input gradations by performing pseudo gradation processing using an error diffusion method, and performs quantization of a plurality of pixels. The gradation is expressed in terms of area according to the value. Details of the error diffusion processing will be described later.
The blocking unit 201 divides the input image information into predetermined area units. The quantization condition control unit 202 changes and controls the quantization condition in units of areas blocked by the blocking unit 201. The quantization condition control unit 202 controls the quantization condition in units of blocks based on the additional information input from the input terminal 101. The quantization condition controlled and output by the quantization condition control unit 202 includes, for example, a quantization threshold. In the case of binarization, there is one quantization threshold value. However, in the case of multi-value quantization, a threshold value corresponding to the number of gradations after quantization is output. Further, the number of gradations may be included in the quantization condition.
The control unit 210 includes a CPU 211, a ROM 212, a RAM 213, and the like. The CPU 211 executes a control program stored in the ROM 212 to control the operations and processes of the above-described components such as the error diffusion unit 200 and the quantization condition control unit 202. The RAM 213 is used as a work area for the CPU 211. The blocks described above may be hardware controlled by the control unit 210, but may be logical function blocks realized by execution of a program by the control unit 210.

<Configuration of error diffusion unit 200>
FIG. 3 is a block diagram illustrating details of the error diffusion unit 200. In this embodiment, an error diffusion process in which the quantization value is binary will be described as an example.
The adder 300 adds the target pixel value of the input image information and the distributed quantization error of the peripheral pixels already binarized. The comparison unit 301 compares the quantization threshold value from the quantization condition control unit 202 with the pixel value to which the error is added by the adder 300. If the value is larger than the quantization threshold value, “1” is set. Then, “0” is output. For example, when expressing the gradation of a pixel with an accuracy of 8 bits, it is generally expressed by “255” which is the maximum value and “0” which is the minimum value.
It is assumed that dots (ink, toner, etc.) are printed on paper when the quantization value is “1” (ie, 255). The subtractor 302 calculates an error, that is, a quantization error between the quantization result and the above-described addition result of the target pixel value and the distributed error, and a future quantization process is performed based on the error distribution calculation unit 303. The error is distributed to the applied peripheral pixels. The error distribution ratio preliminarily possesses an error distribution table 304 experimentally set based on the relative distance between the pixel of interest and its surrounding pixels. The error distribution calculation unit 303 distributes the quantization error obtained by the subtracter 302 based on the distribution ratio described in the distribution table 304. The distribution table 304 in FIG. 3 shows a distribution table for four pixels that are not quantized among the eight pixels around the pixel of interest, but is not limited thereto.

<Processing procedure by quantization condition control unit and error diffusion unit>
Next, the entire operation procedure including the quantization condition control unit 202 will be described with reference to the flowchart of FIG. Now, an example in which the quantized value is binary will be described. Note that the flowchart of FIG. 4 is realized by the CPU 211 reading and executing a program related to the flowchart.
In S401, the control unit 210 initializes the variable i. The variable i is a variable for counting the vertical address of the pixel to be processed, that is, the target pixel.
In S402, the control unit 210 initializes the variable j. The variable j is a variable for counting the horizontal address of the target pixel.
Subsequently, in S403, the control unit 210 determines whether or not the coordinates (i, j), which is the address of the current pixel of interest, belong to an area where the multiplexing process is to be executed.

The multiplexed area will be described with reference to FIG. FIG. 5 shows one image in which the number of horizontal pixels is WIDTH and the number of vertical pixels is HEIGHT. The image is divided into blocks of horizontal N pixels and vertical M pixels with the upper left corner of the image as the origin. In this embodiment, the origin is used as a reference point, but the block is formed as a reference point. When the maximum information is multiplexed in this image, N × M blocks are arranged from the reference point. That is, when the number of blocks that can be arranged in the horizontal direction is W and the number of blocks that can be arranged in the vertical direction is H, the following relationship is established.
W = INT (WIDTH / N) ... Formula 1
H = INT (HEIGHT / M) ... Formula 2
However, INT () indicates the integer part of the value in ().
The number of remaining pixels that is not divisible in Equations 1 and 2 corresponds to the end when a plurality of N × M blocks are arranged in the image, and is outside the region where additional information is multiplexed, that is, outside the code multiplexing region. Become. In S403 of FIG. 4, if the target pixel is within a rectangular area of W x N in the horizontal direction and H x M in the vertical direction with respect to the origin, it is determined that it is in the code multiplexing area, otherwise It is determined that it is out of the multiplexing area. Here, N and M are the sizes of blocks by the blocking unit 201, and WIDTH and HEIGHT are the sizes of image data to be processed (that is, image data in which additional information is multiplexed).

If it is determined in S403 that the pixel of interest currently being processed is outside the multiplexing region, the control unit 210 sets the quantization condition C in S404. On the other hand, when it is determined that the pixel of interest currently being processed is within the multiplexing region, the control unit 210 reads additional information to be multiplexed in S405. For ease of explanation, it is assumed that the additional information is represented by one bit each using an array called code []. For example, assuming that the additional information is 48-bit information, the array code [] stores one bit each from code [0] to code [47]. In S405, the information in the array code [] is substituted into the variable bit as follows.
bit = code [INT (i / M) × W + INT (j / N)] ... Equation 3
Here, the index on the right side of Equation 3 indicates the order when the blocks are arranged in raster order. According to Equation 3, each bit of the additional information is associated with each block of the image data to be processed. From this, the number of bits of the additional information code is equal to or less than W × H, and preferably equal to W × H. For example, in FIG. 5, the number of blocks is 4 × 6 = 24. However, if the code is 48 bits, it is desirable to block into 8 × 6 = 48 blocks, for example.

Subsequently, in S406, the control unit 210 determines whether or not the variable bit substituted in S405 is “1”. As described above, since the information in the array code [] is stored by 1 bit for each array element, the value of the variable bit indicates either “0” or “1”.
If the control unit 210 determines “0” in S406, it sets the quantization condition A in S407, and if it determines “1”, it sets the quantization condition B in S408.
Subsequently, in S409, the control unit 210 performs a quantization process based on the set quantization condition. Note that the error diffusion unit 200 may perform the process of S409. This quantization process corresponds to the error diffusion method described in FIG.
Subsequently, in S410, the horizontal direction variable j is incremented by 1, and in S411, the control unit 210 determines whether or not the pixel of interest after the count-up is less than WIDTH, which is the number of horizontal pixels of the image. Repeat the above process until the number reaches WIDTH. When the horizontal processing is completed for the number of WIDTH pixels, the vertical variable i is incremented by 1 in S412. In step S413, the control unit 210 determines whether the pixel of interest after being counted up is less than HEIGHT, which is the number of vertical pixels of the image, and repeats the above-described processing until the number of processed pixels reaches HEIGHT.
With the operation procedure described above, it is possible to change the quantization condition in accordance with additional information in units of blocks each consisting of N × M pixels.

<Quantization conditions>
Subsequently, examples of the quantization conditions A, B, and C will be described. Although there are various factors in the quantization condition in the error diffusion method, in the first embodiment, the quantization condition is a quantization threshold. When the quantization condition C is selected, the pixel of interest is outside the multiplexing region, so any quantization threshold value can be used. As described above, when one pixel is represented by 8 bits of gradation and the quantization level is binary, the maximum value “255” and the minimum value “0” are the quantization representative values. However, “128” which is the intermediate value is often set as the quantization threshold. That is, in the quantization condition C, the quantization threshold is fixed to “128”. When the quantization condition A and the quantization condition B are used, the pixel of interest belongs to a block in the multiplexed region, and thus a difference in image quality due to a difference in quantization condition must be generated. However, the difference in image quality must be expressed so that it is difficult to distinguish visually, and it must be easily identifiable from the paper.

  FIG. 6 is an example showing the quantization conditions A and B. FIG. 6A is a diagram showing a cycle of change of the quantization threshold in the quantization condition A. In the figure, one square is one pixel, a white square is a fixed threshold, and a gray square is a variation threshold. In other words, in the example of FIG. 6A, a matrix of 8 horizontal pixels and 4 vertical pixels is assembled, and a value that protrudes only from the gray square threshold is set as the threshold. Similarly, FIG. 6B is a diagram illustrating the period of change of the quantization threshold in the quantization condition B. In the example of FIG. 6B, unlike FIG. 6A, a matrix of 4 horizontal pixels and 8 vertical pixels is assembled, and a value that protrudes only from a gray square threshold is set as the threshold.

  As described above, when one pixel is expressed by an 8-bit gradation value as described above, for example, “128” is set as the fixed threshold value, and “10” is set as the protruding threshold value. When the quantization threshold is lowered, the quantization value of the pixel of interest tends to be “1” (quantization representative value “255”). That is, in both FIGS. 6A and 6B, the quantized value “1” is easily arranged in the arrangement of gray cells in the drawing. In other words, for each N × M pixel block, there are a block in which dots are generated in the arrangement of gray squares in FIG. 6A and a block in which dots are generated in the arrangement of gray squares in FIG. 6B. Will be mixed.

  A slight change in the quantization threshold in the error diffusion method does not have a significant effect on image quality. In the systematic dither method, the image quality of the gradation expression greatly depends on the dither pattern used. However, in the error diffusion method in which the quantization threshold is regularly changed as described above, the gradation expression for determining the image quality is the error diffusion method. Therefore, even if the arrangement of dots changes slightly or the occurrence of texture changes, the image quality of gradation expression is hardly affected. This is because, even when the quantization threshold changes, the error that is the difference between the signal value and the quantized value is diffused to surrounding pixels, so that the input signal value is stored in a macro manner. In other words, the redundancy is very large with respect to dot arrangement and texture generation in the error diffusion method. In the case of color image data, quantization may be performed for each color component according to the above procedure.

In the above description, multiplexing has been realized by superimposing a predetermined periodicity representing a code on the quantization threshold of the error diffusion method. However, the following scheme is also conceivable.
-A method of directly superimposing periodicity on RGB luminance information-A method of multiplexing RGB periodicity by separating RGB luminance information into luminance-color difference information (for example, YCrCb signal)-RGB luminance information is ink color (for example, CMYK) In the multiplexing of periodicity, the quantization threshold may be used as described above.

<Configuration of additional information separator>
Next, the additional information separation device 106 in the image processing system of FIG. 1 will be described. FIG. 7 is a block diagram showing a configuration of the additional information separation device 106. For ease of explanation, an example of separation from a printed matter in which additional information of 1 bit each is multiplexed in a divided block will be described as in the example of the additional information multiplexing apparatus 102 described above. Naturally, the amount of additional information per block in the multiplexing device is equal to the amount of separation information per block in the separation device.
Image information read by the camera-equipped mobile terminal 104 is input to the input terminal 700. The resolution of the imaging sensor of the camera-equipped mobile terminal 104 to be used is preferably equal to or higher than the resolution of the printer that creates the printed matter. Of course, in order to accurately read the dot dot information of the printed matter, the imaging sensor side needs to have a resolution twice or more that of the printer side by the sampling theorem. However, if it is equal to or greater than that, it is possible to determine that dots are scattered to some extent even if it is not accurate. In the first embodiment, for ease of explanation, it is assumed that the printer resolution and the resolution of the imaging sensor are the same resolution.
The geometric shift detection unit 701 detects a geometric shift of an image captured by the camera-equipped mobile terminal 104. Since the image to be processed has undergone output processing in the printer and shooting processing in the camera-equipped mobile terminal 104, the image information input from the input terminal 700 is largely geometrically different from the image information before the printer output. There may be. Therefore, the geometric deviation detecting unit 701 detects a boundary line between the printed matter and other than the printed matter by edge detection.

  FIG. 8 is a diagram illustrating a captured image input from the input terminal 700. If the printer resolution and the imaging sensor resolution are the same resolution, the image rotation direction (tilt) is caused by the skew when recording on the paper of the printer and the deviation when the camera-equipped mobile terminal 104 is held over the printed matter. Is a major factor to be corrected. Therefore, by detecting the boundary line of the printed matter, it is possible to determine how much the deviation occurs in the rotation direction. This shift amount can be expressed by, for example, the number of pixels shifted in the vertical direction with respect to a predetermined number of pixels in the horizontal direction.

The blocking unit 702 blocks in units of P pixels in the horizontal direction and Q pixels in the vertical direction. This block must be smaller than the N × M pixels that are blocked when the digital watermark is superimposed. That is,
P ≦ N and Q ≦ M ... Formula 4
The relationship holds.
The P × Q pixel unit blocking is performed at regular intervals. That is, the block is formed so that one block of P × Q pixel unit is included in an area assumed to be a block of N × M pixels at the time of multiplexing. That is, the block pitch is basically N pixels in the horizontal direction and M pixels in the vertical direction. In this case, the gap between the blocks is NP pixels in the horizontal direction and MQ pixels in the vertical direction. However, as described above, there is a deviation between the original image and the captured image, and the deviation amount detected by the geometric deviation detection unit 701 is converted into a deviation amount per block and added to the number of skipped pixels. It is necessary to correct. For example, when determining the right P × Q block of a certain P × Q block, instead of simply determining the P × Q block so that the horizontal pitch is N pixels, N pixels are moved rightward. The vertical deviation in this case is obtained from the detected deviation amount. Then, a P × Q block is secured based on the position shifted in the vertical direction by the number of pixels.

  The spatial filters 703 and 704 have different characteristics, and the filtering unit 705 performs digital filtering that calculates a product sum with the surrounding pixels. The coefficients of the spatial filters 703 and 704 are created in accordance with the period of the variation threshold value of the quantization condition at the time of multiplexing. Now, it is assumed that the multiplexing apparatus 102 multiplexes the additional information with the image data by using the two kinds of quantization thresholds shown in FIGS. 6A and 6B according to the value of the additional information. Examples of the spatial filter A 703 and the spatial filter B 704 used in the separation device 106 at that time are shown in FIGS. In the figure, the central portion of 5 × 5 pixels is the pixel of interest, and the remaining 24 pixels are peripheral pixels. In the figure, a blank pixel indicates that the filter coefficient is “0”. As is apparent from FIG. 9, FIGS. 9A and 9B are edge enhancement filters. In addition, the directionality of the edge to be emphasized matches the directionality of the variation threshold used in the multiplexing (specifically, the directionality in which black pixels are easily arranged). That is, FIG. 9A is created to match FIG. 6A, and FIG. 9B is created to match FIG. 6B.

The pixel specifying units 706 and 707 perform a pixel position specifying process on the filtered signal (hereinafter referred to as a converted value) in a block composed of P × Q pixels based on a certain regularity. In the first embodiment, the regularity of the specified pixel position is processed by separating it into periodicity and phase. That is, the pixel specifying units 706 and 707 differ in the periodicity of the pixel positions to be specified, and each execute a plurality of pixel position specifying processes in which the phases are changed. The pixel position specifying method will be described later.
The conversion value adding unit 708 adds the conversion values at the pixel positions specified by the pixel specifying units 706 and 707 for each phase. The pixel position specifying process and the conversion value adding process are equivalent to extracting power of a predetermined frequency vector emphasized by the spatial filters 703 and 704. Here, the frequency vector is a set of frequency components in the horizontal direction and the vertical direction of the image, and indicates a two-dimensional frequency characteristic of the image.
The dispersion value calculation unit 709 calculates dispersion of a plurality of addition values added for each phase in each periodicity.
The determination unit 710 determines the multiplexed code based on the variance value in each periodicity.

  FIG. 10 is a diagram showing an outline of the first embodiment in a two-dimensional frequency domain. The horizontal axis represents the horizontal frequency, and the vertical axis represents the vertical frequency. The origin at the center indicates a direct current component, and becomes a high frequency region as the distance from the origin increases. Circles in the figure indicate cutoff frequencies due to error diffusion. The filter characteristics of the error diffusion method indicate the characteristics of an HPF (high-pass filter) in which the low frequency region is cut off, and the cut-off frequency changes according to the density of the target image. This time, the frequency characteristic generated after quantization changes due to the change of the quantization threshold, but in the image quantized using the quantization threshold shown in FIG. 6A, the frequency on the straight line A in FIG. In an image quantized with the vector and using the quantization threshold shown in FIG. 6B, a large power spectrum is generated in the frequency vector on the straight line B in FIG. That is, if the white portion of the threshold matrix in FIG. 6A is 128 and the black portion is 10, the black portion is likely to be a black pixel. For this reason, in an image quantized using this threshold matrix, it is considered that a frequency characteristic having a cycle of an integer multiple of 4 pixels in the vertical direction and a cycle of an integer multiple of 8 pixels in the horizontal direction, that is, its reciprocal, is likely to appear. . That is, from the image quantized using the threshold matrix of FIG. 6A, the power spectrum of the frequency vector in which the vertical frequency is approximately twice the horizontal frequency is detected. In FIG. 6 (b), the horizontal direction and the vertical direction are interchanged with those in FIG. 6 (a). At the time of separation of additional information, detection of a frequency vector in which this large power spectrum is generated leads to determination of a multiplexed signal. Therefore, it is necessary to emphasize and extract each frequency vector individually.

  FIGS. 9A and 9B correspond to an HPF having a specific frequency vector directionality. That is, the spatial filter in FIG. 9A can emphasize the frequency vector on the straight line A, and the spatial filter in FIG. 9B can emphasize the frequency vector on the straight line B. It becomes possible. For example, it is assumed that a large power spectrum is generated on the frequency vector of the straight line A in FIG. 10 by applying the quantization condition in FIG. At that time, the power spectrum on the straight line A is amplified by the spatial filter of FIG. 9A, but is hardly amplified by the spatial filter of FIG. 9B. In other words, when a plurality of spatial filters are filtered in parallel, amplification is performed only when the spatial vectors have the same frequency vector, and there is almost no amplification with other filters, so a large power spectrum is generated on any frequency vector. It ’s easy to see.

<Processing Procedure by Additional Information Separating Device 106>
FIG. 11 is a flowchart illustrating an operation procedure of the pixel specifying unit 706, the pixel specifying unit 707, the conversion value adding unit 708, the variance value calculating unit 709, and the determining unit 710 of FIG. The procedure of FIG. 11 is executed by each component shown in FIG. Specifically, S1103 is executed by the pixel specifying unit, S1104 is executed by the converted value adding unit, S1107 is executed by the variance value unit, and S1110 and S1111 are executed by the determining unit 710. The parameter setting process and determination process in the meantime may be performed by a control unit (not shown), for example. Further, the periodicity No. value referred to in FIG. 11 corresponds to the sequence of processing blocks from the pixel specifying portion to the variance value portion, and in FIG. is there. In other words, in FIG. 11, the periodicity No. i value is changed and processed in series. However, when realized by hardware as shown in FIG. 7, processing corresponding to different periodicity numbers is executed in parallel. Is done. Therefore, when the procedure of FIG. 11 is executed with the configuration of FIG. 7, S1108 and S1109 related to the parameter i are not performed. The process of FIG. 11 may be performed by executing a program for realizing the entire procedure of FIG. 11 by a processor such as a CPU (not shown) or a control unit. In the following description, it is assumed that the additional information separator 106 is the main operation.

In FIG. 11, in S1101 and S1102, variables are initialized, and the values of variables i and j used in this flow are initialized to zero.
Step S1103 is a step of determining regularity factors of pixel positions specified by the pixel specifying units 706 and 707, that is, two factors of “periodicity” and “phase”. In this flow, the variable related to periodicity is i, and the variable related to phase is j.
The conditions of the periodicity and phase are managed by numbers (numbers), and now the factor of the pixel position specifying method is set where the periodicity number (hereinafter abbreviated as No.) is i and the phase number is j. . This example will be described with reference to FIGS. With this factor, the position of the conversion value to be processed in the image data is specified. Note that i and j are indexes indicating periodicity and phase, respectively, but may be referred to as periodicity i and phase j below.
Subsequently, S1104 is a step of adding the converted values at the positions specified in the block. The added value is stored as a variable array TOTAL [i] [j]. Since a plurality of pixels (conversion values) are normally specified according to the parameters i and j, the total value of the conversion values at the specified position is obtained and stored in TOTAL [i] [j] in S1104.
In step S1105, the variable j is counted up and compared with a fixed value J in step S1106. In J, a predetermined number of times of performing the pixel position specifying process by changing the phase is stored. If the variable j is less than J, the process returns to S1103, and the pixel position specifying process and the adding process of the pixel (that is, the conversion value) at the specified position are repeated by the new phase number by j after counting up. It is. If the value of the index j is different, the position of the specified pixel is also different.

  When the pixel position specifying process and the adding process in which the phase is shifted are completed a set number of times, a dispersion value B [i] of the addition result TOTAL [i] [j] is calculated in S1107. That is, it is evaluated how much each addition result varies due to the phase difference. Here, i is fixed, and a variance value of J TOTAL [i] [j] from 0 to J−1 is obtained for the index j. The variance value is B [i].

In step S1108, the variable i is counted up and compared with a fixed value I in step S1109. I stores the number of pixel position specifying processes by changing the periodicity. If the variable i is less than I, the process returns to S1102, and using the new periodicity No. condition by i after counting up, the pixel position specifying process, the conversion value adding process, and the variance calculation are performed again. Is repeated. The variance value indicates a variation from the average value of the value “1” (or the maximum value of the pixel such as 255) to the pixel corresponding to the specific phase index. The smaller the variation, the smaller the value. The value increases.
If it is determined in step S1109 that i has been set a predetermined number of times, I dispersion values B [i] can be calculated. In S1110, the maximum value of the variance values is detected from the set of I variance values, and the value of i at that time is substituted into the variable imax.

  S1111 is a code determination process, and determines that the code having a periodicity No. of imax is a multiplexed code, and ends. However, if the power spectrum does not appear due to the filtering process on the image in the block, the decoding process currently being performed is not a decoding process corresponding to the multiplexing method, and it is determined that the decoding has failed, and the process ends. It is desirable to do. Conversely, if the power spectrum appears, it can be determined that the decoding is successful. This determination can be realized by determining that B [imax] does not satisfy a predetermined condition, for example. As the predetermined condition, for example, it can be adopted that B [imax] is equal to or greater than a predetermined threshold, but the variance B is not a normalized value, and a fixed threshold may not be appropriate. Therefore, in S1111, for example, the maximum possible variance value Bmax is obtained, B [imax] / Bmax is compared with a predetermined threshold value, and if it is smaller than the threshold value, it is determined that decoding has failed, and the process is terminated. Alternatively, it may be determined that decoding has failed when the predetermined condition is not satisfied with a predetermined number of consecutive blocks without determining with one block. The variance value Bmax can be obtained by the following procedure, for example. That is, for the image in which the pixel value (conversion value) at the position corresponding to the specific phase number is “1” (or the maximum value of the pixel value such as 255) and the other pixel values are 0, the procedure of FIG. The variance is obtained by using Bmax. The value is stored in association with each quantization threshold matrix and prepared. In this way, B [imax] / Bmax is normalized to take a value between 0 and 1, and can be compared with a fixed threshold value. This threshold value may be determined experimentally.

  Now, an example of I = 2 and J = 4 is shown. FIG. 12 and FIG. 13 show the method for specifying the pixel position in the form of a table when the block size is P = Q = 16. In the figure, one square in the block corresponds to one pixel. In the figure, the block shape is a square of P = Q, but it is not limited to a square and may be other than a rectangle.

  12 shows a pixel position specifying method when periodicity No. = 0 (corresponding to the pixel specifying unit A706 in FIG. 7), and FIG. 13 shows a pixel position specifying method when periodicity No. = 1 (FIG. 7). Corresponding to the pixel specifying portion B707). In the figure, the value shown for each pixel in the block indicates the pixel corresponding to phase No. j. Although the parameter j indicates the phase, the pixel position in the pixel block is specified by the periodicity and the phase. Therefore, in FIGS. 11 and 12, the value of the phase No. j is assigned to the pixel at the specified position. Marked and shown. For example, a pixel displaying “0” corresponds to a pixel at a position specified when j = 0. That is, both FIG. 12 and FIG. 13 correspond to a pixel specifying method when there are four types of phases and the phase No. j is 0-3. That is, the matrix shown in FIG. 12 or 13 is specified according to the periodicity No. value, and the pixel position in the specified matrix is specified according to the phase No. value.

  The periodicity of the pixel position corresponding to the same phase index (phase No.) shown in FIGS. 12 and 13 coincides with the periodicity of FIG. 6A and the periodicity of FIG. ing. As described above, in both FIGS. 6A and 6B, the quantized values “1” (in the case of binary values “0” and “1”) are easily arranged in the arrangement of gray cells in the drawing. . Therefore, for example, in the case of a block that has been in the quantization condition A at the time of multiplexing, the quantized value “1” is likely to be arranged with the periodicity of FIG. Filtering further amplifies the frequency component. Therefore, when the quantized value “1” is concentrated on the pixel at the position corresponding to the specific phase index, and the conversion value is specified and added with the periodicity of FIG. 12, the dispersion of the addition result increases.

  On the other hand, when the block that was under the quantization condition A is filtered using a spatial filter that does not match, and the pixel position is specified by the periodicity of FIG. Get smaller. That is, since the periodicity of the quantized value is different from the periodicity of the specified pixel position, the added value of the conversion values due to the difference in the phase of the specified pixel position becomes average, and the variation becomes small. On the contrary, in the block that was in the quantization condition B at the time of multiplexing, the dispersion value is small in FIG. 12, and the dispersion value is large in FIG.

  If the example of the flowchart shown in FIG. 4 is applied, bit = 0 is set as the quantization condition A, and bit = 1 is set as the quantization condition B. Therefore, when the dispersion value of periodicity No. = 0 is large (that is, when imax = 0), bit = 0, and conversely, when the dispersion value of periodicity No. = 1 is large (that is, when imax = 1). ) Can be determined as bit = 1. That is, by associating the quantization condition with the spatial filter characteristics and the periodicity of the condition for specifying the pixel position, multiplexing and demultiplexing (or demultiplexing) of an image and a code can be easily realized. In this embodiment, the periodicity numbers are two types, 0 and 1, and the multiplexing code in the block is 1 bit, but it is needless to say that the multiplexing code may be more than this. Naturally, the type of quantization condition, the type of spatial filter, and the type of periodicity No. (value of I) in the pixel position specifying condition coincide with each other.

  As is clear from the above description, the predetermined value J in the procedure of FIG. 11 indicates the number of types of code information that may be multiplexed in the image data. The predetermined value I indicates the number of phases that can be taken during one period. In this example, since the phase is a discrete value in units of pixels, the number of pixels in one cycle may be set as the value of I.

The procedure of FIG. 11 is applied to each block shown in FIG. 5, and a code corresponding to the quantization condition for each block is determined. Then, for example, the W × H codes are arranged in the order of the blocks to reproduce the embedded code information. In this example, for example, the quantization condition A corresponds to “0” and the quantization condition B corresponds to “1”, so that the original code information is reproduced by arranging them in the block order. In addition, 2 ⁿ +1 different quantization conditions (that is, quantization threshold patterns) are prepared, and 2 ⁿ quantization conditions are associated with n-bit codewords, so that n bits per block. Code information can be multiplexed.

  Further, the procedure of FIG. 11 need only be executed for one direction of the image if the multiplexed code can be specified by performing the procedure of FIG. 11 for one direction. For example, the quantization condition (that is, the code pattern) shown in FIG. 6 can be distinguished by executing FIG. 11 in one direction. However, if the multiplexed code cannot be reduced to one in only one direction, it is executed for each of the horizontal direction and the vertical direction, and after determining the value of imax independently for each direction, the values are combined. The final imax may be determined. For example, when a plurality of variance values B [i] are substantially the same in one direction, for example, a variance value that is within a predetermined value for the difference from the maximum variance value is taken as a maximum value candidate, and the variance value of these candidates Is left as a candidate for imax. Then, imax candidates are determined in both the horizontal and vertical directions, and the value of the periodicity index i common to both directions is determined as the final imax. Then, it is determined that the code associated with the index value imax is a code multiplexed on the image.

  The procedure of FIG. 11 extracts the information corresponding to the threshold matrix by determining the correlation between the image information and the pixel pattern of the additional information, or determining the correlation between the distribution of pixel values for each block and the threshold matrix. It can be paraphrased as a procedure. That is, the degree of correlation between the image information in which the additional information is multiplexed and the pixel pattern (pattern corresponding to the quantization threshold) corresponding to each bit or bit group of the multiplexed additional information is obtained. If the correlation is lower than a predetermined level, it is determined that decoding has failed, and if there is a predetermined level or higher, it is determined that bits or bit groups corresponding to correlated pixel patterns are multiplexed. This means that the multiplexed additional information can be decoded by executing the procedure for obtaining the degree of correlation, not limited to the procedure of FIG. Alternatively, the degree of coincidence between the spatial frequency of the image information included in the block and the spatial frequency of the quantization threshold matrix pattern is determined for each block of the image in which the additional information is multiplexed. If the degree of coincidence is equal to or higher than a predetermined level, it can also be referred to as a decoding method for decoding a code corresponding to the quantization threshold matrix.

  In the first embodiment, the codes can be easily separated without comparing the power values of the frequencies corresponding to the regularity of the quantization condition by orthogonal transform. Moreover, separation processing can be realized at a very high speed because of processing in the real space region.

  Although the first embodiment has been described above, the quantization conditions A and B, the spatial filters A and B, and the pixel specifying units A and B are examples, and the present invention is not limited to this. Other periodicities may be provided, and the number of taps of the spatial filter, the block size for specifying the pixel position, and the like may be larger or smaller than the above example.

  Further, in the operation procedure of FIG. 11, in order to easily explain the idea of the present invention, it is described by the repetition process of the variable i that is the periodicity No. and the variable j that is the phase No. However, in practice, iterative processing with pixel addresses in a block composed of P × Q pixels is easier to implement. That is, as shown in FIGS. 12 and 13, two types of information of periodicity No. and phase No. are stored in advance as a table for each pixel address in the block, and the corresponding periodicity is stored. This is a method of adding a conversion value to each of the No. and phase No. variables. In this method, the addition value of each set of periodicity No. and phase No. can be calculated in parallel only by processing P × Q pixels.

  Further, in the operation procedure of FIG. 11, the variance of the addition result of the converted value at the specified position after the spatial filter is calculated, and the sign is determined by comparing the variance values. However, the present invention is not limited to this. Absent. A method based on comparison of evaluation functions without using a variance value is also conceivable. The bias of the addition result of the converted values at the specified positions can be easily evaluated only when there is one phase when the phase is shifted.

For example, in order to evaluate the degree of variation, the following evaluation function can be considered in addition to the variance value.
1. 1. The difference between the maximum value and the minimum value of the addition values obtained by adding the conversion values at the specified positions 2. Either the difference between the maximum value of the addition value obtained by adding the converted values at the specified position and the second largest value, or the difference between the minimum value and the second smallest value. The maximum value of the difference in the order before and after when a histogram is created by adding the converted values at the specified positions.

  The evaluation functions 1, 2, and 3 are absolute difference values, but the relative ratio between these difference values and the conversion value, or the sum of the pixel value and the conversion value, etc. should be used as the evaluation function. Can do. In addition, although the quantization value has been described by taking binarization as an example, it is not limited to this.

  As described above, according to the first embodiment, the quantization condition is changed for each block of M × N pixels, and the image is quantized according to the quantization condition. Can be embedded. As a result, compared to conventional information embedding methods, for example, the method of embedding information by orthogonal transformation, information on the image is suppressed so that the deterioration of image quality is suppressed and the embedded information can be extracted with high accuracy. Can be embedded.

[Continuous shooting of multiplexed prints]
Next, continuous shooting of a printed material (referred to as a multiplexed printed material) in which additional information is multiplexed by the camera-equipped mobile terminal 104 will be described. The camera-equipped mobile terminal 104 undergoes a plurality of steady deteriorations depending on the shooting conditions during shooting. The steady deterioration is, for example, deterioration that occurs at a fixed position in an image shot under a fixed shooting condition, and the cause and type of the deterioration do not necessarily have to be fixed. However, since deterioration that makes it difficult to decode additional information is a problem, the type and cause of deterioration that causes decoding failure may vary depending on the multiplexing method. In the case of an image in which additional information is multiplexed by the method described with reference to FIG. 4 and the like, it becomes difficult to decode the additional information due to deterioration that changes the luminance level (or density level) and spatial frequency characteristics in the block. . For example, the brightness level locally increases due to the reflection of light, or the brightness level locally decreases due to dust attached to the image sensor or low-pass filter, etc. Can be.

  FIG. 14A is a photographed image when a printed matter is photographed with the camera light on. The black circle is a reflection of the camera light, and the reflection will continue to exist as long as the camera light is on. FIG. 14B and FIG. 14C illustrate the effect of camera lens aberration when a printed matter is photographed with a camera. FIG. 14B is an image taken with the focus (star) of the camera in the center, and blur (portion surrounded by a one-dot chain line) occurs in the periphery due to the camera aberration. FIG. 14C is an image in which the focus (star) of the camera is adjusted to the peripheral part, and the blur of the peripheral part is improved to some extent, but the image of the central part is conversely blurred (a part surrounded by a one-dot chain line) ).

  Next, the additional information separation method will be described in detail with reference to FIG. FIG. 15 shows functional blocks of the camera-equipped mobile terminal 104 according to the present embodiment. Note that FIG. 15 is described as a functional block, but FIG. 15 can also be said to be a flowchart showing a flow of processing by these functions. In this example, each function is realized by executing a decoding application (that is, a program) by the control unit 120, but it can also be realized by dedicated hardware.

  First, the user starts up a decoding application of the camera-equipped mobile terminal 104. This decoding application includes a shooting condition setting function S1501 for setting shooting conditions, a shooting function S1502 for shooting a printed matter based on the shooting condition setting function S1501, and additional information multiplexed on the image obtained from S1502. A decoding function S1503 for decoding is provided. Then, the decoding application uses a multiplexed data evaluation function S1504 for evaluating the multiplexed data for each image obtained from S1503 (that is, additional information multiplexed on the image), and the multiplexing for each image evaluated in S1504. A multiplexed data merge function S1505 for merging (or synthesizing) data and finally obtaining accurate additional information is provided.

  FIG. 16 illustrates an example of the additional information separation method of FIG. Hereinafter, each function of FIG. 15 will be described in detail with reference to FIG.

  In the shooting condition setting function S1501, a plurality of shooting conditions can be set. For example, the light ON of the mobile terminal with camera considering the shortage of ambient light, the light OFF for suppressing the influence of the reflected light, and the like can be cited as examples of the shooting conditions. In addition, there are shooting conditions such as setting a plurality of shutter speeds considering the moving speed of the camera-equipped mobile terminal at the time of shooting, and specifying the shooting resolution considering the processing speed and memory capacity. Shooting conditions other than these shooting conditions may be set.

  Further, the current shooting conditions may be dynamically set according to the shooting conditions set in the past. For example, after shooting with the camera shooting condition setting set to auto, the next shooting condition may be determined according to the shooting condition at the time of auto. Further, when the shooting conditions of the camera of the camera-equipped mobile terminal 104 are automatically set according to the surrounding environment, the shooting information of the camera-equipped portable terminal may be indirectly changed by changing the surrounding environment. For example, shooting information for automatically increasing the shutter speed may be set by increasing the amount of light.

  The shooting condition setting function S1501 sets four shooting conditions according to the light ON / OFF and the focus position as the shooting condition 1601 in FIG. For example, four conditions can be set by combining the two focal positions shown in FIGS. 14B and 14C and the light on / off shown in FIG. 14A. That is, since the user shoots under each shooting condition, the user shoots four times.

  The shooting function S1502 performs shooting based on the setting of the shooting condition setting function S1501. It is desirable for the camera-equipped mobile terminal 104 to change the setting according to the shooting conditions for each shooting. The setting change according to the shooting condition is performed under the control of the control unit 120. The shooting may be performed continuously after setting a plurality of shooting conditions. Alternatively, the shooting conditions may be set for each shooting and repeated shooting may be performed. This shooting may be performed in the still image continuous shooting mode, or may be cut out one frame after being shot in the moving image mode and divided as image data. In addition, when shooting with a digital still camera instead of the camera function of the camera-equipped mobile terminal 104, a plurality of continuously shot images are taken into a personal computer or the like, a decoding application is launched on the personal computer OS, and the captured multiple images are captured. It may be processed. (A) to (d) of the photographed image 1602 in FIG. 16 are the results of photographing the same subject under the four photographing conditions 1601 preset in S1501. Further, the photographed image 1602 displays a blurred portion due to reflection of light and lens aberration as shown in FIG.

In the decoding function S1503, the multiplexed information embedded in each image obtained from the imaging function S1502 is decoded to obtain multiplexed data. Details of this decoding process have been described above with reference to FIG. The single-image decoding result 1603 in FIG. 16 is multiplexed data obtained by performing decoding by the decoding function S1503 for each of the captured images (a) to (d) in FIG. Here, correspondence between the reflection, the blurred portion, and the multiplexed data will be described. For example, the target image is divided into five regions in the horizontal direction, each region is further divided into a plurality of blocks, and additional information is multiplexed in each region. As an example of multiplexed data obtained from each area of the captured image (a) in FIG. 16, multiplexed data c is obtained from the leftmost area, and multiplexed data B is obtained from the second area from the left. Show.
In the captured images (a) and (c) of FIG. 16, the portion where the light is reflected is included in the second region from the right. Of the multiplexed data in the decoding result 1603 of the photographed image (a) in FIG. 16, the second data from the right is information decoded from the area corresponding to the reflected portion, and the multiplexed data D was obtained. It has been shown. On the other hand, the second data from the right among the multiplexed data in the decoding result 1603 of the captured image (c) in FIG. 16 is information decoded from the area corresponding to the reflected portion, and the multiplexed data C is obtained. Has been shown. In the photographed images (a) and (b) in FIG. 16, the blurred part is included in both end regions. The first and fifth data from the right of the multiplexed data in the decoding result 1603 in FIG. 16 is information decoded from the area corresponding to the blurred part in the peripheral part. In the photographed images (c) and (d) in FIG. 16, the blurred portion is included in the central region. The third data from the right of the multiplexed data in the decoding result 1603 in FIG. 16 is information decoded from the area corresponding to the blurred portion at the center.

  In the multiplexed data evaluation function S1504, an evaluation value corresponding to the data (that is, additional information) obtained from the decoding function S1503 is calculated. The evaluation is performed for each area as a decoding unit. There are two evaluation value calculation methods: a method of calculating an evaluation value using only image information and a method of calculating an evaluation value using information other than image information. First, an evaluation value calculation method using image information will be described. As the simplest example, the determination value used in the determination unit 710 in FIG. 7 may be used as an evaluation value as it is, or a new evaluation value may be calculated from the determination value. Alternatively, the evaluation value may be calculated from image information completely newly. The evaluation value is not limited to this as long as it is an evaluation value obtained from image information.

  As an example of a new evaluation value, several methods including an evaluation value calculation method using error detection will be described. First, in the method using error detection, an error detection function can be provided by adding an error detection code to the additional information at the input of the additional information 101 in FIG. Since a plurality of methods such as CRC have been proposed as error detection codes, they will not be described in detail here. The present invention is not limited to this as long as it detects a data error. In this example, as shown in FIG. 5, the additional information is multiplexed for each block. Therefore, the error detection code is also multiplexed into a block reserved for the error detection code. For example, each area as a unit of decoding shown in FIG. 16 includes a plurality of blocks, and some of them are allocated for error detection codes to be added. The error detection code may be the above-described CRC, but may be a simple checksum such as a parity bit in order to shorten the code length.

  In the method of evaluating using the determination result by the determination unit 710, the area to be evaluated may be a block at the time of decoding processing. For example, in the example of FIG. 7, two variance values obtained for each block are evaluated, and a higher evaluation value is given as the threshold pattern is remarkably shown. Therefore, for example, the absolute value of the difference between two variance values obtained for one block can be considered as the evaluation value. In this case, it can be evaluated that the smaller the difference is, the more difficult it is to determine the code multiplexed in the block, and the lower the accuracy of the data obtained by decoding.

  The evaluation value may be obtained by evaluating the additional information using information different from the image information. For example, information other than image information includes shooting conditions, camera models, information obtained from camera sensors, and the like. For example, when the light of the camera is ON, the evaluation value of the reflected part of the light can be set to the minimum value. In addition, when the focus of the camera is the peripheral portion, the evaluation value of the peripheral portion can be increased. The light is on and the focal length can be obtained from the camera. However, in order to know the reflection of light in the image and the out-of-focus part (the part that is out of focus), the light image in the evaluation target image is recognized and the blur of the image is evaluated. In some cases, it is necessary to perform image analysis. Depending on the model of the camera, it can be divided into those that can acquire uncompressed captured images and compressed captured images. If compression is primarily performed on saturation, the multiplexed signal of saturation can be lost. Therefore, in a camera that compresses a captured image, it is desirable to change the evaluation value by changing the importance of saturation in the process of calculating the evaluation value. For example, in a model that compresses a captured image, when additional information is multiplexed with saturation, the evaluation value obtained separately is further multiplied by a predetermined coefficient (for example, a positive coefficient of 1 or less). In order to multiplex additional information in saturation, for example, a photographed color image such as RGB is converted into a luminance-color difference color system (for example, L * a * b *), and the color difference components are described with reference to FIG. The additional information may be multiplexed by quantizing according to the procedure described above.

  Then, after acquiring data from the decoding function S1503, in S1504 which is a data evaluation function, evaluation (for example, including error detection) is performed on the data. If an error detection code is used as in the above-described example, the evaluation value can be set to the maximum value when no error is detected, and the evaluation value can be set to the minimum value when an error is detected. Even when information other than image information is used, an evaluation value can be obtained in the manner described above. The data evaluation 1604 in FIG. 16 is an example of evaluation values corresponding to the multiplexed data of the decoding result 1603 in FIG. In this example, a larger number indicates a higher rating. The evaluation value of the data evaluation 1604 in FIG. 16 has a feature that it becomes lower in the reflection and blurring portions. For example, as a result of determining each block included in the leftmost region of the captured image (a) of FIG. 16 using FIG. 7, an evaluation value of 3 is calculated. Similarly, as a result of determining each block included in the second region from the left of the photographed image (a) in FIG. 16 using FIG. 7, an evaluation value of 8 is calculated.

  In the multiplexed data merge function S1505, the multiplexed data is adopted and merged based on the evaluation value obtained from the multiplexed data evaluation function S1504. The data 1605 in FIG. 16 employs data multiplexed in an area equal to or greater than the threshold with the threshold set to 8 from the evaluation value of the data evaluation 1604 in FIG. The merge result 1606 in FIG. 16 is obtained by merging the adopted data 1605 in FIG. 16, and finally accurate data can be acquired. In addition, in the adopted data 1605 of FIG. 16, when there are a plurality of data decoded from the same area and the data are different from each other, for example, data having a high evaluation value may be preferentially adopted and merged depending on the evaluation value. .

  Even when decoding (or decoding or demultiplexing) the additional information multiplexed on the image from the image including stationary noise such as light reflection or image blur, for example, by the configuration and procedure described above. Thus, errors in the decoded additional information can be suppressed. For example, assume that a photographed image (a) is obtained by photographing a printed matter of an image in which multiplexed data “ABCDE” is embedded. When the multiplexed data is acquired only from the photographed image (a), the multiplexed data “CBCDD” is obtained due to the influence of the blurred portion or the like, and the embedded multiplexed data “ABCDE” cannot be obtained. There was a fear. However, in the present embodiment, the multiplexed data “ABCDE” can be obtained as a result by merging the highly evaluated multiplexed data from a plurality of captured images obtained under a plurality of imaging conditions, and the decoded additional data can be obtained. Information errors can be suppressed.

  In the present embodiment, the focus shift and the reflection of the light that is turned on by the camera have been described as examples of noise. However, the error can be reduced by decoding the additional information from the same image captured under different imaging conditions and merging after evaluating the other stationary noise. Here, the photographing condition is a setting related to the generation of stationary noise, and is preferably a setting that can be controlled by the camera.

(Second Embodiment)
In the shooting condition setting of the first embodiment, there are innumerable combinations of shooting conditions that are assumed, and it is not realistic to perform shooting under all the shooting conditions, try decoding, and evaluate the decoded information. In the second embodiment, a configuration capable of accurately separating additional information even in such a case will be described.

  The flowchart of FIG. 17 has a configuration in which S1701 and S1702 are added to FIG. 15 of the first embodiment. Therefore, the same blocks as those in FIG. 15 are denoted by the same reference numerals, description thereof is omitted, and details of newly added S1701 and S1702 will be described. Note that FIG. 15 has been described as a block diagram, but a flowchart of a procedure executed by the control unit 120 may be used, and FIG. 17 uses FIG. 15 from that viewpoint. The same applies to FIG. 19 described later.

  FIG. 18 illustrates an example of the additional information separation method of FIG. In all data acquisition determination S1701, it is determined whether all data after merging has been acquired. As a result of the determination in S1701, if all the data has been acquired (YES in S1701), the process ends. On the other hand, as a result of the determination in S1701, if all the data cannot be acquired (NO in S1701), the process proceeds to the shooting condition calculation function S1702.

  The shooting condition calculation function S1702 can specify a portion that cannot be accurately decoded from the data obtained from the multiplexed data merge function S1505, and set shooting conditions suitable for the portion. For example, if there are concentrated portions that cannot be decoded, the shooting conditions suitable for those portions can be obtained. If the parts that cannot be decoded are dispersed, a plurality of shooting conditions can be set.

  Hereinafter, this will be described in detail with reference to FIG. (A) and (b) of FIG. 18 are images obtained by photographing a multiplexed printed material having a black central portion. Specifically, when the image (a) in FIG. 18 is captured under certain imaging conditions, the decoding result of the image (a) in S1503 is evaluated in S1504. Based on the evaluation result, a portion whose evaluation value is lower than a predetermined value, that is, a portion that cannot be accurately decoded is not employed in the merge processing in S1505. If it is determined in S1701 that the data of all the areas has not been acquired, an area in which the data cannot be acquired is specified in S1702, and shooting conditions suitable for the area are set. For example, information decoded from the image (a) is evaluated, and it is determined that the evaluation value of the white portion has not reached the threshold value. If it is not known whether the distance of the out-of-focus portion (white portion) is too close or too far, in S1702, both the setting for shortening the focal length and the setting for increasing the length based on the shooting condition of the image (a) are performed. Determine the shooting conditions. In addition to the focus, photographing conditions such as white balance and shutter speed suitable for a non-focused portion (white portion) may be set. In S1501, the shooting conditions are set (or nothing is required if set in S1702), and in S1502, the image (b) is shot under the shooting conditions. If the evaluation value of the information obtained by decoding the white part of the image (b) reaches the threshold value, it is determined that all data has been obtained by the merge in S1505. As described above, in the image (b) photographed by changing the photographing condition, a portion that cannot be accurately decoded in the image (a) can be accurately decoded. As described above, in the image (a), since the shooting condition is adjusted to the central portion, the shooting condition is suitable for a black portion, and thus there is a high possibility that the multiplexed signal in the white portion is deteriorated. Therefore, accurate data of the white portion cannot be acquired even if the image is taken under the same shooting conditions. The image (b) is an image captured by setting shooting conditions suitable for a deteriorated white portion of the image (a). Since the shooting condition is suitable for the white part, the data of the degraded part of the white part of the image (a) can be correctly decoded. Conversely, the multiplexed signal in the black portion is degraded. Finally, if the result of the image (a) and the result of the image (b) are merged, accurate multiplexed data can be obtained.

  (C) and (d) of FIG. 20 show an example in which a part of a printed matter obtained by multiplexing the same additional information in different areas a plurality of times cannot be correctly decoded due to deterioration over time. Specifically, if there is a portion that cannot be decoded accurately from the decoding result of the image (c), the image (d) is acquired by shooting under shooting conditions suitable for the candidate location. In the image (d), a portion that cannot be accurately decoded in the image (c) can be accurately decoded. In the image (c), the candidate portion in which the same data as the aged deterioration portion is embedded exists in the peripheral portion. However, in the image (c), since the focus, which is a photographing condition, is focused on the central portion, the focus is not achieved in the peripheral portion, and correct data cannot be decoded in the peripheral portion. Therefore, accurate data cannot be acquired even if images are taken under the same shooting conditions. In the image (d), since the shooting condition is matched with the candidate portion in which the same data as the aged deterioration portion is embedded, the data of the candidate portion can be correctly decoded. That is, if the result of the image (c) and the result of the image (d) are merged, accurate multiplexed data can be obtained. However, this is premised on knowing when decoding that the same information is multiplexed in a plurality of areas. It may be known from a part of the combined data that the same information is multiplexed in a plurality of areas. This example can also be applied to an image in which a light source is reflected in a specific part.

  As described above, even when the additional information is redundantly multiplexed, the information obtained from the portion where the decoding was successfully performed is adopted as the decoded information. In this way, for example, additional information multiplexed on the image can be extracted from an image including stationary noise such as a focus shift or an aging deterioration of the original image while suppressing errors. Other than these methods, any method may be used as long as the shooting conditions are set from the merged data.

(Third embodiment)
In this embodiment, in the first embodiment or the second embodiment, the user is prompted to move the camera or the printed material, and the movement of the camera or the printed material is used to suppress the influence of stationary noise.

  An example of suppressing the influence of stationary noise will be described with reference to FIG. The captured images (a) and (b) in FIG. 20 are obtained by capturing the captured image (b) by moving the camera to the right after capturing the captured image (a). When viewing the printed matter as a reference, moving the camera to the right moves the light reflection to the right of the printed matter. The location where accurate additional information could not be acquired due to reflection in the captured image (a) is the location where accurate additional information can be acquired in the captured image (b) as the position of the reflected light moves. become. Then, by merging the results of the captured image (a) and the captured image (b), it becomes possible to finally acquire accurate additional information.

  The flowchart of FIG. 19 has a configuration in which S1901 is added to the flowchart of FIG. 17 of the second embodiment. Therefore, here, the same steps as those in the flowchart of FIG. 17 are denoted by the same step numbers, description thereof is omitted, and details of newly added S1901 will be described.

  The user movement notification function S1901 notifies that the user moves after the shooting function S1502 or when the determination in the all data acquisition determination S1701 is NO. The notification method is not limited to this as long as it is a method that can notify the user, such as voice, screen display, and vibration. This notification includes at least a message prompting the camera to move.

  Further, the direction to be moved may be displayed on the screen from the portion that cannot be accurately decoded by the photographing condition calculation function S1702. Furthermore, information indicating the movement amount may be included. In this case, in S1901, for example, stationary noise (for example, reflection of light) included in the image and its position are specified, and the moving direction of the camera that can shoot the area so as not to include noise is determined. . Further, the movement amount may be determined. Then, the determined moving direction (and moving amount) is displayed.

  In addition, when the camera is moved, it is necessary to determine a matching portion between images in the merge of data in S1505 and to determine the correspondence of regions between images. This may be done, for example, by obtaining the correlation between images. The first image to be photographed is preferably a reference image including the entire image recorded on, for example, a paper medium. In this case, the correspondence with the image captured by moving the camera is determined based on the reference image, and the decoded codes are merged according to the correspondence. Further, the correspondence may be determined based on the codes of the decoding results of the images before and after the movement, and the codes may be merged.

  Also in the present embodiment described above, it is possible to reduce the influence of stationary noise generated in an image and to realize good decoding of additional information. This embodiment can also be implemented in combination with the first embodiment.

  In addition, although all the above-described embodiments have been described with a camera-equipped mobile terminal, even by an information processing apparatus that does not have a camera function for capturing an image, by decoding each embodiment for a captured image, The invention of the embodiment can be realized. In this case, it is necessary to set camera shooting conditions from the terminal and change the camera settings. Furthermore, in this case, not only shooting conditions that can be realized by the function of the camera, but also shooting conditions can be given or changed by devices other than the camera connected to the terminal device.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

102: additional information multiplexing device, 104: mobile terminal with camera, 106: additional information separating device, 120: control unit, S1501: shooting condition setting function, S1502: shooting function, S1503: decoding function, S1504: multiplexed data evaluation Function, S1505: Multiplexed data merge function

Claims (12)

Photographing means for photographing an image in which additional information is multiplexed under a plurality of photographing conditions and recording each image data;
Extraction means for extracting additional information multiplexed from the image data of each of the plurality of shooting conditions;
Evaluation means for evaluating additional information extracted from the image data of each of the plurality of imaging conditions;
An image processing apparatus comprising: a combining unit configured to combine the additional information with an evaluation exceeding a threshold based on the evaluation by the evaluation unit. The extraction means extracts additional information multiplexed in the area from a plurality of areas constituting the image,
The evaluation means evaluates the additional information for each region,
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes the additional information with the evaluation exceeding the threshold extracted from each of the plurality of regions based on the evaluation by the evaluation unit.   The image processing apparatus according to claim 1, further comprising setting means for setting the plurality of photographing conditions.   The evaluation means determines whether or not the additional information is correct and extracted based on an error detection code added to the additional information. If no error is detected, the evaluation exceeds the threshold. If an error is detected, the evaluation is performed. 4. The image processing apparatus according to claim 1, wherein the evaluation is less than a threshold value. The additional information is multiplexed by performing quantization using a threshold matrix corresponding to information to be multiplexed for each block constituting the image,
The extraction unit determines a correlation between a distribution of pixel values for each block and the threshold matrix, and extracts information corresponding to the threshold matrix,
The image processing apparatus according to claim 1, wherein the evaluation unit sets a higher evaluation as the accuracy of information extraction by the extraction unit is higher.   6. The imaging condition according to claim 1, wherein the imaging condition includes at least one of a focal position of an optical system included in the imaging unit and an on / off setting of an imaging light. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the photographing unit photographs the image while switching the plurality of photographing conditions. It is determined whether or not all of the additional information multiplexed on the image has been combined by the combining means, and if not combined, further includes means for changing the shooting condition,
The image processing apparatus according to claim 1, wherein the additional information extracted from the image data photographed under the changed photographing condition is further synthesized based on the evaluation.   The composition means further comprises means for judging whether or not all of the additional information multiplexed on the image has been synthesized, and outputting a message prompting the user to move if it has not been synthesized. An image processing apparatus according to any one of claims 1 to 6. The image processing apparatus according to claim 1, wherein the evaluation unit evaluates the additional information using information different from the image data.   The program for functioning a computer as an image processing apparatus as described in any one of Claims 1 thru | or 10. A shooting step of shooting an image in which additional information is multiplexed under a plurality of shooting conditions and recording each image data;
An extraction step of extracting the multiplexed additional information from the image data of each of the plurality of imaging conditions;
An evaluation step of evaluating additional information extracted from the image data of each of the plurality of shooting conditions;
And a synthesis step of synthesizing the additional information with evaluations exceeding a threshold based on the evaluation.

Images