JP7455584B2 - Information processing device, control method for information processing device, program, and storage medium - Google Patents

Description

The present invention relates to an information processing device that extracts additional information embedded in a recorded image, a control method for the information processing device , a program , and a storage medium .

Patent Document 1 discloses that additional information (electronic watermark information) is embedded in the image data of an image captured by a camera, and additional information is extracted from the captured image of a recorded image recorded based on the image data in which the additional information is embedded. A technique for extracting is disclosed. In the technique disclosed in Patent Document 1, in order to reliably extract additional information (confidential information), additional information and position information in which the additional information is embedded are embedded in image data (image signal). ing. Specifically, the additional information is embedded in the first signal component of the image data, and the position information is embedded in the second signal component of the image data.

Patent No. 4015288

Various types of information, such as text document information, audio information, and video information, can be embedded in image data as additional information. However, if different additional information is embedded in each of multiple areas of image data in a manner that makes it difficult to visually distinguish them in the recorded image, the user cannot determine which additional information is included in which part of the recorded image. I don't know if there are any. Therefore, a plurality of pieces of additional information may be included in a captured image taken to extract additional information.

However, the technique disclosed in Patent Document 1 does not describe how to notify a plurality of extracted additional information, and cannot appropriately notify a user of desired additional information.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique that can appropriately notify predetermined additional information from a plurality of additional information extracted from a captured image.

In order to achieve the above object, one embodiment of the present invention is an information processing apparatus, which includes an extraction means for extracting additional information included in a captured image, and a plurality of additional information extracted by the extraction means. display control means for controlling the display means to selectably display information regarding each of the plurality of additional information on the display means; notification control means for controlling the notification to be performed based on additional information corresponding to the information that has been extracted , and the display control means for displaying the information regarding the first additional information extracted from the first captured image. Even if the first additional information is included in one or more additional information extracted from a second captured image captured after the first captured image, If the one or more pieces of additional information include second additional information that is not displayed on the display means, the information regarding the first additional information is not additionally displayed on the display means. The present invention is characterized in that information regarding additional information No. 2 is controlled to be added and displayed .

According to the present invention, it becomes possible to appropriately notify predetermined additional information from additional information extracted from a captured image.

FIG. 1 is a block configuration diagram of an information processing system according to an embodiment. FIG. 2 is a block diagram showing a firmware configuration of a multiplex encoding processing device. 5 is a flowchart showing the contents of restoration processing. A diagram showing a color correction table. A diagram explaining the tetrahedral interpolation method. A diagram showing a color separation table. FIG. 3 is a diagram illustrating an error distribution method using an error diffusion method. Conceptual diagram of mask data. 5 is a flowchart showing the processing contents of multiplex encoding processing. FIG. 2 is a block diagram showing a firmware configuration of a multiplexing unit. A diagram showing a mask pattern. 5 is a flowchart showing the processing details of recording processing. FIG. 2 is a block diagram showing the hardware configuration of a multiplex decoding processing device. FIG. 2 is a block diagram showing a firmware configuration of a multiplex decoding processing device. 4A and 4B are diagrams for explaining an imaging area and an embedding position of additional information. FIG. 3 is a diagram illustrating determination of frequency characteristics and regions multiplexed in units of a plurality of blocks. FIG. 3 is a diagram illustrating the detected position of a block. FIG. 7 is a diagram showing additional information in a captured image. 5 is a flowchart showing the processing contents of multiplex decoding processing. 5 is a flowchart showing processing details of notification processing in the information processing system of the first embodiment. 7 is a flowchart showing the processing contents of notification processing in the information processing system of the second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an example of an embodiment of an information processing apparatus , a method for controlling the information processing apparatus , a program , and a storage medium will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential to the solution of the present invention. Further, the relative positions, shapes, etc. of the components described in the embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
First, the information processing system according to the first embodiment will be described in detail with reference to FIGS. 1 to 20.

FIG. 1 is a block configuration diagram of an information processing system 10 according to this embodiment. The information processing system 10 according to the first embodiment includes a multiplex encoding processing device 12 that embeds additional information in image data, and a multiplex decoding processing device 14 that extracts additional information from a captured image and uses it.

=Multiplex encoding processing device=
<Hardware configuration>
The multiplex encoding processing device 12 acquires the image data A and the additional information B, and generates a recorded material C based on the image data A in which the additional information B is embedded. The multiplexing/encoding processing device 12 includes a multiplexing device 16 that embeds additional information B in image data A, and a recording device 18 that records based on the image data A in which the additional information B has been embedded in the multiplexing device 16. There is. The multiplexing device 16 (first information processing device) includes a CPU (central processing unit) 20, a ROM 22, and a RAM 24. The CPU 20 uses the RAM 24 as a work area and performs calculations, judgments, controls, etc. according to programs and various parameters stored in the ROM 22.

Image data A is input to the multiplexing device 16 from the input terminal 26. Image data A is multi-tone image data including color components. Additional information B is input to the multiplexing device 16 from the input terminal 28. Additional information B can be text document information, audio information, video information, compressed information such as text documents, audio, images, and videos, or converted information converted into other binary values. Note that the multiplexing encoding processing device 12 may be a recording device in which the processing of the multiplexing device 16 is executed by a control unit.

<Firmware configuration>
Next, the firmware configuration of the multiplex encoding processing device 12 will be explained. FIG. 2 is a block diagram showing the firmware configuration of the multiplex encoding processing device 12. As shown in FIG. The image data A input to the multiplex encoding processing device 12 is processed in each of the configurations described below, and then sent to the recording device 18.

- Acquisition Unit The image data A sent to the multiplex encoding processing device 12 is first processed in the acquisition unit 30. Here, the image data A input from the input terminal 26 is, for example, irreversible image data obtained by compressing document data in JPEG format. The irreversible image data includes the quantization table used during compression and the image data size.

The acquisition unit 30 acquires, from the image data A input from the input terminal 26, various parameters used when compressing the image data A. That is, if the image data A is irreversible image data obtained by compressing document data in the JPEG format, the quantization table, image data size, etc. will be acquired. The acquired parameters are sent to a restoring unit 32, which will be described later, and are used to extract image data in which additional information B is embedded. The data is also sent to a compression planning unit (not shown) and used for processing to calculate the degree of compression.

- Restoration Unit The restoration unit 32 decodes encoded image data and extracts image data. Note that in the following explanation, extracted image data will be explained as a JPEG image. With reference to FIG. 3, a composite process for decoding encoded image data, which is executed by the restoration unit 32, will be described. FIG. 3 is a flowchart showing the process of decoding encoded image data. The series of processes shown in the flowchart of FIG. 3 is performed by the CPU 20 loading the program code stored in the ROM 22 into the RAM 24 and executing it. Alternatively, the functionality of some or all of the steps in FIG. 3 may be performed in hardware, such as an ASIC or electronic circuit. Note that the symbol S in the description of each process means a step in the flowchart (hereinafter, the same shall apply in this specification).

Here, it is assumed that an image in the JPEG data format is divided into, for example, N 8-pixel square blocks (8×8 pixel blocks). In this case, in the restoration process, first, N=1 is set (S302), the N-th block is Huffman encoded in units of 8-pixel square blocks (S304), and the quantization table acquired by the acquisition unit 30 is used to inversely quantize the (S306), and inverse DCT transforms it (S308). Thereafter, it is determined whether the Nth block is the last block (S310), and if it is determined that it is not the block after the difference, N is incremented (S312) and the process returns to S304. In S310, if it is determined that the block is the last block, this restoration process ends.

Note that since JPEG decoding processing is a well-known technique, the following description will outline Huffman encoding, inverse quantization, and inverse DCT (Discrete Cosine Transform). Huffman encoding is a compression method that reduces the overall number of bits by assigning a code with a bit number close to that of frequently occurring data. In Huffman encoding, a Huffman code is defined in advance in specifications, and the code is originally decoded into data. Inverse quantization is a process of developing image data by inverse quantization using the quantization table acquired by the acquisition unit 30 (the quantization table used when compressing the image data).

Inverse DCT transformation is a process of performing inverse transformation to return image data that has been DCT-converted into a direct current component (DC component) and an alternating current component (AC component) to the original image density component information. JPEG compression is often performed in a luminance Y, color difference Cb, Cr format, and in that case, data subjected to inverse DCT processing also becomes a YCbCr format. A value in YCbCr format is converted into an image signal value in RGB format using equation (1) below.
R = Y + 1.402×Cr
G = Y - 0.344 x Cb - 0.714 x Cr... (1)
B = Y + 1.772×Cb

- Image correction unit The image correction unit 34 performs image correction processing on the RGB data decoded by the restoration unit 32. Corrections performed by the image correction unit 34 include brightness adjustment, contrast adjustment, and color balance adjustment to brighten or darken the overall color, as well as various corrections such as backlight correction and red-eye correction assuming photographic recording. . By processing these corrections in an integrated manner in the image correction section 34, processing that does not depend on the recording device 18 can be realized.

-Resolution Conversion Unit The resolution conversion unit 36 converts the image data processed by the image correction unit 34 or the like to a resolution compatible with the recording device 18. That is, the amount of scaling is derived from the image data and the resolution of the recording device 18, and the enlargement or reduction processing is performed based on this amount of scaling. Examples of scaling processing include the nearest neighbor method, bilinear method, bicubic method, etc., and are appropriately selected in consideration of processing characteristics and processing speed.

- Color Correction Unit The color correction unit 38 performs a conversion process on the image data so that the image recorded by the recording device 18 has a suitable color. For example, when an image displayed on a display device is recorded by the recording device 18, the color reproduction ranges of the display device and the recording device 18 do not necessarily match. The color reproduction range of the recording device 18 may be narrower for a certain color, and the color reproduction range of the recording device 18 may be wider for another color. Therefore, it is necessary to compress and expand colors as appropriate while minimizing image deterioration.

In this embodiment, these processes are executed in RGB format. That is, in consideration of the reproducibility of the recording device 18, the RGB values input to the color correction unit 38 are converted into RGB values for the recording device 18 (hereinafter also referred to as "RGB for the recording device"). This conversion can also be performed by calculations such as a matrix. Generally, a three-dimensional color correction table 40 is used. If the input RGB values are 8 bits for each color (256 gradations), it is not practical from the viewpoint of storage capacity to hold all combinations. For this reason, a table thinned out at predetermined intervals is used as the color correction table 40.

FIG. 4 is a diagram showing an example of the color correction table 40. The color correction table 40 is a table in which each color has 256 gradations as 17 grid points and shows the corresponding RGB values for the recording device (17×17×17=4913 grid points). Values between grid points are calculated using interpolation processing. The interpolation method can be selected from among several methods, and in this embodiment, the tetrahedral interpolation method was used. The tetrahedral interpolation method is linear interpolation using four lattice points, with the divided unit of the three-dimensional space being a tetrahedron. In this tetrahedral interpolation method, first, the tetrahedron is divided into smaller tetrahedrons as shown in FIG. 5(a). When the converted values of each store are respectively f(p0), f(p1), and f(p3), the interpolated value f(p) can be obtained using the following equation (2).

Here, w0, w1, w2, and w3 are the volume ratios of the small tetrahedrons at positions opposite to each vertex pi (see FIG. 5(b)). In this way, the RGB values for the recording device corresponding to the RGB values of the sun are calculated. In that case, the output may be set to 8 bits or more in consideration of gradation. Further, since the color correction table also depends on the color reproduction range of the printing apparatus, for example, when the printing paper (printing medium) used for printing is different, it is preferable to prepare a table corresponding to the printing paper (printing medium) used for printing.

Ink Color Conversion Unit The ink color conversion unit 42 converts the RGB values for the recording device converted by the color correction unit 38 into ink color values. This conversion uses a color separation table 44 in which combinations of RGB values for the recording device and ink color values are associated in advance. Like the color correction unit 38, the ink color conversion unit 42 also uses a table of 17 grid points.

FIG. 6 is a diagram showing an example of the color separation table 44. The color separation table 44 assumes four ink colors: cyan (C), magenta (M), yellow (Y), and black (K), and shows the values of the four colors corresponding to each grid point. There is. These values are determined taking into consideration that ink will not overflow on the recording surface of recording paper (recording medium) and that ink will not bleed when adjacent to each other. Therefore, if the recording paper (recording medium) used for recording is different, it is preferable to prepare a color separation table 44 corresponding to the recording paper (recording medium) used for recording. Further, similarly to the color correction unit 38 described above, the ink color values corresponding to the RGB values for the recording device can be interpolated by tetrahedral interpolation processing.

- Density correction unit In the recording device 18 that records using the inkjet method, as the amount of ink applied to form dots on the recording paper (recording medium) increases, the overlap of the dots increases and the recording of the image The concentration becomes difficult to increase. The density correction unit 46 corrects the density in order to make such density responsiveness uniform. Such density correction makes it easier to ensure the accuracy of creating the color correction table 40 and color separation table 44. The recording device 18 that uses C, M, Y, and K inks performs density correction for these ink colors. In this embodiment, a one-dimensional density correction table 48 is used. As the density correction table 48, it is sufficient to prepare a table corresponding to 8 bits of input (256 gradations) for each ink color, and it is possible to correspond the input signal value and the output signal value after correction without any particular thinning. You can use the attached table.

- Gradation Conversion Unit The tone conversion unit 50 converts the multi-bit data, which has been converted for each ink color and subjected to density correction, into the number of tones that can be recorded in the recording device 18. In this embodiment, the conversion is performed into two gradations (1 bit): recording "1"/non-recording "0". Uses an error diffusion method that can reproduce the key. Furthermore, 8-bit data from 0 to 255 is assumed as an input signal.

FIG. 7 is an explanatory diagram of an error distribution method in the error diffusion method. The signal value L of the target pixel is compared with the threshold value TH. In this embodiment, the threshold value for binarizing 0 to 255 is set to 127, and it is determined whether the target pixel is "1" (recorded) or "0" (non-recorded) as shown below. .
L>TH ・・・・・・ 1 (record)
L≦TH ・・・・・・ 0 (non-recording)

According to this determination result, the quantization representative value V is set as follows.
1 (record) 255
0 (non-recording) ・・・・・・ 0

By setting the quantization representative value V in this way, the generated error E (=LV) is distributed to surrounding pixels according to the distribution coefficient shown in FIG.

Then, the value La obtained by adding the distributed error Ea to the signal value L of the next target pixel is compared with the threshold value TH, and it is determined that the target pixel is "1" (recorded) or "0" ( (non-recording).
La>TH ・・・・・・ 1 (record)
La≦TH ・・・・・・ 0 (non-recording)

By performing such processing on all pixels and all ink colors C, M, Y, and K, printable 1-bit print data for each ink color is obtained. The gradation modulation section 52 transmits the acquired recording data to the recording device 18.

Here, the recording device 18 includes a reception memory that stores received recording data, a CPU, and a recording section that applies ink to a recording medium. The recording device 18 stores the input recording data in a receiving memory. The recording unit converts the recording data into ink droplet data corresponding to the nozzles of the recording head. Recording processing is performed by ejecting ink onto a recording medium using a recording head that manipulates ink droplet data in a direction that intersects the conveyance direction of the recording medium. Although this embodiment shows the configuration of a serial scan type inkjet recording device, it is also applicable to a full line type inkjet recording device or a recording device that records on a recording medium using a method other than the inkjet method. be.

- Additional information Additional information 54 is additional information B embedded in image data A in the multiplex encoding processing device 12, and is text document data or the like. The text document data is, for example, numerical data in which numbers and letters are assigned to numerical values using a known character code, and this numerical data is transmitted as additional information 54 to a multiplexing unit 56 (described later).

As a specific example, text document data corresponding to the character string "hello" will be explained. The text document data is numerical data, so-called binary data. Binary data is information of ``0'' or ``1'', and the continuous connection of this ``0'' or ``1'' information has a specific meaning. The correspondence between binary data and characters is defined by a "character code." In the case of "Shift JIS", which is one of the character codes, "h" corresponds to binary data "01101000". Similarly, "e" corresponds to the binary data "01100101," "l" corresponds to "01101100," and "o" corresponds to the binary data "01101111." Therefore, the character string "hello" can be expressed as "0110100001100101011011000110110001101111" in binary data. Conversely, if the binary data "0110100001100101011011000110110001101111" can be obtained, the character string "hello" can be obtained. The additional information 54 corresponds to such numerical data converted into binary data.

- Multiplexing unit The multiplexing unit 56 receives the image data processed by the resolution conversion unit 36 and additional information 54 (such as a text document converted into binary data of “0” and “1”), and converts the additional information into 54 into the image data. In this embedding process (multiplexing process), the additional information 54 is added to the image so that the additional information 54 can be read from the recorded image recorded based on the image data in which the additional information 54 is embedded. Embed in data. For example, so that the binary data of "0" and "1" of the additional information 54 can be read, the image data is subjected to masking processing, and the information of "0" and "1" corresponding to the binary data is embedded. In this embodiment, mask processing is performed on the image data to give a predetermined region of the image data different periodicities corresponding to binary data of "0" and "1".

FIGS. 8A and 8B are diagrams showing mask data corresponding to binary data of "0" and "1". The mask data shown in FIG. 8 has a size corresponding to a 5 px (pixel) x 5 px area. In addition, the mask data is created by combining patterns with different periodicities as shown in FIG. 8(a) or FIG. 1" binary data is embedded. When reading a recorded image, it is possible to read binary data of "0" and "1" by recognizing the periodicity corresponding to the binary data of "0" and "1" through frequency analysis of the read data. I can do it.

The multiplexing unit 56 embeds the additional information 54 by giving the image data periodicity corresponding to the binary data of "0" and "1" based on the binary data (numeric data) of the additional information 54. As an example of a method for embedding the additional information 54 into image data, a method will be described in which the image data is one-color gray image data and binary data of "0" and "1" are embedded in the entire image data.

The size of the image data to be multiplexed is 640 px in height and 480 px in width, and the mask data is 5 px x 5 px as in FIGS. 8(a) and 8(b). It is assumed that binary data of "0" is expressed by the mask data of FIG. 8(a), and binary data of "1" is expressed by the mask data of FIG. 8(b). In the mask data of FIGS. 8A and 8B, a 5×5 pixel block is divided into a black block 801, a white block 802, and a diagonal block 803. The black block 801 corresponds to the numerical value "+2", the white block 802 corresponds to the numerical value "0", and the diagonal block 803 corresponds to the numerical value "-1". In the code below, when maskA is true, the mask data in FIG. 8(a) corresponding to "0" is used, and when maskA is false, the mask data in FIG. 8(b) corresponding to "1" is used. use Pseudo-code for applying the numerical values corresponding to the black, white, and diagonal blocks in FIGS. 8(a) and 8(b) to the entire image data is shown below.

Pseudo code:
――――――――――――――――――――――――――――
1 int i, j, k, l;
2 int width = 640, height=480;
3 unsigned char *data = image data;
4 int **maskA = mask data;
6 bool isMaskA = true;
5 for(j = 0; j <height; j+=5){
6 for(i = 0; i <width; i+=5){
7 for(k = 0; k <5; k++){
8 for(l = 0; l <5; l++){
if(isMaskA == true){
9 data[(i+k)+(j+l)*width] += maskA[k][l];
}
Ten }
11}
12}
13 }
――――――――――――――――――――――――――――

As shown in the above pseudocode, the entire image is divided into blocks of 5px x 5px, and the data of maskA is added to each block, thereby forming the patterns shown in FIGS. 8(a) and (b). It becomes.

In the above explanation, one-color clay image data is used as an example, but RGB components may be applied and a mask pattern may be applied in three dimensions, or the RGB components may be converted to YUV, Lab, etc. , Lch, or other color components may be used. Furthermore, in this embodiment, the mask data is divided into blocks of 5 px x 5 px as a method of applying the mask data, but the method is not limited to this. For example, other block size units or mask shapes may be used. Any method may be used as long as it is capable of distinguishing patterns when the recorded material C in which multiplexed images are recorded is captured.

As a pattern corresponding to the binary data (numeric data) of the additional information 54, a pattern that is as inconspicuous to human vision as possible may be formed. If the image data is gray image data of one color, a pattern must be formed using luminance components, so it is visually noticeable. Although it depends on the pattern shape and frequency components, changes in color components tend to be less visually noticeable than changes in brightness components. For example, a color image having RGB components is converted into a color space such as YCbCr, Lab, or Yuv, and separated into a luminance component and a color component. By applying the mask data to the color component rather than the luminance component, it is possible to form a pattern that is less visually noticeable. Further, for example, in a color image having RGB components, if red is dominant in a 5 px x 5 px area where a pattern is to be applied, it is preferable to apply the pattern using the red component.

In this embodiment, mask data is divided into blocks of 5 px x 5 px and added to image data. However, the block size unit and mask shape are arbitrary, and addition, subtraction, multiplication, division, and the like may be combined as a method of incorporating mask data into image data. Any method may be used as long as it is capable of distinguishing the pattern of the mask data when capturing an image of a recorded object in which the additional information 54 is embedded. Therefore, the multiplexing unit 56 is a processing unit for embedding the additional information 54 in image data so that the additional information 54 can be extracted in a captured image of a recorded material based on the image data in which the additional information 54 is embedded. It is.

<Multiplex encoding processing>
Next, the specific processing contents of the multiplex encoding process in the multiplexing device 16, that is, the process of embedding additional information into input image data will be explained. FIG. 9 is a flowchart showing the processing details of multiplex encoding processing according to this embodiment. The series of processes shown in the flowchart of FIG. 9 is performed by the CPU 20 loading the program code stored in the ROM 22 into the RAM 24 and executing it. Alternatively, the functionality of some or all of the steps in FIG. 9 may be performed in hardware, such as an ASIC or electronic circuit.

In the multiplex encoding process, first, image data for recording is acquired (S902). That is, in S902, the acquisition unit 30 first acquires parameters from the input image data A. Thereafter, the restoring unit 32 decodes the encoded image data A using the acquired parameters. For example, image data A is data that has been imaged in advance with a camera-equipped mobile terminal (smartphone) and stored in a memory within the mobile terminal in JPEG format. The acquisition unit 30 decompresses the acquired JPEG image data and generates three-color, 8-bit RGB image data of a still image. Further, in S902, the image correction unit 34 may perform correction or processing processing on the acquired image data as necessary.

Next, the multiplexing unit 56 acquires the additional information 54 (S904). The additional information 54 is, for example, text document data entered using a key on a smartphone. The text document data is, for example, numerical data in which numbers and characters are assigned to numerical values using the well-known character code shift JIS.

Then, the resolution conversion unit 36 performs resolution conversion processing on the acquired image data A based on the selected paper size (size of the recording medium) and the resolution of the recording device 18 (S906). For example, if the selected paper size is 2L, the resolution of the image data A is converted in accordance with the number of pixels of the input resolution in the recording device 18. Specifically, when the input resolution in the recording device 18 is 600 dpi, the number of pixels of paper size 2L is set to 3000 pixels x 4000 pixels, for example. In this case, for image data with a pixel count of 1500 pixels x 2000 pixels, resolution conversion is performed so that the number of pixels in the vertical and horizontal directions is doubled. When it is desired not to change the aspect ratio of image data, resolution conversion is performed with the same enlargement and reduction ratios in the vertical and horizontal directions.

After that, the multiplexing unit 56 performs a multiplexing process to embed the additional information 54 in the image data after the resolution conversion process (S908). The image data in which the additional information 54 is embedded, that is, the multiplexed image data, is processed by the color correction section 38, ink color conversion section 42, density correction section 46, and gradation conversion section 50 as appropriate. , the generated recording data is transmitted to the recording device 18 (S910).

<Firmware configuration of multiplexing unit>
Here, the multiplexing unit 56 and the multiplexing process executed by the multiplexing unit 56 will be described. FIG. 10 is a block diagram showing the firmware configuration of the multiplexing section 56. The multiplexing section 56 includes a color space conversion section 58, a block position setting section 60, a numerical conversion section 62, a pattern selection section 64, and an information multiplexing section 66.

- Color Space Conversion Unit The color space conversion unit 58 is a processing unit that converts the color space of the image data whose size has been changed in the resolution conversion unit 36 into a color space for information multiplexing. For example, as shown in equation (3) below, the color space in which information is multiplexed is set to U of YUV, and the RGB color space of image data is converted to the YUV color space.
Y = 0.299 x R + 0.587 x G + 0.114 x B
U = -0.169 x R - 0.331 x G + 0.500 x B... (3)
V = 0.500 x R - 0.419 x G - 0.081 x B

・Block position setting unit In this embodiment, the image data to be recorded is divided into a plurality of block areas, and the density of each pixel is modulated in block units to create a pattern corresponding to the mask data in FIGS. 8(a) and 8(b). By forming the additional information 54, the additional information 54 is embedded. The block position setting unit 60 sets the position coordinates of a block in accordance with the specified size of one block for the plain image of the specified color in the image data processed by the color space conversion unit 58. . For example, the size of a YUV U-color plain image is 640 px in height and 480 px in width, and the block size is 5 px in height and 5 px in width. In this case, the number of vertical blocks is 128 (640÷5), the number of horizontal blocks is 96 (480÷5), and the total number of blocks is 12288 (128×96). For example, the upper left coordinates of each block are determined and set as the block position.

- Numerical conversion unit The numerical conversion unit 62 converts the acquired additional information 54 into numerical data. For example, assume that the additional information 54 is a Shift JIS character string. In this case, the digitization unit 62 previously holds a conversion map in which characters and numbers are associated with each other in the Shift JIS format, and converts the character string into a numeric string using the conversion map. For example, in the case of the character string "hello", the converted numerical data becomes "0110100001100101011011000110110001101111".

- Pattern selection section The pattern selection section 64 has registered mask patterns for performing density modulation of each pixel in block units, and selects mask patterns to be applied to the additional information 54 digitized by the digitization section 62. Select. FIGS. 11(a) and 11(b) are diagrams in which the mask patterns of FIGS. 8(a) and 8(b) having different frequency characteristics are digitized. The patterns in FIGS. 8(a) and 11(a) correspond to "0" in the binary data in the additional information 54, and the patterns in FIGS. 8(b) and 11(b) correspond to "0" in the binary data in the additional information 54. Corresponds to "1".

- Information multiplexing unit The information multiplexing unit 66 converts the image data processed by the color space conversion unit 58, the position of each block set by the block position setting unit 60, and the mask pattern selected by the pattern selection unit 64. and get. Then, from each of the acquired information, image data is generated by applying a mask pattern to the image data.

As described above, when the image size is 640 px in height and 480 px in width, and the size of one block is 5 px in height and 5 px in width, the total number of blocks is 12,288. When capturing a recorded image when reading the additional information 54, it is not necessarily possible to capture the entire image. Therefore, in order to make it possible to extract additional information even from a captured image that is a portion of the recorded image, the same additional information is embedded in multiple locations in the recorded image.

For example, if 96 blocks are used as one additional information, the same additional information will be embedded in 128 (12288÷96) areas for the total number of blocks, 12288. Therefore, the image data is divided into 128 areas, and 96 blocks of additional information, each block having a height of 5 px and a width of 5 px, are embedded in one of these areas. By treating 96 blocks as one piece of additional information, 96 bits of additional information can be set. However, in order to know the starting position of the 96 bits, the 8 bits of "11111111", which are not expressed as characters in Shift JIS, are included at the beginning of the additional information. Therefore, 88 (96-8) bits of data can be freely set as additional information.

On the other hand, a plurality of different additional information 54 can also be embedded in the image data. In this case, if n pieces of additional information 54 are to be embedded, each type of additional information 54 is embedded at 128/n locations. For example, in the case of four types of additional information 54, each additional information 54 will be embedded in 32 (128÷4) areas. In addition, to each type of additional information 54, information for making it possible to determine the type is added after the first 8 bits of "11111111". For example, if there are four types, one of "00", "01", "10", and "11" is assigned. Therefore, in this case, 86 (96-8-2) bits of data can be freely set as additional information.

The data that fits within 96 bits is a numeric string of "0" and "1" of additional information digitized by the digitization unit 62, and a numeric value is presented for each block of 5px x 5px, and the corresponding numeric value is A mask pattern is selected. A mask pattern corresponding to the additional information is embedded in a 5px×5px block in the image data. For example, take the image data as a YUV U color plane, process it block by block (5px x 5px), and apply the mask pattern values in Figures 11(a) and (b) to the YUV U color plane values. do. For example, as shown in equation (4) below, the value of the YUV U color plane (U value) is subjected to addition/subtraction processing according to the numerical value of the mask pattern, and the reference value for performing the addition/subtraction processing is set to "10".
U value after application = YUV U value + reference value × mask pattern value... (4)

For example, if the U value of one pixel in one block is "20" and the numerical value of the applied mask pattern is "0", then from the above formula (4), 20 + 10 × 0 = 20, and after application The U value is "20". In addition, if the U value of one pixel in one block is "30" and the numerical value of the applied mask pattern is "2", then from the above formula (4), 30 + 10 × 2 = 50, and the applied U The value will be "50".

As described above, in this embodiment, multiplexing is achieved by adding the value obtained by multiplying the numerical value of the mask pattern applied to each pixel by the reference value, but this is not limitative. It's not a thing. That is, the mask pattern can be applied as long as the mask pattern can be embedded on the U color plane. For example, the U value of YUV may be multiplied by the numerical value of the mask pattern.

<Recording process>
Next, the recording process, which is a series of processes up to the generation of recorded material C by the multiplex encoding processing device 12, will be described. FIG. 12 is a flowchart showing the contents of storage processing in the multiplex encoding processing device 12.

When recording processing is started in response to an instruction from a user, first, the multiplexing unit 56 of the multiplexing device 16 performs multiplex encoding processing (S1202). As described above, image data (multiplexed image data) in which additional information 54 is embedded is obtained through this multiplexed encoding process. Next, the color correction unit 38 performs appropriate color correction on the multiplexed image data (S1204). Thereafter, recording data is generated (S1206), the generated recording data is transmitted to the recording device 18, recording is performed in the recording device 18 based on the recording data (S1208), and the recording process ends. Recorded matter C is generated by the recording in S1208. Note that in S1206, the ink color conversion unit 42 first converts the multiplexed image data after color correction into ink color values, and the density correction unit 46 corrects the density. Then, the gradation converter 50 converts the density-corrected multiplexed image data into the number of gradations to generate recording data.

=Multiplex decoding processing device=
Next, the multiplex decoding processing device (information processing device) 14 will be explained. The multiplex decode processing device 14 extracts additional information B from the recorded image recorded on the recorded material C generated by the multiplex encode processing device 12, and notifies the extracted additional information B. Specifically, by capturing a recorded image recorded in a recorded object C based on image data subjected to multiplex encoding processing using an imaging device such as a camera, and analyzing the image data of the captured image. , extracts additional information B embedded in the image and notifies it.

<Hardware configuration>
First, the hardware configuration of the multiplex decoding processing device 14 will be explained. FIG. 13 is a block diagram showing the hardware configuration of the multiplex decoding processing device 14. As shown in FIG. As the multiplex decoding processing device 14, for example, a mobile terminal with a camera that holds an image sensor can be used. In the following description, a case will be described in which the above mobile terminal is used as the multiplex decoding processing device 14.

The multiplex decoding processing device 14 includes an image sensor 68 capable of capturing a recorded image in the recorded object C, and a separation device 70 (second information processing device) that extracts additional information 54 from image data of the captured image. . The multiplex decoding processing device 14 also includes a CPU 72, a ROM 74, a RAM 76, a secondary storage device 78, a display 80, a key input device 82, a wireless LAN 84, and a speaker 86.

The CPU 72 uses the RAM 76 as a work area and performs calculations, determination, control, etc. according to programs and various parameters stored in the ROM 74. A secondary storage device 78 such as a hard disk stores various data such as image files and a database that stores image analysis results. The display 80 presents the results of various processes executed by the CPU 72 to the user. The key input device 82 is realized by, for example, the display 80 having a touch panel function, and performs processing instructions, settings, character input, etc. by operating the touch panel.

A wireless LAN (Local Area Network) 84 is connected to the Internet, accesses sites connected to the Internet, and displays the screen of the site on the display 80. The wireless LAN 84 is also used for data transmission and reception. The speaker 86 outputs audio when the extracted additional information is audio data or video data with audio. Furthermore, if there is motion data at the Internet connection destination, the audio when the video data is played back is output.

Note that the configuration of the multiplex decoding processing device 14 is not limited to the above-described camera-equipped mobile terminal. That is, a configuration may be adopted in which image data of a captured image is transmitted from a separately provided imaging device to a separation device for separating additional information. Note that in this case, a device for utilizing the extracted additional information, such as a speaker or a display, is connected to the separation device. Further, in this case, as the imaging device, for example, a digital camera, a video camera, etc. can be used. As the separation device, for example, a personal computer, a smartphone, etc. can be used.

<Firmware configuration>
Next, the firmware configuration of the multiplex decoding processing device 14 will be explained. FIG. 14 is a block diagram showing the firmware configuration of the multiplex decoding processing device 14. The multiplex decoding processing device 14 includes an imaging section 88 and a color adjustment section 90 in the imaging sensor 68 . Further, the multiplexing decoding processing device 14 includes a detection section 92, a separation section 94, and an analysis section 96 in the separation device 70. In this embodiment, digitized additional information such as text document data, audio data, video data, etc. is embedded in the recorded image in the recorded material C.

In the following explanation, it is assumed that different pieces of additional information are repeatedly embedded in each predetermined area throughout the entire recorded image of the recorded object C. For example, it is assumed that four types of additional information A, B, C, and D are repeatedly embedded in each predetermined area (see FIG. 15(b)).

- Imaging Unit The imaging unit 88 captures a recorded image of the recorded object C using an imaging element in the image sensor 68, and converts it into image data. FIG. 15(a) is a diagram illustrating capturing of a recorded image on recorded object C. FIG. A recording image 1504 based on image data subjected to multiplex encoding processing is recorded on a recording medium 1502 corresponding to recording material C. A region 1508 is imaged by a camera-equipped mobile terminal 1506 as the multiplex decoding processing device 14 . That is, the imaging unit 88 uses the mobile terminal 1506 to image an area 1508 including the recorded image 1504 on the recording medium 1502.

For example, a CCD can be used as the imaging element of the imaging section 88. A CCD senses light using a photodiode (light receiving element) and converts the light into voltage. At this time, the color of the light can be converted into data using a color filter such as RGB or CMY arranged for each image sensor. The detection signal of the photodiode is transmitted to the color adjustment section 90.

- Color Adjustment Unit The color adjustment unit 90 converts the output data of the photodiode in the imaging unit 88 into image data in which one pixel is RGB 8-bit data. Before converting into image data, color interpolation processing such as RGB is performed on the output data of the photodiode depending on the light source at the time of imaging. In this interpolation process, adjustments are made so that a white subject can be imaged as white when capturing an image using a digital camera, a mobile terminal with a camera, or the like. Since the imaging unit 88 uses a photodiode to detect light emitted from a light source such as the sun or a light and reflected from a subject, the color of the image will differ depending on the light source. Therefore, the color adjustment unit 90 performs interpolation processing depending on the light source.

As a general interpolation processing method, there is a method using Kelvin (K), which is a numerical unit of the color of light expressed by the color temperature indicating the hue of the light source. Generally, daytime sunlight is 5,500K, and incandescent light bulbs are 3,000K; when the color temperature is high, it appears blue, and when it is low, it appears red. Therefore, the color of the captured image differs depending on the light source. Generally, digital cameras and mobile terminals with cameras are equipped with a so-called auto white balance adjustment function that detects the color temperature using a sensor when taking an image and automatically adjusts it so that a white subject appears white. has been done. The white balance can also be manually adjusted depending on the light source, such as sunlight or an incandescent light bulb.

The color adjustment unit 90 generates image data with white balance adjusted for the output data of the photodiode. The image data processed by the color adjustment section 90 is sent to the separation device 70.

・Detection unit The detection unit 92 receives the image data color-adjusted by the color adjustment unit 90, and detects the position where additional information is embedded (multiplexing position) by determining the frequency characteristics of the image data. do. FIG. 16(a) is a diagram illustrating the difference in frequency characteristics in a two-dimensional frequency domain. In FIG. 16(a), the horizontal axis is the frequency in the horizontal direction, the vertical axis is the frequency in the vertical direction, the central origin indicates the DC component, and the further away from the origin, the higher the frequency range becomes. In this embodiment, the frequency characteristics change due to multiplexing processing.

For example, a change in frequency characteristics when applying the mask data in FIG. 8(a) causes a large power spectrum on the straight line l1 in FIG. 16(a). Furthermore, due to the change in frequency characteristics when the mask data in FIG. 8(b) is applied, a large power spectrum occurs on the straight line l2 in FIG. 16(a). When separating the additional information 54, a frequency vector in which such a large power spectrum occurs is detected in order to determine the multiplexed signal. Therefore, it is necessary to individually emphasize and extract each frequency vector.

For this purpose, an HPF (high pass filter) having frequency characteristics similar to the mask patterns in FIGS. 11(a) and 11(b) can be used. The spatial filter corresponding to the mask pattern of FIG. 11(a) can emphasize the frequency vector on the straight line l1. The spatial filter corresponding to the mask pattern of FIG. 11(b) can emphasize the frequency vector on the straight line l2.

For example, it is assumed that a large power spectrum is generated on the frequency vector of the straight line l1 due to the quantization conditions to which the mask pattern of FIG. 11(a) is applied. In this case, the amount of change in the power spectrum is amplified by the spatial filter corresponding to the mask pattern of FIG. 11(a), but hardly amplified by the spatial filter corresponding to the mask pattern of FIG. 11(b).

That is, when filtering is performed using a plurality of spatial filters in parallel, the power spectrum is amplified only by the spatial filters with matching frequency vectors, and is hardly amplified by the other spatial filters. Therefore, by identifying the spatial filter that amplified the power spectrum, it is possible to determine on what frequency vector the large power spectrum has occurred. In this way, by determining the frequency characteristics, it is possible to extract additional information. However, when determining the frequency characteristics, if the extraction position of the additional information shifts, it becomes difficult to extract it correctly.

FIG. 16(b) is a diagram illustrating a recording area in which a recorded image is formed in a recorded object. A recorded image 1504 recorded on a recording medium 1502 as recorded material C includes an area 1602 multiplexed in units of a plurality of blocks, and additional information is embedded in the area 1602 so that it has specific frequency characteristics. It is.

FIGS. 17(a) and 17(b) are diagrams for explaining detection positions when determining frequency characteristics. It is assumed that the recorded image 1700 is multiplexed into four blocks. Areas 1702 and 1704 indicate areas where frequency characteristics are determined on a block-by-block basis. FIG. 17A shows that a region 1702 is shifted from the position of the multiplexed block. FIG. 17(b) shows that region 1704 matches the position of the multiplexed block. In this case, in the region 1704, it is possible to correctly determine the frequency characteristics, but in the region 1702, the power spectrum of a specific frequency vector decreases, making it difficult to correctly determine the frequency characteristics.

The detection unit 92 detects the block position based on whether the power spectrum of a specific frequency vector is strong or weak. Therefore, the detection unit 92 performs frequency characteristic determination on the image data of the captured image while shifting the block position, that is, the frequency characteristic determination area for each block, and detects the position of the multiplexed block.

Separation Unit The separation unit 94 uses the position of the block detected by the detection unit 92 as a reference, and extracts the multiplexed additional information using the determination result of the frequency characteristic of each block. For example, assume that a plurality of types of additional information are multiplexed and arranged on a recording medium 1502 as recorded material C. The number of blocks for each type of additional information is 126 blocks (12 blocks horizontally x 9 blocks vertically). Under this condition, an image captured by the camera-equipped mobile terminal 1506 is as shown in FIG. 18, for example.

FIG. 18 shows a captured image 1701 obtained by capturing a recorded image 1504 in recorded material C using a camera-equipped mobile terminal 1506, as shown in FIG. 15(a). In FIG. 18, a captured image 1802 indicates a captured image, and a block 1804 indicates a block to be multiplexed. In the captured image 1802, all blocks are included for two types of additional information, and only some blocks are included for four pieces of additional information.

Additional information "0" and "1" are embedded in each block of a block group in which one type of additional information is embedded by multiplex encoding processing. Therefore, additional information embedded in each block is determined based on the frequency vector for each block. That is, for a block in which the frequency vector of the straight line l1 (see FIG. 16(a)) exceeds a predetermined threshold value, the additional information embedded in that block is determined to be "0". Further, for a block in which the frequency vector of the straight line l2 (see FIG. 16(a)) exceeds a predetermined threshold value, the additional information embedded in that block is determined to be "1".

- Analysis unit The analysis unit 96 analyzes the numerical string separated as additional information by the separation unit 94, and converts it into the format of the original embedded additional information. For example, it is assumed in advance that the additional information to be multiplexed is text document data, and the character code is a value obtained by converting the character code into a numerical value using "Shift JIS". For Shift JIS 1x and codes (half-width characters), conversion corresponding to numerical values and characters can be performed by combining the upper 4 bits and the lower 4 bits. For example, if the upper 4 bits are "0100" and the lower 4 bits are "0001", this numerical string is determined to be the character "A".

In this way, by holding a conversion map in advance and associating this conversion map with a numerical value string, the numerical value string can be converted into characters. The separated numerical value string as additional information is temporarily held in the RAM 76 so that it can refer to the "shift JIS" conversion map previously held in the secondary storage device 78.

For example, assume that the numerical value string of the additional information separated by the separation unit 94 is "0110100001100101011011000110110001101111". In this case, if it is made to correspond to the conversion map, it will be as follows. The upper 4 bits "0110" and the lower 4 bits "1000" form the character "h". The upper 4 bits "0110" and the lower 4 bits "0101" form the character "e." The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1111" form the character "o". Therefore, "hello" is extracted as a character string.

When the additional information is extracted, for example, the extracted character string is displayed on the display 80 and notified. Further, if the extracted character string is a URL (Uniform Resource Locator), it connects to the network via the wireless LAN 84 and uses a browser to display and notify the screen of the URL destination on the display 80. If the URL is a video site, the display 80 displays the video and the speaker 86 outputs audio to notify the user.

Thus, in this embodiment, the detection section 92, the separation section 94, and the analysis section 96 (that is, the separation device 70) function as an extraction section that extracts additional information. Furthermore, the analysis unit 96 functions as a notification unit that notifies additional information. Although details will be described later, if there is extracted additional information, the analysis unit 96 displays the extracted additional information on the display 80 in a selectable manner. Then, the analysis unit 96 selects the additional information selected by the user from among the plurality of additional information displayed on the display 80 and notifies the selected additional information. That is, in this embodiment, the analysis section 96 functions as a display control section that controls the display contents of the display 80 so as to display a plurality of extracted additional information. Furthermore, the analysis section 96 functions as a selection section that selects predetermined additional information from among a plurality of pieces of additional information.

<Multiplex decoding process>
Next, the specific processing contents of the multiplex decoding process in the multiplex decoding processing device 14, that is, the process of extracting additional information embedded in the recorded image of the recorded object C will be explained. FIG. 19 is a flowchart showing the details of the multiplex decoding process according to this embodiment. The series of processes shown in the flowchart of FIG. 19 is performed by the CPU 72 loading the program code stored in the ROM 74 into the RAM 76 and executing it. Alternatively, the functionality of some or all of the steps in FIG. 19 may be performed in hardware, such as an ASIC or electronic circuit.

When the multiplex decoding process is started, first, a captured image of a recorded image in the multiplexed recorded object C is acquired (S1902). That is, in 1902, the image capturing unit 88 captures the recorded image based on the user's operation. Alternatively, a captured image obtained by capturing the recorded image output from an imaging device (not shown) connected to the multiplex decoding processing device 14 is obtained. The acquired captured image is transmitted to the color adjustment unit 90 together with a sensor value obtained during image capturing, in which the captured light is converted into a color data value.

Next, the color adjustment unit 90 performs color adjustment of the acquired captured image (S1904). The color adjustment unit 90 receives color data values from the imaging unit 88, performs white balance adjustment, and generates the white balance adjusted color data as image data. The color adjustment unit 90 transmits the generated image data to the separation device 70. Alternatively, the generated image data is stored in the secondary storage device 78.

Then, the detection unit 92 receives the image data whose white balance has been adjusted by the color adjustment unit 90 from the color adjustment unit 90 or acquires it from the secondary storage device 78, and determines the frequency characteristics of the image data. and detects the multiplexed position (S1906).

Thereafter, the CPU 72 determines whether the detection unit 92 has been able to detect the multiplexed position from the image data (S1908). If it is determined that it could not be detected, the process returns to S1902. On the other hand, if it is determined that the detection has been successful, the separation unit 94 determines the frequency characteristics based on the image data generated by the color adjustment unit 90 and the multiplexed position detected by the detection unit 92. is used to extract the multiplexed additional information as numerical data (S1910). The separation unit 94 transmits the extracted numerical data to the analysis unit 96. Alternatively, it is temporarily stored in the RAM 76 and the analysis unit 96 is notified of this.

Next, the analysis unit 96 acquires numerical data corresponding to the separated additional information, analyzes the numerical data, and converts the numerical data into additional information such as characters (S1912). Then, the CPU 72 determines whether acquisition of additional information from all the numerical data extracted by the analysis unit 96 has been completed (S1914). If it is determined that acquisition of additional information from all numerical data has not been completed, the process returns to S1902. Further, when it is determined that acquisition of additional information from all numerical data has been completed, this multiplex decoding process is ended.

If the additional information can be extracted from the recorded image based on the multiplexed image data, the result is displayed on the display 80 or the like or the network is accessed through notification processing, which will be described later. If additional information cannot be extracted from the multiplexed recorded image, for example, it is possible that the captured image does not include an area large enough to extract additional information. In this case, since only part of the additional information could be extracted, the data is not complete and it is necessary to capture the image again.

In order to determine whether the additional information has been extracted, for example, the additional information itself may include a value indicating the amount of data of the additional information. Then, the data amount of the additional information may be determined by extracting the data amount. When determining whether it is data amount or character data, for example, you can decide on a combination of numerical strings in advance, and then determine the number of bits immediately after that numerical string as the data amount. It becomes possible to extract.

If only part of the additional information can be extracted, for example, the extracted content may be stored in the secondary storage device 78, and when additional information can be extracted, the additional information may be combined. In other words, the additional information may be extracted multiple times. Furthermore, although it is determined in S1914 that the extraction data acquisition is complete, the additional information that has been extracted may be displayed on the display 80 or the like.

<Notification of additional information>
Here, notification to the user of the additional information extracted by the multiplex decoding process will be explained. In the multiplex decoding processing device 14, when the extraction of additional information is completed by the multiplex decoding process, a notification process for notifying the extracted additional information is executed. FIG. 20 is a flowchart showing the details of the notification process. The series of processes shown in the flowchart of FIG. 20 is performed by the CPU 72 loading the program code stored in the ROM 74 into the RAM 76 and executing it. Alternatively, the functionality of some or all of the steps in FIG. 20 may be performed in hardware, such as an ASIC or electronic circuit.

In the notification process, first, the analysis unit 96 acquires additional information extracted in the multiplex decoding process (S2002). In S2002, all the additional information extracted by the multiplex decoding process is acquired. That is, if a plurality of pieces of additional information are acquired through the multiplex decoding process, in S2002, the multiple pieces of additional information extracted through the multiplex decoding process are acquired.

Next, the analysis unit 96 determines whether or not there is a plurality of pieces of additional information that has been acquired (S2004). A notification is made (S2006), and this notification processing is ended. That is, in S2006, if the acquired additional information is text information, the character string is displayed on the display 80, and if the additional information is a URL, the screen at the URL destination is displayed on the display 80, and if necessary, the speaker Output audio.

Further, if it is determined in S2004 that there is a plurality of additional information, the analysis unit 96 displays information regarding the acquired plurality of additional information in a selectable manner on the display 80 (S2008). That is, in S2008, the number of additional information is made clear to the user and displayed on the display 80 so that the user can select it. Specifically, for example, each piece of information regarding additional information is assigned a number and displayed so that the number can be selected. Alternatively, information related to additional information is arranged and displayed so that the information can be selected. Alternatively, information related to additional information is displayed in a list form using a list box or the like.

Information related to additional information may be any information that can distinguish between multiple pieces of acquired additional information. For example, if the additional information is text document data, it is information indicating that it is a text document; If it is a URL, this is information indicating that it is a URL. Note that the information regarding additional information may include the additional information itself. Therefore, if it is text document data, the character string or the like represented by the data may be displayed.

The display of information regarding additional information in S2008 is not limited to the example described above, and may be displayed in any form as long as the number of additional information is clear and the additional information can be selected. It's okay. Further, in S2008, in addition to displaying information regarding additional information, a notification prompting the user to select desired information from among the displayed plurality of information may be sent via the display 80 or the like.

After displaying information regarding a plurality of additional information in a selectable manner on the display 80, the analysis unit 96 next determines whether or not the information regarding the displayed additional information has been selected (S2010). In this embodiment, the information regarding the additional information displayed on the display 80 can be selected by the user via the key input device 82. Therefore, in S2010, it is determined whether information related to the displayed additional information has been selected by operating the key input device 82. Note that S2010 is repeatedly executed, for example, until it is determined that information regarding additional information has been selected. In this case, if the above determination is performed for a predetermined period of time, it may be determined that no selection has been made by the user, and this notification process may be terminated.

If it is determined in S2010 that information related to additional information has been selected, a notification based on the additional information corresponding to the information related to the selected additional information is performed (S2012), and this notification processing is ended. That is, in S2012, if the additional information corresponding to the selected additional information is text information, the text is displayed on the display 80, and if the additional information is a URL, the screen at the URL destination is displayed on the display 80. At this time, audio is output from the speaker as necessary.

=Effects of this embodiment=
As described above, in the information processing system 10 according to the first embodiment, when a plurality of pieces of additional information included in a recorded image are extracted, information related to the extracted additional information is displayed in a selectable manner. Then, in response to the user's selection of information regarding additional information, a notification is executed based on the additional information corresponding to the selected information. Thereby, in the information processing system 10, the user can appropriately utilize the extracted additional information.

(Second embodiment)
Next, an information processing system according to the second embodiment will be described with reference to FIG. 21. In the following description, the same reference numerals as those used in the first embodiment are used for the same or corresponding configurations as those in the first embodiment described above, and a detailed explanation thereof will be omitted.

The information processing system according to the second embodiment differs from the information processing system according to the first embodiment in that when displaying information related to extracted additional information, information corresponding to the same additional information is not displayed. It's different.

When capturing recorded images of recorded material C with embedded additional information using a mobile terminal with a camera, the embedded additional information differs depending on the position where the mobile terminal is held up, that is, the imaging position of the mobile terminal. It ends up. In order to reliably extract additional information, the same additional information is embedded in multiple areas in the recorded image. Then, by repeating the multiplex decoding process while changing the imaging position of the mobile terminal, additional information can be reliably extracted.

However, when multiplexing and decoding processing is repeated while changing the imaging position on the mobile terminal, the same additional information will be extracted many times. Therefore, when displaying extracted additional information in a selectable manner on the display 80, if all the extracted additional information is displayed, the display 80 will repeatedly display the same information, increasing the amount of information displayed and causing the user to It becomes difficult to select desired additional information.

Therefore, in this embodiment, when repeating the multiplex decoding process while changing the imaging position, only the extracted additional information that has not yet been displayed is displayed on the display 80. In the following description, parts that are different from the information processing system 10 according to the first embodiment will be described in detail, and descriptions of the same parts will be omitted.

=Multiplex decoding processing device=
<Notification of additional information>
FIG. 21 is a flowchart showing the processing details of the notification process executed after the multiplex decoding process in the information processing system 10 according to the second embodiment. FIG. 21 is a flowchart showing the details of the notification process. The series of processes shown in the flowchart of FIG. 21 is performed by the CPU 72 loading the program code stored in the ROM 74 into the RAM 76 and executing it. Alternatively, the functionality of some or all of the steps in FIG. 21 may be performed in hardware such as an ASIC or electronic circuit.

In the notification process in FIG. 21, first, additional information extracted in the multiplex decoding process is acquired (S2102). Then, it is determined whether or not there is a plurality of pieces of additional information that has been acquired (S2104), and if it is determined that there is not a plurality of pieces of additional information, the process advances to S2110, which will be described later. Note that the specific processing contents of S2102 and S2104 are the same as those of S2002 and S2004 described above. Furthermore, if it is determined in S2104 that there is a plurality of additional information, the analysis unit 96 determines whether there is any additional information that matches each other among the plurality of additional information acquired in S2102 (S2106). .

If it is determined in S2106 that there is no matching additional information, the process advances to S2110, which will be described later. Further, if it is determined in S2106 that there is additional information that matches each other, the analysis unit 96 regards the additional information that matches each other as one additional information (S2108), proceeds to S2110, and displays information regarding the additional information on the display. 80, it is displayed in a selectable manner. In S2110, information regarding additional information is displayed in the same manner as in S2008 above. Specifically, if there is one piece of additional information, only that piece of additional information is displayed on the display 80 in a selectable manner. Furthermore, if there is a plurality of pieces of additional information, all pieces of additional information are displayed on the display 80 in a selectable manner. Further, if additional information is already displayed, new additional information is displayed on the display 80 in a selectable manner in addition to the displayed additional information. Further, in S2110, the additional information displayed is stored in, for example, the RAM 76.

Next, the analysis unit 96 determines whether a method related to the displayed additional information has been selected (S2112). The specific processing content of S2112 is the same as 2010 above. If it is determined in S2112 that information related to additional information has been selected, a notification based on the additional information corresponding to the information related to the selected additional information is performed (S2114), and this notification processing is ended. Note that the specific processing content of S2114 is the same as that of S2012 above. Further, if it is determined in S2112 that information related to additional information is not selected, multiplex decoding processing is executed (S2116). That is, in S2116, multiplex decoding processing shown in FIG. 19 is executed. At this time, for example, a notification is given to the user to prompt the user to acquire captured images of different imaging areas, and multiplex decoding processing is performed based on the captured images.

When the multiplex decoding process in S2116 is completed, the analysis unit 96 next acquires the additional information extracted in the multiplex decoding process (S2118). Then, the analysis unit 96 determines whether or not there is a plurality of acquired additional information (S2120), and if it is determined that there is not a plurality of additional information, the process proceeds to S2126, which will be described later. Note that the specific processing contents of S2118 and S2120 are the same as those of S2002 and S2004 described above. Further, if it is determined in S2118 that there is a plurality of additional information, the analysis unit 96 determines whether there is any additional information that matches each other among the additional information acquired in S2118 (S2122). If it is determined in S2122 that there is no matching additional information, the process advances to S2124, which will be described later. Further, if it is determined in S2122 that there is additional information that matches each other, the analysis unit 96 regards the additional information that matches each other as one additional information (S2124). Thereafter, the process advances to S2126, and the analysis unit 96 determines whether there is any additional information displayed on the display 80 among the additional information acquired in the multiplex decoding process in S2116 (processed in S2124).

In S2126, it is determined whether there is additional information displayed on the display 80 by comparing the additional information acquired in the multiplex decoding process in S2116 and the additional information stored in the RAM 76. If it is determined in S2126 that the acquired additional information is not included in the additional information stored in the RAM 76, that is, it is not the additional information displayed on the display 80, the process advances to S2110, and the acquired additional information is displayed on the display. At 80, additional information is displayed in a selectable manner. Further, if it is determined in S2126 that the acquired additional information is included in the additional information stored in the RAM 76, that is, there is additional information displayed on the display 80, the additional information is not displayed on the display 80. Additional information is acquired (S2128). Thereafter, the process advances to S2110, and the additional information acquired in S2128 is additionally displayed in a selectable manner on the display 80.

=Effects of this embodiment=
As described above, in the information processing system 10 according to the second embodiment, multiplex decoding processing is repeatedly performed, and information regarding additional information acquired through the multiplexing decoding processing is selectively displayed on the display 80. Note that the information regarding additional information displayed on the display 80 excludes information regarding additional information that has already been displayed, and additionally displays information regarding additional information that has not yet been displayed. This simplifies the information related to additional information displayed on the display 80, making it easier for the user to select information related to desired additional information.

Other Embodiments
The above-described embodiment may be modified as shown in (1) to (8) below.

(1) In the above embodiment, the information regarding the additional information acquired through the multiplex decoding process is displayed in a selectable manner on the display 80, and the additional information is notified according to the user's selection. However, the present invention is not limited to this. It is not something that will be done. That is, a predetermined condition may be set and additional information satisfying the predetermined condition may be notified. For example, the type of additional information is set as the predetermined condition. Specifically, when the text display mode is set to select text document data, additional information such as a URL is not selected, but additional information that becomes text document data is automatically selected, and the text document data is displayed on the display 80. A string based on is displayed.

(2) Although not specifically described in the above embodiment, the information regarding the additional information displayed in a selectable manner on the display 80 may be such that only one can be selected, or multiple items can be selected using check boxes or the like. You can do it like this. Further, although not specifically described in the above embodiment, the information regarding the additional information displayed on the display 80 may be displayed, for example, in the order in which the additional information is extracted.

(3) Although not specifically described in the above embodiment, information regarding the priority is embedded in the additional information, and the display order of information regarding the additional information on the display 80 is determined based on the information regarding the priority. It's okay. Alternatively, additional information to be notified may be automatically selected and notified from among the extracted additional information based on the information regarding the priority.

Although not specifically described in the above embodiment, a priority is set for each type of additional information, and based on the set priority, each type of additional information is added on the display 80 in the order in which it is extracted. Information related to information may also be displayed. Alternatively, additional information to be notified may be automatically selected and notified from among the extracted additional information based on the set priority.

The priority is determined, for example, according to the data size of the additional information and the simplicity of the data. For example, the easier it is to display, the higher the priority. Specifically, the smaller the data size, the higher the priority. Also, in the case of video data and text data, the text data has a higher priority.

(4) Although not specifically described in the above embodiment, information regarding the position where the additional information is embedded is embedded in the additional information, and the information regarding the position embedded in the additional information is displayed on the display 80 along with the information regarding the additional information. You can do it like this. The information regarding the position may be any information that can limit the position in the recorded image, such as coordinate values, top, bottom, left, and right, and may also include whether or not the image is rotated if the image is rotated. This allows the user to more appropriately select desired additional information if the position where the additional information is embedded in the recorded image has meaning.

(5) Although not specifically described in the above embodiment, an object is set in advance, and information regarding the object is embedded in the additional information. Then, among the plurality of extracted additional information, additional information including information regarding the set object may be preferentially selected and notified. Alternatively, an object may be recognized from a captured image, and from among a plurality of extracted additional information, additional information including information regarding the recognized object may be preferentially selected and notified. Note that the object is, for example, personal information such as a father, mother, child, etc.

(6) In the above embodiment, the multiplex encoding processing device 12 is composed of the multiplexing device 16 and the recording device 18, but the present invention is not limited to this. That is, the multiplexing encoding processing device 12 includes a multiplexing section that can embed additional information in image data, and a recording section that can record based on the image data in which the additional information has been embedded in the multiplexing section. It is also possible to have a different configuration.

(7) The present invention provides a program that implements one or more functions of the above embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. This can also be realized by reading and executing processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

(8) The embodiments described above and the various forms shown in (1) to (7) above may be combined as appropriate.

10 Information processing system 70 Separation device 92 Detection section 94 Separation section 96 Analysis section

Claims (12)

Extracting means for extracting additional information included in the captured image;
a display control means for controlling the display means to selectably display information regarding each of the plurality of additional information extracted by the extraction means;
Notification control means for controlling to perform notification based on additional information corresponding to information selected by the user among information regarding each of the plurality of additional information displayed under the control of the display control means ;
has
The display control means displays a second captured image captured after the first captured image while displaying information on the first additional information extracted from the first captured image on the display means. Even if the first additional information is included in the one or more additional information extracted from the image, the one or more additional information is not additionally displayed on the display means. An information processing apparatus characterized in that, when information includes second additional information that is not displayed on the display means, control is performed so that information regarding the second additional information is added and displayed. further comprising storage control means for controlling the storage means to store additional information corresponding to the information displayed by the display control means;
The display control means includes:
Among the one or more additional information extracted from the second captured image, additional information included in the additional information stored in the storage means is controlled not to be added and displayed on the display means,
Among the one or more pieces of additional information extracted from the second captured image, additional information that is not included in the additional information stored in the storage means is controlled to be added and displayed on the display means.
The information processing device according to claim 1, characterized in that: By comparing the one or more additional information extracted from the second captured image with the additional information stored in the storage means, the one or more additional information extracted from the second captured image is further comprising determining means for determining whether there is additional information corresponding to the information displayed on the display means,
The information processing apparatus according to claim 2, wherein the display control means controls whether or not to display the additional information based on a determination by the determination means. When information related to the first additional information extracted from the first captured image is displayed on the display means, the user is prompted to acquire a captured image in a different imaging area from the first captured image. The information processing device according to claim 1, wherein the information processing device provides a prompting notification. 5. The information processing apparatus according to claim 1, wherein the information displayed under the control of the display control means is information that allows the corresponding additional information to be distinguished. The display control means controls to display information regarding each of a plurality of additional information ,
6. The information processing apparatus according to claim 1, wherein the information displayed under the control of the display control means includes a number assigned to each of the plurality of additional information. The notification control means may display, as the notification, additional information corresponding to the information selected by the user as a text, display a screen at a URL destination of a URL that is additional information corresponding to the information selected by the user, 7. The control apparatus according to any one of claims 1 to 6, wherein the control is performed so as to output at least one of audio data or audio of video data with audio, which is additional information corresponding to the information selected. The information processing device described. The additional information is information about the position where the additional information is embedded , and includes information that can limit the position in the image ,
The information processing apparatus according to any one of claims 1 to 7, wherein the display control means displays information regarding the position together with information regarding additional information. 9. The display control means controls the information regarding each of the plurality of additional information to be displayed in descending order of data size of each additional information. Information processing device. A method for controlling an information processing device capable of extracting additional information included in a captured image, the method comprising:
a display control step of controlling the display means to selectably display information regarding each of the plurality of additional information extracted from the captured image;
a notification control step of performing control to perform notification based on additional information corresponding to information selected by the user among information regarding each of the plurality of additional information displayed by the control in the display control step ;
has
The display control step includes displaying a second captured image captured after the first captured image while displaying information on the first additional information extracted from the first captured image on the display means. Even if the first additional information is included in the one or more additional information extracted from the image, the one or more additional information is not additionally displayed on the display means. An information processing apparatus characterized in that when the information includes second additional information that is not displayed on the display means, control is performed to add and display information regarding the second additional information. Control method. A program for causing a computer to function as the information processing apparatus according to claim 1 . A computer-readable storage medium storing a program for operating a computer as the information processing apparatus according to claim 1.

Images