JP7438752B2 - Information processing device, control method for information processing device, storage medium, and program - Google Patents

Description

The present invention relates to an information processing apparatus, a control method for the information processing apparatus, a storage medium, and a program that acquire additional information embedded as a digital watermark in an image and perform predetermined processing.

Patent Document 1 describes that a Uniform Resource Locator (hereinafter referred to as URL) is embedded in an image as additional information, and a user accesses a network by reading the embedded URL using software.

Japanese Patent Application Publication No. 2005-051793

However, with the method of Patent Document 1, additional information embedded in an image cannot be changed. Therefore, the processing performed by reading the additional information embedded by the software is always the same processing, and it is not possible to perform processing that is tailored to the situation, such as the date and time or location when the user read the image. Ta.

Therefore, the present invention provides an information processing device , a control method for the information processing device, a storage medium, and a program that can read additional information from an image and execute processing according to the situation such as the date and time and place where the user read the image. provide.

Therefore, the information processing apparatus of the present invention includes: an acquisition unit that captures an image to obtain the captured image ; an extraction unit that extracts additional information embedded in the image as a digital watermark from the captured image; It is characterized by comprising a control means for controlling the additional information extracted by the extraction means to be displayed on the touch panel when touch information corresponding to touching is acquired.

According to the present invention, an information processing apparatus , a control method for an information processing apparatus, a storage medium, and a program are capable of reading additional information from an image and executing processing according to the situation such as the date and time and place when the user read the image. can be provided.

FIG. 2 is a diagram showing a hardware configuration for multiplex encoding. FIG. 2 is a diagram showing the hardware configuration of the additional information multiplexing device in FIG. 1(a). FIG. 2 is a diagram showing the configuration of hardware that extracts additional information from image information. FIG. 2 is a block diagram showing a firmware configuration that performs multiplex encoding processing. 3 is a flowchart showing image restoration processing for decoding image information. FIG. 3 is a diagram showing a color correction table. It is an image diagram of tetrahedral interpolation. FIG. 3 is a diagram showing an example of a color separation table. FIG. 3 is a diagram illustrating an error distribution method in the error diffusion method. FIG. 3 is a diagram showing mask data corresponding to binary data. 3 is a flowchart showing multiplex encoding processing. FIG. 2 is a block diagram showing a firmware configuration of a multiplexing unit. FIG. 3 is a diagram digitizing mask patterns with different frequency characteristics. 7 is a flowchart showing recording processing after multiplex encoding processing. FIG. 3 is a block configuration and image diagram of firmware for multiplex decoding processing. FIG. 2 is a diagram showing differences in frequency characteristics and a diagram of a recording medium multiplexed on a block-by-block basis. FIG. 3 is a diagram for explaining detection positions when determining frequency characteristics. 3 is a flowchart showing multiplex decoding processing. 2 is a flowchart showing an image diagram and extraction processing for extracting an image. 2 is a flowchart showing an image diagram and extraction processing for extracting an image. 2 is a flowchart showing an image diagram and extraction processing for extracting an image. 2 is a flowchart showing an image diagram and extraction processing for extracting an image. 2 is a flowchart showing an image diagram and extraction processing for extracting an image. It is a flow chart showing extraction processing.

An embodiment regarding an image in which additional information is embedded as a digital watermark will be described below. The digital watermark here is information embedded in an image in a state that is invisible or difficult to visually recognize. Such information can be extracted by the device analyzing the image. Details of this analysis will be described later. In the following description, embedding of information as a digital watermark will be expressed as "multiplexing" of the information into an image.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. First, the basic configurations of a multiplex encoding processing section that embeds additional information in image information and a multiplexing decoding processing section that extracts additional information from a photographed image (imaging information) will be explained.

First, the multiplex encoding processing section and multiplex encoding processing will be explained.
FIG. 1 is a diagram showing a hardware configuration for multiplex encoding in which additional information B is embedded in image information A in this embodiment. This hardware acquires image information A and additional information B, and generates printed matter C in which additional information B is embedded in the image information A.

The hardware in FIG. 1A includes an additional information multiplexing device 102 and a recording device 103. The additional information multiplexing device 102 performs a process of embedding additional information B in image information A, and the recording device A printed matter C in which the additional information B is embedded in the image information A is generated. The hardware in FIG. 1(b) includes an additional information multiplexing unit 105 and a recording unit 106 inside a recording device 103, and the additional information multiplexing unit 105 performs a process of embedding additional information B into image information A. , the recording unit 106 generates a printed matter C in which the additional information B is embedded in the image information A. The hardware configuration that can realize this embodiment may be either the configuration shown in FIG. 1(a) or FIG. 1(b).

In the configuration of FIG. 1A, image information A input from the input terminal 100 is multi-gradation image information including color components. Additional information B input from the input terminal 101 includes text document information, audio information, video information, compressed text document information, audio information, image, video information, and other information converted into binary values. It is. The multiplexing device 102 performs multiplexing processing to embed additional information B in image information A, as described later. The recording device 103 generates a printed matter C by performing a recording operation based on the image information A' in which the additional information B is embedded.

In the configuration of FIG. 1(b), a multiplexing unit 105 corresponding to the multiplexing device 102 of FIG. 1(a) is included in the recording device 103, and as in the configuration of FIG. Image information A is input from 100, and additional information B is input from input terminal 101. A multiplexing unit 105 in the recording device 103 performs a process of embedding additional information B into image information A. Further, the recording unit 106 generates a printed matter C by performing a recording operation based on the image information A' in which the additional information B is embedded. The process of generating printed matter C based on image information A' in which additional information B is embedded in this manner is also referred to as "multiplex encoding process."

FIG. 2 is a diagram showing the hardware configuration of the additional information multiplexing device 102 in FIG. 1(a). The CPU 202 is a central processing unit, and executes, for example, a process of multiplexing additional information B according to a program. ROM203 stores programs executed by CPU202. The RAM 204 provides a memory for temporarily storing various information when the CPU 202 executes a program. The secondary storage device 205 is, for example, a hard disk, and is a storage medium for storing image files, additional information, and the like. The display 206 displays a user interface screen, processing details, and the like. The key input device 207 accepts processing instructions, settings, character input, etc. through the operation of a device such as a keyboard. The network interface 208 is connected to, for example, a LAN (Local Area Network). Further, the LAN is connected to the Internet. The CPU 202 accesses a site connected to the Internet through the network interface 208, displays a screen of the site on the display 206, and transmits and receives information. The additional information multiplexing device 102 is, for example, an information processing device such as a computer or a smartphone, but may be any other type of device as long as it can execute the process of embedding the additional information B in the image information A.

FIG. 3 is a diagram showing a hardware configuration for extracting additional information B from image information A' recorded on printed matter C. Here, a mobile terminal 301 will be described as an example of hardware that extracts additional information B from image information A'.

The mobile terminal 301 captures an image of a printed matter C that has undergone multiplex encoding processing using an imaging device such as a camera, and extracts additional information B embedded in the image by analyzing the captured image.

A camera-equipped mobile terminal (information processing device) 301 holding an image sensor 302 has a function of photographing printed matter C. The additional information B separation device 303 extracts additional information B by analyzing the image captured by the image sensor 302, as will be described later. A CPU (central processing unit) 304 executes an information processing method according to a program, and a ROM 305 stores a program executed by the CPU 304. The RAM 306 functions as a memory for temporarily storing various information when the CPU 304 executes a program. A secondary storage device 307 such as a hard disk stores a database including image files and image analysis results. The display 308 presents the processing results of the CPU 304 to the user.

A key input device 309 uses a display 308 with a touch panel function to issue processing instructions, input characters, etc. by operating the touch panel. A wireless LAN (Local Area Network) 310 is connected to the Internet, accesses sites connected to the Internet, and displays the site's screen on the display 308. The wireless LAN 310 is also used for transmitting and receiving data. The speaker 311 outputs audio when the extracted additional information B is audio information or video data with audio. Furthermore, if video data is available at the Internet connection destination, audio is output when the video data is played back. The accelerometer 312 is equipped with a motion sensor, and shows the number of steps taken while carrying the mobile terminal 301 and the direction in which the mobile terminal 301 is facing.

The gyroscope 313 senses what attitude the mobile terminal 301 is maintaining, further improving the accuracy of the accelerometer 312. A GPS (Global Positioning System) 314 communicates with an artificial satellite and presents the position of the mobile terminal 301 on the earth. The second image sensor 315 is an image sensor that has the same function as the image sensor 302, and can take images and videos like the image sensor 302. Clock 316 manages time. Although the user can manually adjust the time, it is also possible to acquire the time on the server via the wireless LAN 310 and adjust it automatically. Since the time changes depending on the country and location on the globe, the user has to adjust it manually or automatically.

The mobile terminal 301 can be configured using a smartphone, a tablet, a PC, a digital camera, a video camera, or a wearable terminal. Furthermore, the mobile terminal 301 is not limited to a configuration in which the image sensor 302 is built-in. For example, a configuration may be adopted in which the image sensor 302 is controlled by a device other than the mobile terminal 301 and the captured image is transmitted to the additional information separation device 303. As the image sensor 302, a digital camera, a video camera, or the like can be used. Further, as the additional information B separation device 303, a personal computer, a smartphone, etc. can be used, and any configuration that can extract the additional information B from the printed matter C is sufficient.

FIG. 4 is a block diagram showing the basic firmware configuration for performing multiplex encoding processing. The attached information acquisition unit 401 acquires various parameters used when compressing the image information A. The obtained various parameters are used in the restoration unit 402 for processing to extract image information from the compressed image. Further, the obtained various parameters are used in processing for calculating the degree of compression. For example, the input image is irreversible image information obtained by compressing document information in a JPEG format, and is recorded on a recording medium. The irreversible image information includes the quantization table used during compression and the image information size. The acquired image information size information and quantization table are sent to the image information restoration unit 402.

The image information restoration unit 402 decodes the encoded image information and extracts the image information. In the following, the input image will be explained as a JPEG image.

Here, FIG. 5 is a flowchart showing image restoration processing for decoding encoded image information. The image restoration process will be explained below using this flowchart. The processing in FIG. 5 is realized, for example, by the CPU 202 reading a program stored in the ROM 203 into the RAM 204 and executing it. Here, it is assumed that a JPEG format image is divided into N 8-pixel square blocks (8×8 pixel blocks).

When the image restoration process is started, the CPU 202 sets n=1 in S51, and performs Huffman encoding on the nth block in S52. Huffman encoding is a compression method that reduces the overall number of bits by assigning a code with a short number of bits to frequently occurring data. In Huffman decoding, a Huffman code is defined in advance in specifications and then decoded into original data. After that, in S53, the CPU 202 performs dequantization using the quantization table. In the dequantization, the quantization table (quantization table used when compressing the image information) acquired by the attached information acquisition unit 401 It is developed into image information by inverse quantization using . Next, in S54, the CPU 202 performs inverse DCT transformation on the expanded image information. In the inverse DCT transformation, image information that has been DCT-converted into a direct current component (DC component) and an alternating current component (AC component) is inverse transformed to return it to the original image density component data.

Then, in S55, the CPU 202 sets n=n+1, and in S56, the CPU 202 determines whether N=n, and if it is not determined, returns to S52 and repeats the process. In this way, the above processing is executed for all N blocks of the target screen.

JPEG compression is often performed in a luminance Y, color difference Cb, Cr format, and in that case, data subjected to inverse DCT processing also becomes a YCbCr format. A value in YCbCr format is converted into an image signal value in RGB format using Equation 1 below.
Formula 1:
R = Y + 1.402×Cr
G = Y - 0.344 x Cb - 0.714 x Cr
B = Y + 1.772×Cb

Returning to the block diagram of FIG. 4, the image correction unit 403 performs image correction processing on the RGB data composited by the image information restoration unit 402. Image corrections include brightness adjustment, contrast adjustment, and color balance adjustment to brighten or darken the overall color, as well as various corrections such as backlight correction and red-eye correction for photographic recording. By processing these corrections in an integrated manner in the image correction unit 403, it is possible to realize processing that does not depend on the recording device.

A resolution conversion unit 404 converts the image information to a resolution compatible with the recording device. Enlargement or reduction processing is performed based on the amount of magnification derived according to the input image information and the resolution of the recording device. As the scaling process, nearest neighbor, bilinear, bicubic, etc. exist, and may be selected as appropriate in consideration of processing characteristics and processing speed.

The color correction unit 405 performs conversion processing on the image information so that the image recorded by the recording device 103 has a suitable color. For example, when recording an image displayed on a display device, the color reproduction ranges of the display device and the recording device do not necessarily match. The reproduction range of a recording device may be narrower for a certain color, and the reproduction range of the recording device may be wider for another color. Therefore, it is necessary to compress and expand colors as appropriate while minimizing image deterioration.

In this embodiment, these conversion processes are performed in RGB format. That is, in consideration of the reproducibility of the printing apparatus 103, the RGB values input to the color correction unit 405 are converted into RGB values for the printing apparatus (hereinafter also referred to as "RGB for printing apparatus"). This conversion can also be performed by calculations such as a matrix. Generally, a three-dimensional color correction table 411 is used. When input RGB values are 8 bits for each color (256 gradations), it is not practical to retain all combinations from the viewpoint of storage capacity, so the color correction table 411 is thinned out at predetermined intervals. Use a table.

Here, FIG. 6 is a diagram showing the color correction table 411, and FIG. 7 is an image diagram of tetrahedral interpolation. The color correction table 411 of this embodiment is a table in which each color has 256 gradations as 17 grid points, and the corresponding RGB values for the printing device are described (17*17*17=4913 grid points). Values between grid points are calculated using interpolation processing. The interpolation method can be selected from among several methods, and in this embodiment, the tetrahedral interpolation method was used. The tetrahedral interpolation method is linear interpolation using four lattice points, with a tetrahedron as the division unit of the three-dimensional space. In this tetrahedral interpolation method, the three-dimensional space is first divided into tetrahedrons as shown in Fig. 7(a), and then it is determined to which of the divided tetrahedrons the target point p belongs. decide. The four vertices of the tetrahedron are designated as p0, p1, p2, and p3, respectively, and the tetrahedron is further divided into smaller small tetrahedrons as shown in FIG. 7(b). If the converted values of each point are respectively f(p0), f(p1), f(p2), and f(p3), then the interpolated value f(p) can be obtained using the following equation 2.

Here, w0, w1, w2, and w3 are the volume ratios of the small tetrahedrons at positions opposite to each vertex pi. In this way, the RGB values for the recording device corresponding to the target RGB values are calculated. In that case, the output may be set to 8 bits or more in consideration of gradation. Further, since the color correction table also depends on the color reproduction range of the recording device, for example, if the recording paper (recording medium) used for recording is different, it is desirable to prepare a table corresponding to that.

Returning to the block diagram of FIG. 4, the ink color conversion unit 406 converts the recording device RGB values converted by the color correction unit 405 into ink color values. This conversion uses a color separation table 412 in which combinations of RGB values for the printing device and ink color values are associated in advance. Similar to the color correction unit 405, the ink color conversion unit 406 also uses the table of grid points 17.

Here, FIG. 8 is a diagram showing an example of the color separation table 412. In the color separation table 412 of this embodiment, four colors of cyan (C), magenta (M), yellow (Y), and black (K) are assumed as ink colors, and four color values corresponding to each grid point are assumed. is included. These values are determined by taking into consideration that the ink will not overflow on the recording surface of the recording paper and that the ink will not bleed when adjacent to each other. Therefore, if the recording paper used for recording is different, it is desirable to prepare a color separation table 412 corresponding to the recording paper. Further, similarly to the color correction unit 405 described above, the ink color values corresponding to the RGB values for the printing apparatus can be interpolated by tetrahedral interpolation processing.

Returning to the block diagram of FIG. 4, the density correction unit 407 corrects the recording density of the image. Generally, in an inkjet recording apparatus, as the amount of ink applied to form dots on recording paper increases, the overlap of the dots increases, making it difficult to increase the recording density of an image. The density correction unit 407 corrects the density in order to make such density responsiveness uniform.

By density correction, the color correction table 411 and color separation table 412 can be created with high precision. In a recording apparatus that uses C (cyan), M (magenta), Y (yellow), and K (black) inks, density correction is performed for these ink colors. In this embodiment, a one-dimensional density correction table 413 is used. As for the table, it is sufficient to prepare a table corresponding to 8 bits of input (256 gradations) for each ink color, and a table that associates the input signal value with the corrected output signal value without any thinning. can be used.

The gradation conversion unit 408 converts the multi-bit data, which has been converted for each ink color and subjected to density correction, into the number of gradations that can be printed by the printing apparatus. In this embodiment, the conversion is performed into two gradations (1 bit): recording "1"/non-recording "0". An error diffusion method is used that can reproduce gradations. Furthermore, 8-bit data from 0 to 255 is assumed as the input signal.

FIG. 9 is a diagram illustrating an error distribution method in the error diffusion method. The signal value of the target pixel is defined as the signal value L, the threshold value TH is defined as the signal value L, and the threshold value TH is compared. In this embodiment, in order to binarize 0 to 255, the threshold value TH is set to 127, and as shown below, the target pixel is either "1" (recorded) or "0" (non-recorded). Determine whether
L>TH ・・・・・・ 1 (record)
L≦TH ・・・・・・ 0 (non-recording)

According to this determination result, the quantization representative value V is set as follows.
1 (record) 255
0 (non-recording) ・・・・・・ 0

By setting the quantization representative value V in this way, the generated error E (=LV) is distributed to surrounding pixels according to the distribution coefficient shown in FIG.

The value La obtained by adding the distributed error Ea to the signal value L of the next target pixel is compared with the threshold value TH, and it is determined whether the target pixel is "1" (recorded) or "0" as shown below. ” (non-recording).
La>TH ・・・・・・ 1 (record)
La≦TH ・・・・・・ 0 (non-recording)

By performing such processing for all pixels and all ink colors C, M, Y, and K, printable 1-bit print data for each ink color is obtained.

Returning to the block diagram of FIG. 4, additional information 409 is additional information B embedded in image information A in the multiplexing device 102 of FIG. 1(a) or the multiplexing unit 105 of FIG. etc. The text document information is, for example, numerical information in which numbers and letters are assigned to numerical values using already known character codes, and this numerical information is transmitted to the multiplexing unit 410 as additional information 409.

As a specific example, text document information corresponding to the characters "hello" will be explained. The text document information is assumed to be numerical information, so-called binary data. Binary data is information of ``0'' or ``1'', and the continuous connection of this ``0'' or ``1'' information has a specific meaning. The correspondence between binary data and characters is defined by a "character code." In the case of "Shift JIS", which is one of the character codes, "h" corresponds to binary data "01101000".

Similarly, "e" corresponds to the binary data "01100101," "l" corresponds to "01101100," and "o" corresponds to the binary data "01101111." Therefore, the character "hello" can be expressed as "0110100001100101011011000110110001101111" in binary data. Conversely, if the binary data "0110100001100101011011000110110001101111" can be obtained, the characters "hello" can be obtained. Additional information 409 corresponds to such numerical information converted into binary data.

Additional information multiplexing section 410 receives the image information converted by resolution converting section 404 and additional information 409, and embeds additional information 409 into the image information. In this embedding process (multiplexing process), the additional information 409 (such as a text document converted into binary data of "0" and "1") can be read from the recorded image of the image information in which the additional information 409 is embedded. The additional information 409 is embedded in the image information as shown in FIG. For example, so that the binary data of "0" and "1" of the additional information 409 can be read, the image information is masked and the information of "0" and "1" corresponding to the binary data is embedded.

Here, FIGS. 10A and 10B are diagrams showing mask data corresponding to binary data of "0" and "1". By performing mask processing on the image data, image information in a predetermined area is given different periodicity corresponding to binary data of "0" and "1". Details of the processing in additional information multiplexing section 410 that uses these mask data will be described later.

FIG. 11 is a flowchart showing multiplex encoding processing. The multiplex encoding process will be explained below using this flowchart. The processing in FIG. 11 is realized, for example, by the CPU 202 reading a program stored in the ROM 203 into the RAM 204 and executing it. When the multiplex encoding process is started, the CPU 202 acquires image information A for recording using the attached information acquisition unit 401 and the image information restoration unit 402 in S11. For example, image information A is data that has been photographed in advance by the mobile terminal 301 and stored in the memory within the mobile terminal in JPEG format. The acquired JPEG image information A is decompressed to generate three-color, 8-bit RGB image information of a still image. Further, the image correction unit 403 performs correction or processing on the acquired image information A, if necessary.

After that, the CPU 202 acquires additional information 409 to be embedded in the image information A in S12. For example, obtain text document information keyed on a smartphone. The text document information is, for example, numerical information in which numbers and characters are assigned to numerical values using the well-known character code shift JIS. The numerical information is transmitted to multiplexing section 410 as additional information 409. Then, in S13, the CPU 202 performs resolution conversion processing on the acquired image information A based on the arbitrarily selected size of the recording medium and the resolution of the recording device 103. For example, when the size of the selected recording medium is 2L, the resolution of the image information A is converted in accordance with the number of pixels of the input resolution in the recording device 103.

Specifically, when the input resolution of the recording device 103 is 600 dpi (dots per inch), the number of pixels of the recording medium size 2L is set to 3000 pixels x 4000 pixels. In this case, for image information A having a pixel count of 1500 pixels x 2000 pixels, resolution conversion is performed so that the number of pixels in the vertical direction and horizontal direction is doubled. When it is not desired to change the aspect ratio of an input image, resolution conversion is performed with the same enlargement and reduction ratios in the vertical and horizontal directions. Thereafter, in S14, the CPU 202 uses the additional information multiplexing unit 410 to perform additional information multiplexing processing for embedding the additional information 409 in the image information A, and ends the process.

FIG. 12 is a block diagram showing the firmware configuration of the multiplexing section 410 in this embodiment. Each processing unit in the multiplexing unit 410 will be explained below.

The color space conversion unit 1201 is a processing unit that converts the color space of the image information whose size has been changed by the resolution conversion unit 404 into a color space for information multiplexing. For example, as shown in Equation 3 below, the color space in which information is multiplexed is the color space U of YUV, and the RGB color space of image information is converted to the YUV color space.
Formula 3:
Y = 0.299 x R + 0.587 x G + 0.114 x B
U = -0.169 x R - 0.331 x G + 0.500 x B
V = 0.500 x R - 0.419 x G - 0.081 x B

The block position setting unit 1202 sets position coordinates of blocks, which will be described later. In this embodiment, image information is divided into a plurality of block areas, and the density of each pixel is modulated in block units to form a pattern corresponding to the mask data in FIGS. 10(a) and 10(b). Embed information 409. Therefore, the block position setting unit 1202 acquires the image information after color space conversion, and sets the block position coordinates for the plain image of the specified color according to the size of one specified block. . For example, the size of a U color plain image in the YUV color space is 640 px in height and 480 px in width, and the block size is 8 px in height and 8 px in width. In this case, the number of vertical blocks is 80 (=640÷8), the number of horizontal blocks is 60 (=480÷8), and the total number of blocks is 4800 (=80×60). For example, the upper left coordinates of each block can be set as the block position.

The digitization unit 1203 converts the received additional information 409 into digitized data. For example, the additional information 409 is a Shift JIS character string. In this case, a conversion map in which characters and numbers are associated in accordance with the Shift JIS format is stored in advance, and the conversion map is used to convert the character string into a numerical value string. For example, in the case of the character string "hello", the converted numerical string becomes "0110100001100101011011000110110001101111".

The pattern selection unit 1204 has registered mask patterns for performing density modulation of each pixel in block units, and selects a mask pattern to be applied to the additional information 409 digitized by the digitization unit 1203. .

Here, FIGS. 13(a) and 13(b) are diagrams in which the mask patterns of FIGS. 10(a) and 10(b) having different frequency characteristics are digitized. As mentioned above, the mask patterns in FIGS. 10(a) and 13(a) correspond to the binary data "0" of the additional information 409, and the patterns in FIGS. 10(b) and 13(b) correspond to This corresponds to “1” in the binary data of the additional information 409. The black area 901 in FIGS. 10(a) and 10(b) corresponds to "2" in FIGS. 13(a) and 13(b). Similarly, a white area 902 corresponds to "0", and a diagonally shaded area 903 corresponds to "-1".

The information multiplexing unit 1205 acquires the image information subjected to color space conversion in the color space conversion unit 1201, the position of each block set in the block position setting unit 1202, and the mask pattern selected in the pattern selection unit 1204. . The information multiplexing unit 1205 generates image information by applying a mask pattern to the image information from these acquired information.

When photographing a recorded image of printed matter C, it is not always possible to photograph the entire image. Therefore, the same additional information is embedded in multiple locations in the recorded image so that the additional information can be extracted by just photographing a portion of the recorded image of the printed matter C. For example, when 80 blocks are used as one additional information, the same additional information is divided into 60 (=4800÷80) areas and embedded for the total number of blocks, 4800. Therefore, the image information is divided into 60 areas, and 80 blocks of additional information, each block having a vertical width of 8 px and a width of 8 px, are embedded in one of these areas. By treating 80 blocks as one piece of additional information, 80 bits of additional information can be set. However, in order to know the starting position of the 80 bits, the 8 bits of "11111111", which are not expressed as characters in Shift JIS, are included at the beginning of the additional information. Therefore, 72 (=80-8) bits of data becomes additional information.

The data that fits within 80 bits is a numeric string of "0" and "1" of additional information digitized by the digitization unit 1203. A numeric value is defined for each block of 8px x 8px, and a value corresponding to the numeric value is defined for each block of 8px x 8px. A mask pattern is selected. A mask pattern corresponding to the additional information is embedded in an 8px×8px block of image information. For example, the image data is a YUV U color plane, which is processed block by block, and the values of the mask patterns shown in FIGS. 12(a) and 12(b) are applied to the values of the YUV U color plane. For example, as shown below, the values of the YUV U color plane (U value) are added and subtracted according to the numerical values of these mask patterns, and the reference value for the addition and subtraction processing is set to 10.
Formula 4:
U value after application = YUV U value + reference value × mask pattern value

For example, if the U value of one pixel in one block is "20" and the numerical value of the applied mask pattern is "0", the U value is processed as shown in equation 5 below.
Formula 5:
U value after application = 20 + 10 × 0 = 20

Further, when the U value of one pixel in one block is "30" and the numerical value of the applied mask pattern is "2", the U value is processed as shown in Equation 6 below.
Formula 6:
U value after application = 30 + 10 × 2 = 50

In this manner, in this embodiment, multiplexing is achieved by adding the value obtained by multiplying the numerical value of the mask pattern applied to each pixel by the reference value. The method for applying the mask pattern is not limited to the method of this embodiment as long as the mask pattern can be embedded on the U color plane, and for example, the U value of YUV may be multiplied by the numerical value of the mask pattern. .

Such multiplex encoding processing is executed by the additional information multiplexing device 102 in FIG. 1(a) or the additional information multiplexing unit 105 in FIG. 1(b). The multiplex encoding process may or may not be processed by the recording device 103. The multiplexed encoded image information generated in the multiplexing device 102 or the multiplexing unit 105 is transmitted to the recording device 103 or the recording unit 106.

FIG. 14 is a flowchart showing image information recording processing after multiplex encoding processing. The process in FIG. 14 is realized, for example, by a CPU included in the recording device reading a program stored in a ROM into a RAM and executing it. Hereinafter, the recording process of image information after the multiplex encoding process will be explained using this flowchart.

When the recording process of the image information after the multiplex encoding process is started, the CPU acquires image information (multiplexed image data) in which the additional information 409 is embedded in the additional information multiplexing unit 410 in S31. Next, in S32, the CPU performs appropriate color correction on the multiplexed image data using the color correction unit 405. Next, in S33, the CPU converts the image information after color correction into an ink color value in the ink color conversion unit 406, density correction unit 407, and gradation conversion unit 408, and then corrects the density. Recording data is generated by converting the image information into the number of gradations. The generated recording data is sent to the recording engine. Then, in S34, the CPU controls the recording engine to generate printed matter C by applying ink of each color to the recording medium based on the recording data.

The multiplexing decoding processing section and multiplexing decoding processing will be explained below.
FIG. 15(a) is a diagram showing a block configuration of basic firmware for multiplex decoding processing in this embodiment, and FIG. 15(b) shows a photograph of a printed matter subjected to multiplex encoding processing using a mobile terminal 301 equipped with a camera. FIG. Additional information embedded in printed matter is extracted through multiplex decoding processing. The details of the multiplex decoding processing section will be explained below.

The imaging sensor 302 includes an imaging section 1501 and a color adjustment section 1502. The additional information separation device 303 includes a multiplexing position detection section 1503, an additional information separation section 1504, and an extracted data analysis section 1505. In this embodiment, it is assumed that digitized additional information data such as text document data, audio data, and video data is added as a pattern onto the printed material 1411. For example, the same information is repeatedly added to each region of the entire printed matter C as the additional information. The imaging unit 1501 converts the multiplexed and encoded printed matter C into captured image data using an imaging element within the imaging sensor 302 .

The CCD, which is the image sensor included in the image sensor 302, is already a well-known technology, so a detailed explanation will be omitted. Briefly, in a CCD, a light receiving element called a photodiode senses light and can convert the light into voltage. At this time, the light can be converted into color data by passing it through color filters such as RGB and CMY arranged for each element. Sensor values captured by the imaging unit 1501 are sent to a color adjustment unit 1502.

The color adjustment unit 1502 converts the sensor value taken out by the photodiode in the imaging unit 1501 into image data, for example, as 1-pixel YUV, 8-bit data. It also performs color adjustment processing that is normally performed during shooting, such as white balance and brightness adjustment.

The multiplexing position detection unit 1503 acquires image data obtained by photographing and color-adjusting multiplexed printed matter. The multiplexing position detection unit 1503 acquires frequency characteristics for the acquired image data. The multiplexing position detection unit 1503 detects the multiplexed position by determining the acquired frequency characteristics.

FIG. 16(a) is a schematic diagram showing differences in frequency characteristics in a two-dimensional frequency domain. The horizontal axis shows the frequency in the horizontal direction, and the vertical axis shows the frequency in the vertical direction. The origin, which is the center, shows a DC component, and the further away from the origin, the higher the frequency becomes. In this embodiment, the frequency characteristics of the YUV U component of the multiplexed printed matter change by switching the mask patterns shown in FIGS. 13(a) and 13(b).

For example, suppose that a large power spectrum occurs on the frequency vector of the straight line 1601 in FIG. 16(a) due to the change in the characteristic of the U component shown in FIG. 13(a). Further, it is assumed that due to the change in the characteristic of the U component shown in FIG. 13(b), a large power spectrum occurs on the frequency vector of the straight line 1602 in FIG. 16(a). When separating additional information, the multiplexed signal can be determined by detecting the frequency vector where this large power spectrum occurs.

13A and 13B correspond to an HPF (high pass filter) having a specific frequency vector directionality. FIGS. 13(a) and 13(b) are also used as spatial filters when detecting frequency vectors. That is, the spatial filter in FIG. 13(a) can emphasize the frequency vector on the straight line 1601, and the spatial filter in FIG. 13(b) can emphasize the frequency vector on the straight line 1602. It becomes possible.

For example, suppose that a large power spectrum is generated on the frequency vector of straight line 1601 in FIG. 16(a) due to the mask pattern in FIG. 13(a). In that case, the spatial filter of FIG. 13(a) amplifies the amount of change in the power spectrum, but the spatial filter of FIG. 13(b) hardly amplifies it. That is, when a plurality of spatial filters are filtered in parallel, amplification occurs only when the spatial filters have matching frequency vectors, and almost no amplification occurs when using other filters. Therefore, it is possible to easily determine on which frequency vector a large power spectrum occurs.

In addition, this time, we changed the frequency characteristics of the U component of YUV and detected a frequency vector using the U component when separating additional information from multiplexed printed matter, but if the U component is lost, the additional information cannot be separated. . If the characteristics shift to other Y or V components instead of the U component, additional information can be separated by determining the frequency characteristics using the Y or V components, but it takes more time than analyzing only the U component. Put it away.

As described above, by determining the frequency characteristics, data can be extracted. However, when determining frequency characteristics, if the position to be extracted is shifted, it becomes difficult to extract correctly.

FIG. 17 is a diagram for explaining detection positions when determining frequency characteristics. Image 1701 shows an image that is multiplexed in four blocks of printed matter. Areas 1702 and 1703 indicate areas where frequency characteristics are determined on a block-by-block basis. In FIG. 17, a region 1702 indicates that the position of the multiplexed block is shifted. Area 1703 indicates that it matches the position of the block being multiplexed. In this case, in the region 1703, it is possible to correctly identify a predetermined frequency, but in the region 1702, the power spectrum of a specific frequency vector decreases, making it difficult to identify the predetermined frequency.

The multiplexing position detection unit 1503 detects the block position based on whether the power spectrum of a specific frequency vector is strong or weak. Therefore, the multiplexing position detection unit 1503 determines the frequency characteristics of the acquired image data after shifting the block position, and detects the position of the multiplexed block.

The additional information separation unit 1504 uses the position detected by the multiplexing position detection unit 1503 as a reference and the result of determining the frequency characteristics to extract the multiplexed additional information.

FIG. 16(b) is a diagram showing how each block is multiplexed on a recording medium. In the figure, a recording medium 1603 indicates a printing paper, and a block 1604 indicates a block to be multiplexed. It is assumed that the number of multiplexed blocks is 8 blocks horizontally and 12 blocks vertically, a total of 96 blocks. In FIG. 16(b), it is assumed that additional information "0" and "1" are embedded in each block by multiplex encoding processing.

For example, if the power spectrum of the frequency vector of the straight line 1601 in FIG. 16(a) exceeds a certain threshold, it is determined that the data is "0". Furthermore, if the power spectrum of the frequency vector on the straight line 1602 exceeds a certain threshold, it is determined that the data is "1". Based on the position detected by the multiplexing position detection unit 1503, the frequency characteristics are determined while shifting the position by 96 blocks in block units. In this case, since one bit of "0" or "1" can be determined for each block, a total of 96 bits of data can be extracted. In this way, multiplexed data can be extracted by determining the frequency characteristics while shifting the position.

The extracted data analysis unit 1505 analyzes the numerical string separated as additional information by the additional information separation unit 1504, and converts the data into the format of the additional information originally embedded. For example, it is assumed that the additional information to be multiplexed in advance is text document data, which is a value obtained by converting a character code into a numerical value using "Shift JIS".

In the Shift JIS 1-byte code (half-width characters), conversion corresponding to numerical values and characters can be performed using a combination of the upper 4 bits and lower 4 bits. For example, if the upper 4 bits are "0100" and the lower 4 bits are "0001", the character string is determined to be "A". In this way, by storing a conversion map in advance and associating numerical strings, it is possible to convert to characters.

The separated numerical value string as additional information is temporarily held in the RAM 306 in FIG. 3 so that it can refer to the "shift JIS" conversion map previously held in the secondary storage device 307.

For example, assume that the numerical string of additional information separated by the additional information separation unit 1504 is "0110100001100101011011000110110001101111." In this case, the correspondence to the conversion map is as follows. The upper 4 bits "0110" and the lower 4 bits "1000" form the character "h". The upper 4 bits "0110" and the lower 4 bits "0101" form the character "e." The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1111" form the character "o". Therefore, "hello" is extracted as a character string.

When the additional information is extracted, the extracted character string is displayed on the display 308 in FIG. 3, for example. Further, if the extracted character string is a URL (Uniform Resource Locator), it connects to the network via the wireless LAN 310 and displays the URL destination screen on the display 308 using a browser. Further, if the URL is a video site, the display 308 displays the video, and the speaker 311 outputs audio.

FIG. 18 is a flowchart showing multiplex decoding processing in this embodiment. The process in FIG. 18 is realized, for example, by the CPU 304 of the mobile terminal 301 reading a program stored in the ROM 305 into the RAM 306 and executing it.

In S1801, the CPU 304 uses the image sensor 302 to photograph the multiplexed printed matter C. The image sensor 302 transmits a sensor value obtained by converting the photographed light into a color data value to the color adjustment unit 1502. The process in S1801 corresponds to the process of acquiring an image captured by the image sensor of the image capturing unit 1501 in FIG. 15(a).

In S1802, the CPU 304 performs color adjustment of the acquired captured image. A color adjustment unit 1502 receives color data values from the imaging unit 1501, performs white balance adjustment, and generates white balance adjusted color data as image data. The color adjustment unit 1502 transmits the generated image data to the additional information separation device 303. Alternatively, the generated image data is stored in the secondary storage device 307 in FIG. The process of S1802 corresponds to the process of generating an image in which the color adjustment of white balance has been performed by the color adjustment unit 1502 in FIG. 15(a).

In S1803, the CPU 304 receives image data after white balance from the color adjustment unit 1502, or obtains image data stored in the secondary storage device 307 in FIG. The multiplexing position detection unit 1503 determines the frequency characteristics of the acquired image data and detects the multiplexed position. The process of S1803 corresponds to the process of detecting the multiplexed reference position from the image data by the multiplexing position detection unit 1503 in FIG. 15(a).

In S1804, the CPU 304 determines whether the multiplexed position detection unit 1503 has been able to detect the multiplexed reference position from the image data. If it is determined that the detection was successful, the process advances to S1805; if it is determined that the detection was not possible, the process returns to S1801 and repeats the process.

In S1805, the CPU 304 performs multiplexing using the result of determining the frequency characteristics based on the image data generated by the color adjustment unit 1502 and the multiplexed reference position detected by the multiplexing position detection unit 1503. Extract the additional information that is displayed as numerical data. The process in S1805 corresponds to the process of extracting additional information from image data by the additional information separation unit 1504 in FIG. 15(a). Additional information separation unit 1504 transmits the extracted numerical data to extracted data analysis unit 1505 in FIG. 15(a). Alternatively, the additional information separation unit 1504 temporarily stores it in the RAM 306 in FIG. 3 and notifies the extracted data analysis unit 1505 of this.

In S1806, the CPU 304 uses the extracted data analysis unit 1505 in FIG. 15A to acquire numerical data corresponding to the separated additional information, analyzes the numerical data, and converts it into additional information such as character data. The process of S1806 corresponds to the process of extracting additional information by the extracted data analysis unit 1505 in FIG. 15(a).

In S1807, the CPU 304 determines whether acquisition of the additional information extracted by the extracted data analysis unit 1505 in FIG. 15(a) has been completed. If it is determined that the acquisition of the additional information has been completed, the multiplex decoding process of FIG. 18 is terminated, and if it is determined that the acquisition is not completed, the process of S1801 is repeated.

If additional information can be extracted from the multiplexed printed matter, the result is displayed on the display 308 in FIG. 3 or accessed the network. If additional information cannot be extracted from the multiplexed printed matter, for example, there may be a case where the photographed image data does not include all the area from which additional information can be extracted. In that case, since only part of the additional information could be extracted, the data is not complete and it is necessary to take the image again.

The characteristic configuration of this embodiment will be described below.
FIG. 19(a) is a diagram showing the mobile terminal 201 extracting an image in which additional information is embedded, and FIG. 19(b) is a flowchart showing additional information extraction processing by the mobile terminal 301. . In this embodiment, as additional information, a part of the URL such as "http://www.XXX.XX.XX/" and date threshold information such as "180219" are extracted from the image. Note that "180219" indicates February 19, 2018. In addition to the extracted additional information, date and time information on the date and time when the user extracted the additional information is acquired from the clock 316 of the mobile terminal 301. The additional information extraction process of this embodiment will be described below using the flowchart of FIG. 19(b). The process in FIG. 19(b) is realized, for example, by the CPU 304 of the mobile terminal 301 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, in step S1901, the CPU 304 photographs a printed matter using the image sensor 302 to obtain additional information embedded in the printed matter. Here, the mobile terminal 301 acquires a part of the URL and date threshold information from the printed material. Thereafter, the CPU 304 acquires date and time information from the clock 316 in S1902. Then, in step S1903, the CPU 304 creates a URL from the acquired additional information and date and time information.

Specifically, the date threshold information included in the additional information is compared with the date and time information acquired from the mobile terminal 301, and the URL to be created is determined depending on whether the date in the date and time information is before or after the date in the date threshold information. Make it different. For example, if the date and time information, which is the date when the printed matter was photographed, is "February 17, 2018," this is before February 19, 2018, which is indicated by the date threshold information "180219," so the URL to be combined is Create "http://www.XXX.XX.XX/before/". Also, if the date and time information is "February 20, 2018," this is after February 19, 2018, which is indicated by the date threshold information "180219," so the URL to be synthesized is "http://www. Create “XXX.XX.XX/after/”.

If the date of the date threshold information and the acquired date and time information are the same, either "http://www.XXX.XX.XX/before/" or "http://www.XXX.XX.XX/after /'' is set in advance to be one of the URLs, and one of the URLs is determined based on the settings. Thereafter, in step S1904, the CPU 304 accesses the created URL, displays the content at the URL destination on the mobile terminal 301, and ends the process. In this manner, in this embodiment, the content of subsequent processing can be switched depending on the acquired date and time information.

The case assumed in this embodiment is to embed additional information in a wedding invitation, display a message video from the invitee before the wedding, and display the wedding video after the wedding. A case is possible.

In this embodiment, an example in which a threshold value is set for the date has been described, but the present invention is not limited to this, and a threshold value may be set for the date and time, and the date and time may be acquired as the date and time information.

Further, in this embodiment, the additional information and the date and time information are combined in the mobile terminal 301, but the additional information and the date and time information are sent to the server, the processing content is switched within the server, and a URL is created and the mobile The URL may be transmitted to the terminal, and the mobile terminal 301 may receive the URL.

In this way, the mobile terminal of this embodiment acquires additional information from a captured image, further acquires imaging situation information regarding the situation at the time of imaging, and performs processing based on the additional information and imaging situation information. I do. Thereby, it is possible to realize an information processing device, an information processing method, and a program that can read additional information from an image and execute processing according to the situation such as the date and time and place when the user read the image.

(Second embodiment)
A second embodiment of the present invention will be described below with reference to the drawings. Note that since the basic configuration of this embodiment is the same as that of the first embodiment, the characteristic configuration will be described below.

FIG. 20(a) is a diagram showing an image of extracting an image in which additional information is embedded using the mobile terminal 301, and FIG. 20(b) is a flowchart showing additional information extraction processing. In this embodiment, GPS information of the destination (GPS information indicating the position of a predetermined place) is acquired as additional information. Furthermore, in addition to the extracted additional information, GPS information indicating the current position of the mobile terminal 301 is acquired from the GPS 314 of the mobile terminal 301 . The additional information extraction process of this embodiment will be described below using the flowchart of FIG. 20(b). The process in FIG. 20(b) is realized, for example, by the CPU 304 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, in step S2001, the CPU 304 photographs a printed matter using the mobile terminal 301 and acquires additional information embedded in the printed matter. Here, the mobile terminal 301 acquires GPS information of the destination. Thereafter, in S2002, the CPU 304 acquires GPS information indicating the current position of the mobile terminal 301 from the GPS 314 of the mobile terminal 301. Then, in S2003, the CPU 304 calculates a route between the destination and the current location from the additional information and the GPS information. The route is calculated using a map service available on the Internet. When the user who owns the mobile terminal 301 moves and the GPS information of the current location changes, the route displayed on the map also changes. Thereafter, in step S3004, the CPU 304 displays the calculated route on the display 308 of the mobile terminal 301, and ends the process.

The present embodiment assumes a case in which additional information is embedded in the concert ticket 2011. By acquiring additional information and GPS information of the current location on the mobile terminal 301 and displaying the route from the current location to the concert venue, it is possible to guide the user from the user's location to the concert venue no matter where the user is. can.

Note that the GPS information of the destination and the GPS information of the current location may be transmitted to a route search service on the Internet, and the map and route may be received and displayed.

(Third embodiment)
A third embodiment of the present invention will be described below with reference to the drawings. Note that since the basic configuration of this embodiment is the same as that of the first embodiment, the characteristic configuration will be described below.

FIG. 21(a) is a diagram showing an image of extracting an image in which additional information is embedded using the mobile terminal 301, and FIG. 21(b) is a flowchart showing additional information extraction processing. In this embodiment, the price of the product is acquired as additional information. In addition to the extracted additional information, the user's country of residence information is also acquired from the mobile terminal 301. The additional information extraction process of this embodiment will be described below using the flowchart of FIG. 21(b). The process in FIG. 21(b) is realized, for example, by the CPU 304 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, the CPU 304 photographs the printed matter with the mobile terminal 301 and acquires the additional information embedded in the printed matter 2111 in step S2102. Here, the mobile terminal 301 obtains the price of the product, for example "$2". Thereafter, in step S2102, the CPU 304 acquires the user's country of residence information from the mobile terminal. In this embodiment, the user's country of residence information uses the settings of the mobile terminal 301, but other methods such as GPS history or user input may also be used. Also, you may select any country instead of your country of residence. Then, in step S2103, the CPU 304 uses the Internet exchange conversion service from the acquired price information, for example "$2" and the user's country of residence information, for example "Japan", to convert the price of the price information into the currency of the country of residence. Get the price converted to . Specifically, if the acquired price is "$2" and the exchange rate is $1=100 yen, the converted price information will be "200 yen". The converted price information is displayed in S2104 and the process ends.

The case assumed in this embodiment is a case where additional information is embedded in the package 2011 of a product. For example, assume that you are traveling overseas from Japan to the United States and want to know the price of a local product converted into Japanese yen at the current exchange rate.

Note that in this embodiment, the price of the product obtained from the package of the product is converted to the price of the country of residence, but the present invention is not limited to this. For example, the present invention can be applied to any item that requires price information, such as a receipt, an estimate, or a financial statement.

(Fourth embodiment)
A fourth embodiment of the present invention will be described below with reference to the drawings. Note that since the basic configuration of this embodiment is the same as that of the first embodiment, the characteristic configuration will be described below.

FIG. 22(a) is a diagram showing an image of extracting an image in which additional information is embedded by the mobile terminal 301, and FIG. 22(b) is a flowchart showing additional information extraction processing. In this embodiment, a URL is embedded as additional information in an explanation display 2212 of a painting 2211 in an art museum. The additional information extraction process of this embodiment will be described below using the flowchart of FIG. 22(b). The process in FIG. 22(b) is realized, for example, by the CPU 304 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, in step S2201, the CPU 304 photographs the explanation display 2212 using the mobile terminal 301, and acquires the additional information embedded in the explanation display 2212. An explanation display 2212 describes a simple explanation of the painting 2211, and by photographing the explanation display 2212 with the mobile terminal 301, a URL is obtained as additional information. The same URL is embedded in the description display 2212 of multiple works of art, and the URL destination contains "layout information" indicating where in the museum the work of art is displayed, and "details of each work of art". "Explanatory text" is written.

After that, the CPU 304 acquires GPS information and terminal orientation information from the mobile terminal 301 in S2202. GPS information is obtained from the GPS 314, and the user's location within the museum can be obtained using this GPS information. Furthermore, terminal orientation information is obtained from the accelerometer 312 and gyroscope 313, and the orientation of the mobile terminal 301 can be obtained from this terminal orientation information. In step S2203, the CPU 304 identifies the art object that the user is looking at in the museum based on the layout information of the URL destination, the user's position, and the orientation of the mobile terminal 301, and sends a detailed description of the identified art object to the mobile device. The terminal 301 obtains the information. Thereafter, in step S2204, the CPU 304 displays a detailed explanation of the artwork that the user is viewing on the mobile terminal 301, and ends the process.

By embedding the same additional information in every art object, even if the layout or explanation display of the art object is changed, the user's mobile terminal 301 will display a detailed explanation of the art object that the user is looking at. be done.

In this embodiment, an example of an art museum is shown, but it can also be used in a zoo, an aquarium, or the like. Furthermore, the present invention may be applied to a treasure hunting game so that information can be acquired only when the article and the smartphone are in a predetermined position and in a predetermined orientation. Further, the number of steps taken by the user may be measured using the accelerometer 212 and the gyroscope 213, and the content may be switched according to the number of steps taken by the user.

Furthermore, this embodiment can be more effectively applied by using a camera mounted on a glasses-type wearable terminal. For example, it is possible to acquire additional information from a printed matter viewed from a glasses-type wearable terminal at an art museum and display the corresponding information.

(Fifth embodiment)
A fifth embodiment of the present invention will be described below with reference to the drawings. Note that since the basic configuration of this embodiment is the same as that of the first embodiment, the characteristic configuration will be described below.

FIG. 23(a) is a diagram showing an image of extracting an image in which additional information is embedded by the mobile terminal 301, and FIG. 23(b) is a flowchart showing additional information extraction processing. In this embodiment, additional information is embedded in the printed matter 2311, and the mobile terminal 301 acquires touch information after the printed matter 2311 is photographed. The additional information extraction process of this embodiment will be described below using the flowchart of FIG. 23(b). The process in FIG. 23(b) is realized, for example, by the CPU 304 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, the CPU 304 photographs the printed matter using the mobile terminal 301 and acquires the additional information embedded in the explanation display 2311 in step S2301. After that, in step S2302, the CPU 304 acquires touch information, which is information corresponding to the user touching the display 308, from the display 308 of the mobile terminal 301. A push button is recorded on the printed material 2311, and when the user photographs the printed material 2311 with the mobile terminal 301 and touches the push button of the printed material 2311 displayed on the display 308 of the mobile terminal 301, the mobile terminal 301 is touched. Get information.

In S2303, the CPU 304 determines whether the mobile terminal 301 has acquired touch information. If touch information has been acquired, the process moves to S2304, and if touch information has not been acquired, the process ends. If the process moves to S2304, the CPU 304 displays the additional information acquired in S2301 on the display 308 of the mobile terminal 301, and ends the process.

Note that the additional information is predetermined information and may be any information.For example, printed matter is an advertisement for a product with a push button recorded, and by photographing and touching the push button image on the display, it is possible to obtain additional information. A URL screen containing detailed product information may be displayed on the display. In this way, additional information can be displayed in response to the user's touch action.

(Sixth embodiment)
A sixth embodiment of the present invention will be described below with reference to the drawings. Note that since the basic configuration of this embodiment is the same as that of the first embodiment, the characteristic configuration will be described below.

FIG. 24 is a flowchart showing additional information extraction processing in this embodiment. In this embodiment, additional information is embedded in the printed material, and by photographing the printed material, the mobile terminal 301 can identify information α (viewing information) and the user ID of the user who is permitted to view the information α. Get the included additional information. Some mobile terminals are equipped with two cameras: a camera mounted on the back side of the mobile terminal for photographing the scenery, and a camera mounted on the front side for photographing the user himself/herself. In this embodiment, the mobile terminal 301 acquires additional information using a camera on the back side, and captures the user's face using a camera on the front side to obtain facial authentication information. The additional information extraction process of this embodiment will be described below using the flowchart of FIG. The processing in FIG. 24 is realized, for example, by the CPU 304 reading a program stored in the ROM 305 into the RAM 306 and executing it.

When the additional information extraction process is started, in step S2401, the CPU 304 photographs a printed matter using the mobile terminal 301 and acquires additional information embedded in the printed matter. The additional information includes information α and the user ID of the user who is permitted to view the information α. After that, in step S2402, the CPU 304 photographs the user with the mobile terminal 301 and acquires facial authentication information. In this embodiment, a case is assumed in which a first camera (image sensor 202) photographs a printed matter, and a second camera (second image sensor 215) photographs the user himself/herself. The face authentication information uses a face image acquired by the second image sensor 215 to calculate a user ID indicating a specific user. Although many methods have been proposed for face recognition, any method may be used as long as the user can be identified. Then, in step S2403, the CPU 304 determines whether the user ID obtained by face authentication matches the user ID of the user who is permitted to view the information α obtained in step S2401. If they match, the process moves to S2404; if they do not match, the process ends. When proceeding to S2404, the CPU 304 unlocks the information α and displays the information α on the display 308 of the mobile terminal 301 in S2402.

Note that in this embodiment, face authentication is used to identify the user to unlock the information α, but other methods may be used to identify the user. For example, a user may be identified using fingerprint authentication on a mobile terminal equipped with a fingerprint authentication function, or a user may be identified using voice authentication on a mobile terminal equipped with a voice authentication function.

102 Additional information multiplexing device 103 Recording device 301 Mobile terminal 304 CPU
308 Display

Claims (11)

an acquisition means for capturing an image to obtain a captured image ;
Extracting means for extracting additional information embedded in the image as a digital watermark from the captured image;
a control unit configured to display additional information extracted by the extraction unit on the touch panel when touch information corresponding to the user touching the touch panel is acquired;
An information processing device comprising : The information according to claim 1, wherein the control means controls the additional information extracted by the extraction means to be displayed on the touch panel when the user touches a specific position on the touch panel. Processing equipment. The control means controls the additional information extracted by the extraction means to be displayed on the touch panel when a user touches a specific part of the captured image displayed on the touch panel. The information processing device according to claim 1. a first acquisition means for capturing an image to obtain a captured image;
From the captured image, extract additional information that is embedded in the image as a digital watermark and includes viewing information and a first user ID that is an ID of a user who is permitted to view the viewing information. extraction means;
a second acquisition means for acquiring a second user ID that is the ID of the user who photographed the image;
a control means for controlling to display the viewing information included in the additional information extracted by the extraction means when the first user ID and the second user ID match;
An information processing device comprising : The information processing apparatus according to claim 4 , wherein the second user ID is obtained by face authentication. The first acquisition means acquires the captured image captured by a first camera,
5. The second acquisition means acquires the second user ID by face authentication from an image taken of the user by a second camera different from the first camera. Information processing device. The second acquisition means is configured to obtain an ID of a user identified by fingerprint authentication using a fingerprint authentication function provided in the information processing device or by voice authentication using a voice authentication function provided in the information processing device. The information processing apparatus according to claim 4, wherein the information processing apparatus acquires the second user ID as the second user ID. A program for causing a computer to function as each means of the information processing apparatus according to claim 1. A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing apparatus according to claim 1. an acquisition step of capturing an image to obtain a captured image;
an extraction step of extracting additional information embedded in the image as a digital watermark from the captured image;
a control step of controlling the additional information extracted in the extraction step to be displayed on the touch panel when touch information corresponding to the user touching the touch panel is acquired;
A method for controlling an information processing device, comprising: a first acquisition step of capturing an image to obtain a captured image;
From the captured image, extract additional information that is embedded in the image as a digital watermark and includes viewing information and a first user ID that is an ID of a user who is permitted to view the viewing information. an extraction step;
a second acquisition step of acquiring a second user ID that is the ID of the user who photographed the image;
a control step of controlling to display the viewing information included in the additional information extracted in the extraction step when the first user ID and the second user ID match;
A method for controlling an information processing device, comprising:

Images