JP2021106332A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device, an information processing method, and a program capable of reading additional information from an image and executing a process according to a situation such as a date, time, and place where a user read the image.SOLUTION: The information processing device acquires additional information with a mobile terminal, also acquires change information that changes according to a user's situation, and performs processing in the mobile terminal based on the additional information and the change information.SELECTED DRAWING: Figure 20

Description

The present invention relates to an information processing apparatus, an information processing method, and a program that acquire additional information embedded as a digital watermark in an image and perform a predetermined process.

Patent Document 1 describes that a user accesses a network by embedding a Uniform Resource Locator (hereinafter referred to as a URL) as additional information in an image and reading the URL embedded by software.

Japanese Unexamined Patent Publication No. 2005-051793

However, in the method of Patent Document 1, the additional information embedded in the image cannot be changed. Therefore, the process performed by reading the additional information embedded by the software is always the same process, and for example, it is not possible to perform the process according to the situation such as the date and time and place where the user read the image. rice field.

Therefore, an object of the present invention is to provide an information processing device, an information processing method, and a program capable of reading additional information from an image and executing processing according to a situation such as the date and time and place where the user reads the image. do.

Therefore, the information processing method of the present invention includes an acquisition step of capturing an image in which additional information is embedded as an electronic watermark to acquire imaging information, and an extraction step of extracting the additional information from the imaging information. The method is characterized by further including an imaging information acquisition step of acquiring imaging status information regarding the situation in the imaging, and a processing step of performing a predetermined process based on the additional information and the imaging status information. And.

According to the present invention, it is possible to provide an information processing device, an information processing method and a program capable of reading additional information from an image and executing processing according to a situation such as a date and time or place where the user reads the image. ..

It is a figure which showed the hardware structure for multiplex encoding. It is a figure which showed the hardware configuration of the additional information multiplexing apparatus of FIG. 1 (a). It is a figure which showed the structure of the hardware which extracts the additional information from the image information. It is a block diagram which showed the firmware configuration which performs the multiplexing encoding process. It is a flowchart which showed the image restoration process which decodes the image information. It is a figure which showed the color correction table. It is an image diagram of tetrahedral interpolation. It is a figure which showed an example of the color separation table. It is a figure explaining the error distribution method in the error diffusion method. It is a figure which showed the mask data corresponding to the binary data. It is a flowchart which shows the multiplexing encoding process. It is a block diagram which showed the firmware configuration of the multiplexing part. It is the figure which quantified the mask pattern with different frequency characteristics. It is a flowchart which showed the recording process after the multiplexing encoding process. It is a block configuration and an image diagram of the firmware of the multiplexing decoding process. It is a figure which shows the difference of a frequency characteristic, and the figure of a recording medium multiplexed for each block. It is a figure for demonstrating the detection position at the time of determining a frequency characteristic. It is a flowchart which shows the multiplexing decoding process. It is an image diagram to extract an image and the flowchart which showed the extraction process. It is an image diagram to extract an image and the flowchart which showed the extraction process. It is an image diagram to extract an image and the flowchart which showed the extraction process. It is an image diagram to extract an image and the flowchart which showed the extraction process. It is an image diagram to extract an image and the flowchart which showed the extraction process. It is a flowchart which showed the extraction process.

Hereinafter, embodiments relating to an image in which additional information is embedded as a digital watermark will be described. The digital watermark here is information embedded in an image invisible or difficult to see. Such information can be extracted by the apparatus analyzing the image. Details of this analysis will be described later. In the following description, the embedding of information as a digital watermark is expressed as "multiplexing" of the information in an image.

(First Embodiment)
Hereinafter, the first embodiment of the present invention will be described with reference to the drawings. First, a basic configuration of a multiplexing encoding processing unit that embeds additional information in image information and a multiplexing decoding processing unit that extracts additional information from captured images (imaging information) will be described.

First, the multiplexing encoding processing unit and the multiplexing encoding processing will be described.
FIG. 1 is a diagram showing a configuration of hardware for multiplex encoding in which additional information B is embedded in image information A in the present embodiment. This hardware acquires the image information A and the additional information B, and generates a printed matter C in which the additional information B is embedded in the image information A.

The hardware of FIG. 1A includes an additional information multiplexing device 102 and a recording device 103, and the additional information multiplexing device 102 performs a process of embedding the additional information B in the image information A, and the recording device 103. A printed matter C in which the additional information B is embedded in the image information A is generated. The hardware of FIG. 1B includes an additional information multiplexing unit 105 and a recording unit 106 inside the recording device 103, and the additional information multiplexing unit 105 performs a process of embedding the additional information B in the image information A. , The recording unit 106 generates a printed matter C in which the additional information B is embedded in the image information A. The hardware configuration that can realize this embodiment may be either the configuration shown in FIG. 1A or FIG. 1B.

In the configuration of FIG. 1A, the image information A input from the input terminal 100 is multi-gradation image information including a color component. The additional information B input from the input terminal 101 includes text document information, audio information, moving image information, text document information, audio information, images, compressed information of moving image information, and information converted into other binary values. Is. The multiplexing device 102 performs a multiplexing process for embedding the additional information B in the image information A, as will be described later. The recording device 103 generates printed matter C by performing a recording operation based on the image information A'in which the additional information B is embedded.

In the configuration of FIG. 1 (b), the multiplexing unit 105 corresponding to the multiplexing device 102 of FIG. 1 (a) is included in the recording device 103, and is an input terminal as in the configuration of FIG. 1 (a). Image information A is input from 100, and additional information B is input from input terminal 101. The multiplexing unit 105 in the recording device 103 performs a process of embedding the additional information B in the image information A. Further, the recording unit 106 generates the printed matter C by performing a recording operation based on the image information A'in which the additional information B is embedded. The process of generating the printed matter C based on the image information A'in which the additional information B is embedded is also referred to as "multiplex encoding process".

FIG. 2 is a diagram showing a hardware configuration of the additional information multiplexing device 102 of FIG. 1 (a). The CPU 202 is a central processing unit, and for example, executes a process of multiplexing additional information B according to a program. The ROM 203 stores a program executed by the CPU 202. The RAM 204 provides a memory for temporarily storing various information when the program is executed by the CPU 202. The secondary storage device 205 is, for example, a hard disk, which is a storage medium for storing image files, additional information, and the like. The display 206 displays a user interface screen, processing contents, and the like. The key input device 207 receives processing instructions, settings, character input, and the like via the operation of a device such as a keyboard. The network interface 208 is connected to, for example, a LAN (Local Area Network). Also, the LAN is connected to the Internet. The CPU 202 accesses a site connected to the Internet by the network interface 208, displays the screen of the site on the display 206, and transmits / receives information. The additional information multiplexing device 102 is, for example, an information processing device such as a computer or a smartphone, but may be another form of device as long as the process of embedding the additional information B in the image information A can be executed.

FIG. 3 is a diagram showing a configuration of hardware that extracts additional information B from image information A'recorded on printed matter C. Here, the mobile terminal 301 will be described as an example of hardware for extracting the additional information B from the image information A'.

The mobile terminal 301 extracts the additional information B embedded in the image by capturing the multiplexedly encoded printed matter C with an imaging device such as a camera and analyzing the captured image.

The mobile terminal (information processing device) 301 with a camera that holds the image pickup sensor 302 has a function of photographing printed matter C. The separation device 303 of the additional information B extracts the additional information B by analyzing the image taken by the image pickup sensor 302, as will be described later. The CPU (Central Processing Unit) 304 executes the information processing method according to the program, and the ROM 305 stores the program executed by the CPU 304. The RAM 306 functions as a memory for temporarily storing various information when the program is executed by the CPU 304. The secondary storage device 307 such as a hard disk stores an image file, a database containing image analysis results, and the like. The display 308 presents the processing result of the CPU 304 to the user.

The key input device 309 uses the display 308 having a touch panel function to instruct processing and input characters by operating the touch panel. The wireless LAN (Local Area Network) 310 is connected to the Internet, accesses a site connected to the Internet, and causes a display 308 to display a screen of the site or the like. The wireless LAN 310 is also used for transmitting and receiving data. The speaker 311 outputs audio when the extracted additional information B is audio information or video data with audio. If there is video data at the Internet connection destination, audio is output when the video data is played back. The accelerometer 312 is equipped with a motion sensor and indicates the number of steps when walking with the mobile terminal 301 and which direction the mobile terminal 301 is facing.

The gyroscope 313 senses what kind of posture the mobile terminal 301 maintains, and further improves the accuracy of the accelerometer 312. GPS (Global Positioning System) 314 communicates with an artificial satellite and presents the position of the mobile terminal 301 on the earth. The second image pickup sensor 315 is an image pickup sensor having the same function as the image pickup sensor 302, and can capture an image or a moving image in the same manner as the image pickup sensor 302. The clock 316 manages the time. The user can manually adjust the time, but it is also possible to acquire the time on the server via the wireless LAN 310 and adjust it automatically. Since the time changes depending on the country and the position on the earth, the user adjusts it manually or automatically.

The mobile terminal 301 can be configured by using a smartphone, a tablet, a PC, a digital camera, a video camera, or a wearable terminal. Further, the mobile terminal 301 is not limited to the configuration in which the image pickup sensor 302 is built. For example, the image pickup sensor 302 may be controlled by a device different from the mobile terminal 301 to transmit the captured image to the additional information separation device 303. As the image sensor 302, a digital camera, a video camera, or the like can be used. Further, as the separating device 303 of the additional information B, a personal computer, a smartphone, or the like can be used, and the configuration may be such that the additional information B can be extracted from the printed matter C.

FIG. 4 is a block diagram showing a basic firmware configuration for performing multiplex encoding processing. The attached information acquisition unit 401 acquires various parameters used when compressing the image information A. The acquired various parameters are used by the restoration unit 402 for processing for extracting image information from the compressed image. In addition, the acquired various parameters are used in the process for calculating the degree of compression. For example, the input image is lossy image information obtained by compressing document information in a JPEG format and is recorded on a recording medium. The lossy image information includes the quantization table used for compression and the image information size. The acquired image information size information and the quantization table are sent to the image information restoration unit 402.

The image information restoration unit 402 decodes the encoded image information and extracts the image information. Hereinafter, the input image will be described as a JPEG image.

Here, FIG. 5 is a flowchart showing an image restoration process for decoding the encoded image information. Hereinafter, the image restoration process will be described using this flowchart. The process of FIG. 5 is realized, for example, by the CPU 202 reading the program stored in the ROM 203 into the RAM 204 and executing the program. Here, it is assumed that the JPEG format image is divided into N 8-pixel square blocks (8 × 8 pixel blocks).

When the image restoration process is started, the CPU 202 sets n = 1 in S51, and the CPU 202 sets the nth block in Huffman coding in S52. Huffman coding is a method of compressing frequently data so as to reduce the total number of bits by assigning a code having a short number of bits. In Huffman decoding, the Huffman code is defined in advance in the specifications, and the original data is decoded. After that, the CPU 202 dequantizes using the quantization table in S53. In the dequantization, the quantization table acquired by the attached information acquisition unit 401 (the quantization table used when compressing the image information). It is expanded into image information by inverse quantization using. Next, in S54, the CPU 202 performs inverse DCT conversion of the expanded image information. The inverse DCT transform performs an inverse transform for returning the image information DCT-converted to the DC component (DC component) and the AC component (AC component) to the original image density component data.

Then, the CPU 202 sets n = n + 1 in S55, the CPU 202 determines whether or not N = n in S56, and if it is determined whether or not, returns to S52 and repeats the process. In this way, the above processing is executed for all N blocks of the target screen.

JPEG compression is often performed in the luminance Y, color difference Cb, and Cr formats, and in that case, the data subjected to the inverse DCT processing is also in the YCbCr format. The value in the YCbCr format is converted into the image signal value in the RGB format by the following equation 1.
Equation 1:
R = Y + 1.402 x Cr
G = Y − 0.344 × Cb − 0.714 × Cr
B = Y + 1.772 x Cb

Returning to the block diagram of FIG. 4, the image correction unit 403 performs image correction processing on the RGB data compounded by the image information restoration unit 402. Image corrections include brightness adjustments that brighten or darken the overall color, contrast adjustments, and color balance adjustments, as well as various corrections such as backlight correction and red-eye correction that assume the recording of photographs. By centrally processing these corrections in the image correction unit 403, it is possible to realize processing that does not depend on the recording device.

The resolution conversion unit 404 converts the image information into a resolution corresponding to the recording device. Enlargement or reduction processing is performed based on the input image information, the resolution of the recording device, and the scaling amount derived according to the input image information. As the scaling process, there are near-rest neighbor, bilinear, bicubic method and the like, which may be appropriately selected in consideration of the processing characteristics and the processing speed.

The color correction unit 405 performs conversion processing on the image information so that the image recorded by the recording device 103 has a suitable color. For example, when recording an image displayed on a display device, the color reproduction ranges of the display device and the recording device do not always match. The reproduction range of the recording device may be narrower for one color, and the reproduction range of the recording device may be wider for another color. Therefore, it is necessary to appropriately compress and expand the color while minimizing the deterioration of the image.

In this embodiment, those conversion processes are performed in RGB format. That is, in consideration of the reproducibility of the recording device 103, the RGB value input to the color correction unit 405 is converted into the RGB value for the recording device (hereinafter, also referred to as “RGB for recording device”). This conversion can also be performed by an operation such as a matrix. Generally, a three-dimensional color correction table 411 is used. When the input RGB value is 8 bits (256 gradations) for each color, it is not realistic to hold all combinations from the viewpoint of storage capacity, so the color correction table 411 is thinned out at predetermined intervals. Use a table.

Here, FIG. 6 is a diagram showing a color correction table 411, and FIG. 7 is an image diagram of tetrahedral interpolation. The color correction table 411 of the present embodiment is a table in which 256 gradations of each color are set as 17 grid points, and the corresponding RGB values for the recording device are described (17 * 17 * 17 = 4913 grid points). The value between the grid points is calculated by using the interpolation process. The interpolation method can be selected from several methods and used, and in the present embodiment, the tetrahedral interpolation method is used. The tetrahedron interpolation method is a linear interpolation method in which four grid points are used with the division unit of the three-dimensional space as a tetrahedron. In this tetrahedral interpolation method, first, as shown in FIG. 7A, the three-dimensional space is divided into tetrahedra, and then which of the divided tetrahedra the target point p belongs to is determined. decide. The four vertices of the tetrahedron are set to p0, p1, p2, and p3, respectively, and the tetrahedron is divided into finer small tetrahedrons as shown in FIG. 7 (b). When the conversion values of each point are f (p0), f (p1), f (p2), and f (p3), respectively, the interpolation value f (p) can be obtained by the following equation 2.

Here, w0, w1, w2, and w3 are the volume ratios of the small tetrahedrons at the positions opposite to each vertex pi. In this way, the RGB value for the recording device corresponding to the target RGB value is calculated. In that case, the output may be 8 bits or more in consideration of gradation. Further, since the color correction table also depends on the color reproduction range of the recording device, for example, when the recording paper (recording medium) used for recording is different, it is desirable to prepare a table corresponding to the difference.

Returning to the block diagram of FIG. 4, the ink color conversion unit 406 converts the RGB value for the recording device converted by the color correction unit 405 into the ink color value. For this conversion, a color separation table 412 in which a combination of RGB values for a recording device and an ink color value are associated in advance is used. Similar to the color correction unit 405, the ink color conversion unit 406 also uses the table of grid points 17.

Here, FIG. 8 is a diagram showing an example of the color separation table 412. In the color separation table 412 of the present embodiment, four colors of cyan (C), magenta (M), yellow (Y), and black (K) are assumed as ink colors, and four color values corresponding to each grid point are assumed. Is stored. These values are determined in consideration of the fact that the ink does not overflow on the recording surface of the recording paper and that the inks do not bleed when they are adjacent to each other. Therefore, if the recording paper used for recording is different, it is desirable to prepare a color separation table 412 accordingly. Further, similarly to the color correction unit 405 described above, the ink color value corresponding to the RGB value for the recording device can be interpolated by the tetrahedral interpolation processing.

Returning to the block diagram of FIG. 4, the density correction unit 407 corrects the recording density of the image. Generally, in an inkjet recording apparatus, as the amount of ink applied to form dots on recording paper increases, the overlap of dots increases and the recording density of an image becomes less likely to increase. The density correction unit 407 corrects the density in order to make the responsiveness of such a density uniform.

By the density correction, the color correction table 411 and the color separation table 412 can be created with high accuracy. In a recording device that uses C (cyan), M (magenta), Y (yellow), and K (black) inks, density correction is performed for those ink colors. In this embodiment, a one-dimensional density correction table 413 is used. As the table, it is sufficient to prepare a table corresponding to input 8 bits (256 gradations) for each ink color, and a table in which the input signal value and the corrected output signal value are associated with each other without any particular thinning. Can be used.

The gradation conversion unit 408 converts the multi-bit data that has been converted for each ink color and subjected to density correction into the number of gradations that can be recorded by the recording device. In the present embodiment, it is converted into two gradations (1 bit) of recording "1" / non-recording "0", and as a gradation conversion method, low frequency components of an image are excluded, which is suitable for vision. An error diffusion method that can reproduce gradation is used. Further, it is assumed that 8-bit data of 0 to 255 is used as the input signal.

Here, FIG. 9 is a diagram illustrating an error distribution method in the error diffusion method. The signal value of the target pixel is set to the signal value L, and the threshold value TH is used to compare the signal value L and the threshold value TH. In the present embodiment, the threshold value TH is set to 127 in order to binarize 0 to 255, and the target pixel is either “1” (recorded) or “0” (non-recorded) as shown below. Is determined.
L> TH ・ ・ ・ ・ ・ ・ 1 (Record)
L ≦ TH ・ ・ ・ ・ ・ ・ 0 (not recorded)

The quantization representative value V is set as follows according to the determination result.
1 (Record) ・ ・ ・ ・ ・ ・ 255
0 (non-recorded) ・ ・ ・ ・ ・ ・ 0

By setting the quantization representative value V in this way, the generated error E (= LV) is distributed to the surrounding pixels according to the partition coefficient of FIG.

The value La obtained by adding the distributed error Ea to the signal value L of the next target pixel is compared with the threshold value TH, and the target pixel is "1" (recorded) or "0" as shown below. "(Non-recording) is determined.
La> TH ・ ・ ・ ・ ・ ・ 1 (Record)
La ≤ TH ・ ・ ・ ・ ・ ・ 0 (not recorded)

By executing such processing for all pixels and all ink colors C, M, Y, and K, recordable 1-bit ink color recording data can be obtained.

Returning to the block diagram of FIG. 4, the additional information 409 is additional information B embedded in the image information A in the multiplexing device 102 of FIG. 1A or the multiplexing unit 105 of FIG. 1B, and is text document information. And so on. The text document information is, for example, numerical information in which numbers and characters are assigned to numerical values by using a character code that is already known, and this numerical information is transmitted to the multiplexing unit 410 as additional information 409.

As a specific example, the text document information corresponding to the character "hello" will be described. The text document information is numerical information, so-called binary data. Binary data is information of "0" or "1", and the continuous connection of the information of "0" or "1" has a specific meaning. The correspondence between binary data and characters is defined by the "character code". In the case of "Shift JIS" which is one of the character codes, "h" corresponds to "01101000" of binary data.

Similarly, "e" corresponds to "01100101", "l" corresponds to "01101100", and "o" corresponds to the binary data of "01101111". Therefore, the character "hello" can be expressed as "0110100001100101011011000110110001101111" in the binary data. On the contrary, if the binary data "0110100001100101011011000110110001101111" can be acquired, the character "hello" can be acquired. The additional information 409 corresponds to the numerical information converted into such binary data.

The additional information multiplexing unit 410 receives the image information converted by the resolution conversion unit 404 and the additional information 409, and embeds the additional information 409 in the image information. In this embedding process (multiplexing process), the additional information 409 can be read from the recorded image of the image information in which the additional information 409 (text document converted into binary data of "0" and "1", etc.) is embedded. As described above, the additional information 409 is embedded in the image information. For example, the image information is masked so that the binary data of "0" and "1" of the additional information 409 can be read, and the information of "0" and "1" corresponding to the binary data is embedded.

Here, FIGS. 10A and 10B are diagrams showing mask data corresponding to the binary data of "0" and "1". By applying mask processing to the image data, the image information in a predetermined area is given different periodicity corresponding to the binary data of "0" and "1". The details of the processing in the additional information multiplexing unit 410 using these mask data will be described later.

FIG. 11 is a flowchart showing the multiplexing encoding process. Hereinafter, the multiplexing encoding process will be described using this flowchart. The process of FIG. 11 is realized, for example, by the CPU 202 reading the program stored in the ROM 203 into the RAM 204 and executing the program. When the multiplexing encoding process is started, the CPU 202 acquires the image information A for recording by the attached information acquisition unit 401 and the image information restoration unit 402 in S11. For example, the image information A is data that has been previously photographed by the mobile terminal 301 and stored in a memory in the mobile terminal in a JPEG format. The acquired JPEG image information A is decompressed to generate 3-color 8-bit RGB image information of a still image. Further, if necessary, the image correction unit 403 corrects or processes the acquired image information A.

After that, the CPU 202 acquires additional information 409 to be embedded in the image information A in S12. For example, the text document information keyed on the smartphone is acquired. The text document information is, for example, numerical information in which numbers and characters are assigned to numbers by using a known character code shift JIS. The numerical information is transmitted to the multiplexing unit 410 as additional information 409. Then, in S13, the CPU 202 performs a resolution conversion process on the acquired image information A based on the size of the recording medium arbitrarily selected and the resolution of the recording device 103. For example, when the size of the selected recording medium is 2 L, the resolution of the image information A is converted according to the number of pixels of the input resolution in the recording device 103.

Specifically, when the input resolution of the recording device 103 is 600 dpi (dot per inch), the number of pixels of the recording medium size 2L is set to 3000 pixels × 4000 pixels. In this case, for the image information A having the number of pixels of 1500 pixels × 2000 pixels, the resolution is converted so that the number of pixels in the vertical direction and the number of pixels in the horizontal direction are doubled. If you do not want to change the aspect ratio of the input image, perform resolution conversion with the same enlargement and reduction ratios in the vertical and horizontal directions. After that, in S14, the CPU 202 performs the additional information multiplexing process for embedding the additional information 409 in the image information A by the additional information multiplexing unit 410, and ends the process.

FIG. 12 is a block diagram showing a firmware configuration of the multiplexing unit 410 in the present embodiment. Hereinafter, each processing unit in the multiplexing unit 410 will be described.

The color space conversion unit 1201 is a processing unit that converts the color space of the image information whose size has been changed in the resolution conversion unit 404 into a color space for information multiplexing. For example, as shown in Equation 3 below, the color space for performing information multiplexing is the U of the color space YUV, and the RGB color space of the image information is converted into the YUV color space.
Equation 3:
Y = 0.299 x R + 0.587 x G + 0.114 x B
U = -0.169 x R-0.331 x G + 0.500 x B
V = 0.500 × R − 0.419 × G − 0.081 × B

The block position setting unit 1202 sets the position coordinates of the block, which will be described later. In the present embodiment, the image information is divided into a plurality of block regions, and the density of each pixel is modulated in each block to form a pattern corresponding to the mask data of FIGS. 10A and 10B. Embed information 409. Therefore, the block position setting unit 1202 acquires the image information after the color space conversion, and sets the position coordinates of the block with respect to the plain image of the specified color according to the size of one specified block. .. For example, the size of a U-color plain image in the color space YUV is 640 px in height and 480 px in width, and the block size is 8 px in height and 8 px in width. In this case, the number of vertical blocks is 80 (= 640/8), the number of horizontal blocks is 60 (= 480/8), and the total number of blocks is 4800 (= 80 × 60). For example, the upper left coordinate of each block can be set as the block position.

The digitizing unit 1203 converts the received additional information 409 into digitized data. For example, the additional information 409 is used as a character string of Shift JIS. In this case, a conversion map in which characters and numerical values are associated with each other is stored in advance according to the Shift JIS format, and the character string is converted into a numerical string using the conversion map. For example, in the case of the character string "hello", the converted numerical string "0110100001100101011011000110110001101111" is obtained.

A mask pattern for performing density modulation of each pixel is registered in the pattern selection unit 1204, and a mask pattern to be applied to the additional information 409 digitized by the digitization unit 1203 is selected. ..

Here, FIGS. 13 (a) and 13 (b) are diagrams in which the mask patterns of FIGS. 10 (a) and 10 (b) having different frequency characteristics are quantified. As described above, the mask patterns of FIGS. 10 (a) and 13 (a) correspond to "0" of the binary data of the additional information 409, and the patterns of FIGS. 10 (b) and 13 (b) are Corresponds to "1" of the binary data of the additional information 409. The black region 901 in FIGS. 10 (a) and 10 (b) corresponds to "2" in FIGS. 13 (a) and 13 (b). Similarly, the white area 902 corresponds to "0" and the shaded area 903 corresponds to "-1".

The information multiplexing unit 1205 acquires the image information that has been color-space-converted by the color space conversion unit 1201, the position of each block set by the block position setting unit 1202, and the mask pattern selected by the pattern selection unit 1204. .. The information multiplexing unit 1205 generates image information by applying a mask pattern to the image information from these acquired information.

When the recorded image of the printed matter C is taken, it is not always possible to take the entire image. Therefore, the same additional information is embedded in a plurality of places in the recorded image so that the additional information can be extracted only by photographing a part of the recorded image of the printed matter C. For example, when 80 blocks are regarded as one additional information, the same additional information is divided into 60 (= 4800 ÷ 80) areas and embedded for the total number of blocks of 4800. Therefore, the image information is divided into 60 regions, and 80 blocks of additional information having a vertical width of 8 px and a horizontal width of 8 px as one block are embedded in one of the regions. By treating 80 blocks as one additional information, 80-bit additional information can be set. However, 8 bits of "11111111", which are not expressed as characters in Shift JIS, are included at the beginning of the additional information so that the start position of 80 bits can be known. Therefore, 72 (= 80-8) bit data becomes additional information.

The data that fits in 80 bits is a numerical string of "0" and "1" of the additional information digitized by the digitizing unit 1203, and the numerical value is defined for each block of 8px × 8px and corresponds to the numerical value. The mask pattern to be selected is selected. A mask pattern corresponding to the additional information is embedded in a block of 8px × 8px in the image information. For example, the image data is a YUV U-color plane, which is processed block by block, and the mask pattern values of FIGS. 12 (a) and 12 (b) are applied to the values of the YUV U-color plane. For example, as shown below, the value (U value) of the U color plane of YUV is subjected to addition / subtraction processing according to the numerical values of those mask patterns, and the reference value for performing the addition / subtraction processing is set to 10.
Equation 4:
U value after application = YUV U value + reference value x mask pattern value

For example, when the U value of one pixel in one block is "20" and the numerical value of the mask pattern to be applied is "0", the U value is processed as in the following equation 5.
Equation 5:
U value after application = 20 + 10 x 0 = 20

Further, when the U value of one pixel in one block is "30" and the numerical value of the mask pattern to be applied is "2", the U value is processed as in the following equation 6.
Equation 6:
U value after application = 30 + 10 x 2 = 50

As described above, in the present embodiment, multiplexing is realized by adding the value obtained by multiplying the numerical value of the mask pattern applied for each pixel and the reference value. The method of applying the mask pattern is not limited to the method of the present embodiment as long as the mask pattern can be embedded on the U color plane. For example, the U value of YUV may be multiplied by the numerical value of the mask pattern. ..

Such multiplexing encoding processing is executed by the additional information multiplexing device 102 of FIG. 1A or the additional information multiplexing unit 105 of FIG. 1B. The multiplexing encoding process may or may not be processed by the recording device 103. The image information after the multiplexing encoding process generated by the multiplexing device 102 or the multiplexing unit 105 is transmitted to the recording device 103 or the recording unit 106.

FIG. 14 is a flowchart showing a recording process of image information after the multiplexing encoding process. The process of FIG. 14 is realized, for example, by the CPU included in the recording device reading the program stored in the ROM into the RAM and executing the program. Hereinafter, the image information recording process after the multiplex encoding process will be described using this flowchart.

When the recording process of the image information after the multiplexing encoding process is started, the CPU acquires the image information (multiplexed image data) in which the additional information 409 is embedded in the additional information multiplexing unit 410 in S31. Next, in S32, the CPU performs color correction to the multiplexed image data as appropriate by the color correction unit 405. Next, in S33, the CPU uses the ink color conversion unit 406, the density correction unit 407, and the gradation conversion unit 408 to convert the color-corrected image information into ink color values, correct the density, and then correct the density. Recorded data is generated by converting the image information of the above into the number of gradations. The generated recording data is transmitted to the recording engine. Then, the CPU controls the recording engine in S34 to apply ink of each color to the recording medium based on the recorded data to generate the printed matter C.

Hereinafter, the multiplex decoding processing unit and the multiplex decoding processing will be described.
FIG. 15 (a) is a diagram showing a block configuration of the basic firmware of the multiplex decoding process in the present embodiment, and FIG. 15 (b) is a photograph of a printed matter subjected to the multiplex encoding process by a mobile terminal 301 equipped with a camera. It is a figure which showed the image to do. The additional information embedded in the printed matter is extracted by the multiplexing decoding process. Hereinafter, the details of the multiplexing decoding processing unit will be described.

The image pickup sensor 302 includes an image pickup unit 1501 and a color adjustment unit 1502. The additional information separation device 303 includes a multiplexing position detection unit 1503, an additional information separation unit 1504, and an extraction data analysis unit 1505. In the present embodiment, it is assumed that digitized additional information data such as text document data, audio data, and moving image data is added as a pattern shape on the printed matter 1411. For example, as the additional information, the same information is repeatedly added to the entire printed matter C for each area. The image pickup unit 1501 converts the multiplexed and encoded printed matter C into captured image data by the image pickup element in the image pickup sensor 302.

Since the CCD, which is an image pickup element included in the image pickup sensor 302, is a known technique, detailed description thereof will be omitted. Briefly, in a CCD, a light receiving element called a photodiode senses light and can change the light into a voltage. At that time, light can be converted as color data by passing through a color filter such as RGB or CMY arranged for each element. The sensor value captured by the image pickup unit 1501 is sent to the color adjustment unit 1502.

The color adjusting unit 1502 converts the sensor value taken out by the photodiode in the imaging unit 1501 into image data as, for example, 1 pixel YUV and 8-bit data. In addition, color adjustment processing such as white balance and brightness adjustment, which is normally performed during shooting, is also performed.

The multiplexing position detection unit 1503 acquires image data in which the multiplexed printed matter is photographed and color-adjusted. The multiplexing position detection unit 1503 acquires the frequency characteristics of the acquired image data. The multiplexing position detection unit 1503 detects the multiplexed position by determining the acquired frequency characteristic.

FIG. 16A is a schematic diagram showing the difference in frequency characteristics in the two-dimensional frequency domain. The horizontal axis shows the frequency in the horizontal direction, and the vertical axis shows the frequency in the vertical direction. The central origin indicates a DC component, and as the distance from the origin increases, the frequency becomes high. In the present embodiment, the frequency characteristics of the U component of the YUV of the multiplexed printed matter are changed by switching the mask patterns of FIGS. 13A and 13B.

For example, it is assumed that a large power spectrum is generated on the frequency vector of the straight line 1601 of FIG. 16A due to the change in the characteristics of the U component according to FIG. 13A. Further, it is assumed that a large power spectrum is generated on the frequency vector of the straight line 1602 in FIG. 16A due to the change in the characteristics of the U component according to FIG. 13B. At the time of additional information separation, the multiplexed signal can be determined by detecting the frequency vector in which this large power spectrum is generated.

13 (a) and 13 (b) correspond to an HPF (high-pass filter) having a specific frequency vector directionality. 13 (a) and 13 (b) are also used as a spatial filter at the time of frequency vector detection. That is, the spatial filter of FIG. 13A can emphasize the frequency vector on the straight line 1601, and the spatial filter of FIG. 13B can emphasize the frequency vector on the straight line 1602. It will be possible.

For example, it is assumed that the mask pattern of FIG. 13A now generates a large power spectrum on the frequency vector of the straight line 1601 of FIG. 16A. In that case, the spatial filter of FIG. 13A amplifies the amount of change in the power spectrum, but the spatial filter of FIG. 13B hardly amplifies it. That is, when a plurality of spatial filters are filtered in parallel, amplification occurs only when the spatial filters have the same frequency vector, and amplification hardly occurs in other filters. Therefore, it is possible to easily determine on what frequency vector a large power spectrum is generated.

Further, this time, the frequency vector was detected by the U component at the time of separating the additional information of the printed matter by changing the frequency characteristic in the U component of YUV, but if the U component is lost, the additional information cannot be separated. .. If the characteristics are transferred to another Y component or V component instead of the U component, additional information can be separated by determining the frequency characteristics based on the Y component or V component, but it takes longer than analyzing only the U component. It ends up.

By determining the frequency characteristics as described above, it is possible to extract data. However, when determining the frequency characteristics, if the position to be extracted is deviated, it becomes difficult to extract correctly.

FIG. 17 is a diagram for explaining a detection position when determining a frequency characteristic. Image 1701 shows an image multiplexed in four blocks of printed matter. Regions 1702 and 1703 indicate regions for determining frequency characteristics in block units. In FIG. 17, region 1702 is shown to be offset from the position of the multiplexed block. Region 1703 indicates that it coincides with the location of the multiplexed blocks. In this case, the region 1703 can correctly specify the predetermined frequency, but the region 1702 has a lower power spectrum of the specific frequency vector, which makes it difficult to specify the predetermined frequency.

The multiplexed position detection unit 1503 detects the block position based on whether the power spectrum of a specific frequency vector is strong or weak. Therefore, the multiplexing position detection unit 1503 determines the frequency characteristics of the acquired image data after imaging while shifting the block position, and detects the position of the multiplexed block.

The additional information separation unit 1504 extracts the multiplexed additional information by using the result of determining the frequency characteristic with reference to the position detected by the multiplexing position detection unit 1503.

FIG. 16B is a diagram showing how the recording medium is multiplexed for each block. In the figure, the recording medium 1603 indicates printing paper, and the block 1604 indicates a block to be multiplexed. The number of multiplexed blocks is 8 blocks in the horizontal direction and 12 blocks in the vertical direction, for a total of 96 blocks. In FIG. 16B, it is assumed that additional information "0" and "1" are embedded in each block by the multiplexing encoding process.

For example, if the power spectrum of the frequency vector of the straight line 1601 in FIG. 16A exceeds a certain threshold value, the data is determined to be “0”. Further, if the power spectrum of the frequency vector on the straight line 1602 exceeds a certain threshold value, it is determined as data "1". Based on the position detected by the multiplexing position detection unit 1503, the frequency characteristic is determined while shifting the position by 96 blocks in block units. In this case, since 1 bit of "0" and "1" can be determined for each block, a total of 96 bits of data can be extracted. By determining the frequency characteristics while shifting the positions in this way, the multiplexed data can be extracted.

The extracted data analysis unit 1505 is a process of analyzing data on a numerical string separated as additional information by the additional information separation unit 1504 and converting it into the format of the originally embedded additional information. For example, it is assumed that the additional information to be multiplexed in advance is a text document data and the character code is a numerical value by "Shift JIS".

In Shift JIS 1-byte code (half-width characters), conversion corresponding to numerical values and characters can be performed by combining the upper 4 bits and the lower 4 bits. For example, when the upper 4 bits are "0100" and the lower 4 bits are "0001", it is determined as "A" as a character string. In this way, the conversion map can be stored in advance and converted into characters by associating the numerical strings with each other.

The numerical sequence separated as additional information is temporarily stored in the RAM 306 of FIG. 3, so that the "Shift JIS" conversion map previously stored in the secondary storage device 307 can be referred to.

For example, the numerical string of the additional information separated by the additional information separation unit 1504 is set to "0110100001100101011011000110110001101111". In this case, it becomes as follows when it corresponds to the conversion map. The upper 4 bits "0110" and the lower 4 bits "1000" form the character "h". The upper 4 bits "0110" and the lower 4 bits "0101" form the character "e". The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1100" form the character "l". The upper 4 bits "0110" and the lower 4 bits "1111" form the character "o". Therefore, "hello" is extracted as a character string.

When the additional information is extracted, for example, the extracted character string is displayed on the display 308 of FIG. When the extracted character string is a URL (Uniform Resource Locator), the user connects to the network with the wireless LAN 310 and uses a browser to display the screen of the URL destination on the display 308. If the URL is a video site, the video is displayed on the display 308 and the sound is output on the speaker 311.

FIG. 18 is a flowchart showing the multiplexing decoding process in the present embodiment. The process of FIG. 18 is realized, for example, by the CPU 304 of the mobile terminal 301 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

In S1801, the CPU 304 takes a picture of the multiplexed printed matter C by the image sensor 302. The image pickup sensor 302 transmits the sensor value obtained by converting the captured light into a color data value to the color adjustment unit 1502. The process of S1801 corresponds to the process of acquiring an image captured by the image pickup sensor of the image pickup unit 1501 of FIG. 15 (a).

In S1802, the CPU 304 adjusts the color of the acquired captured image. The color adjustment unit 1502 receives the color data value from the image pickup unit 1501, adjusts the white balance, and generates the color data for which the white balance has been adjusted as image data. The color adjustment unit 1502 transmits the generated image data to the additional information separation device 303. Alternatively, the generated image data is stored in the secondary storage device 307 of FIG. The process of S1802 corresponds to the process of generating an image in which the color of the white balance is adjusted by the color adjustment unit 1502 of FIG. 15 (a).

In S1803, the CPU 304 receives the image data after white balance from the color adjustment unit 1502, or acquires the image data stored in the secondary storage device 307 of FIG. The multiplexing position detection unit 1503 determines the frequency characteristics of the acquired image data and detects the multiplexed position. The process of S1803 corresponds to the process of detecting the multiplexed reference position from the image data by the multiplexing position detection unit 1503 of FIG. 15A.

In S1804, the CPU 304 determines whether or not the multiplexing position detection unit 1503 can detect the multiplexed reference position from the image data. If it is determined that the detection was possible, the process proceeds to S1805, and if it is determined that the detection was not possible, the process returns to S1801 and the process is repeated.

In S1805, the CPU 304 multiplexes using the result of determining the frequency characteristic based on the image data generated by the color adjustment unit 1502 and the multiplexed reference position detected by the multiplexing position detection unit 1503. The additional information provided is extracted as numerical data. The process of S1805 corresponds to the process of extracting the additional information from the image data by the additional information separation unit 1504 of FIG. 15A. The additional information separation unit 1504 transmits the extracted numerical data to the extracted data analysis unit 1505 of FIG. 15 (a). Alternatively, the additional information separation unit 1504 temporarily stores the data in the RAM 306 of FIG. 3, and notifies the extraction data analysis unit 1505 to that effect.

In S1806, the CPU 304 acquires numerical data corresponding to the separated additional information by the extracted data analysis unit 1505 of FIG. 15A, analyzes the numerical data, and converts it into additional information such as character data. The process of S1806 corresponds to the process of extracting additional information by the extraction data analysis unit 1505 of FIG. 15A.

In S1807, the CPU 304 determines whether or not the acquisition of the additional information extracted by the extraction data analysis unit 1505 of FIG. 15A is completed. When it is determined that the acquisition of the additional information is completed, the multiplex decoding process of FIG. 18 is completed, and when it is determined that the acquisition is not completed, the process of S1801 is repeated.

When additional information can be extracted from the multiplexed printed matter, the result is displayed on the display 308 of FIG. 3 or the like, or the network is accessed. When the additional information cannot be extracted from the multiplexed printed matter, for example, it is conceivable that the captured image data does not include the entire area for extracting the additional information. In that case, since only a part of the additional information could be extracted, the data is not complete and it is necessary to take a picture again.

Hereinafter, the characteristic configuration of the present embodiment will be described.
FIG. 19A is a diagram showing a mobile terminal 201 for extracting an image in which additional information is embedded, and FIG. 19B is a flowchart showing a process of extracting additional information by the mobile terminal 301. .. In the present embodiment, as additional information, a part of the URL such as "http://www.XXX.XX.XX/" and the date threshold information such as "180219" are extracted from the image. In addition, "180219" shall indicate February 19, 2018. Further, in addition to the extracted additional information, the date and time information of the date and time when the user extracted the additional information is acquired from the clock 316 of the mobile terminal 301. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 19B. The process of FIG. 19B is realized, for example, by the CPU 304 of the mobile terminal 301 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the extraction process of the additional information is started, the CPU 304 takes a picture of the printed matter by the image sensor 302 in S1901 and acquires the additional information embedded in the printed matter. Here, the mobile terminal 301 acquires a part of the URL and the date threshold information from the printed matter. After that, the CPU 304 acquires date and time information from the clock 316 in S1902. Then, in S1903, the CPU 304 creates a URL from the acquired additional information and the date and time information.

Specifically, the date threshold information included in the additional information is compared with the date and time information acquired from the mobile terminal 301, and the URL to be created is created depending on whether the date of the date and time information is before or after the date of the date threshold information. Make it different. For example, if the date of the date and time information, which is the date when the printed matter was taken, is "February 17, 2018", it is before February 19, 2018 indicated by the date threshold information "180219", so it is used as the URL to be synthesized. Create "http://www.XXX.XX.XX/before/". If the date of the date and time information is "February 20, 2018", it is after February 19, 2018 indicated by the date threshold information "180219", so the URL to be synthesized is "http: // www. Create "XXX.XX.XX/after/".

If the date of the date threshold information and the acquired date and time information are the same, "http://www.XXX.XX.XX/before/" or "http://www.XXX.XX.XX/after" It is set in advance so that it becomes one of the URLs of "/", and it is determined to be one of the URLs based on the setting. After that, the CPU 304 accesses the created URL in S1904, displays the content of the URL destination on the mobile terminal 301, and ends the process. As described above, in the present embodiment, the content of the subsequent processing can be switched according to the acquired date and time information.

As a case assumed by this embodiment, additional information is embedded in the wedding invitation, a message video from the invitee is displayed before the wedding, and a wedding video is displayed after the wedding. A case is possible.

In the present embodiment, an example in which a threshold value is set for a date has been described, but the present invention is not limited to this, and a threshold value may be set for a date and a time, and the date and time may be acquired as date and time information.

Further, in the present embodiment, the additional information and the date and time information are combined in the mobile terminal 301, but the additional information and the date and time information are sent to the server, the processing contents are switched in the server, a URL is created, and the mobile device is created. It may be transmitted to the terminal and the URL may be received by the mobile terminal 301.

As described above, the mobile terminal of the present embodiment acquires additional information from the captured image, further acquires imaging status information regarding the situation at the time of imaging, and processes based on the additional information and the imaging status information. I do. This makes it possible to realize an information processing device, an information processing method, and a program capable of reading additional information from an image and executing processing according to a situation such as a date and time or place where the user reads the image.

(Second embodiment)
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. Since the basic configuration of this embodiment is the same as that of the first embodiment, a characteristic configuration will be described below.

FIG. 20A is a diagram showing an image of extracting an image in which additional information is embedded in the mobile terminal 301, and FIG. 20B is a flowchart showing an extraction process of additional information. In the present embodiment, GPS information of the destination (GPS information indicating the position of a predetermined place) is acquired as additional information. Further, in addition to the extracted additional information, GPS information indicating the current position of the mobile terminal 301 is acquired from the GPS 314 of the mobile terminal 301. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 20 (b). The process of FIG. 20B is realized, for example, by the CPU 304 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the extraction process of the additional information is started, the CPU 304 takes a picture of the printed matter by the mobile terminal 301 in S2001 and acquires the additional information embedded in the printed matter. Here, the mobile terminal 301 acquires GPS information of the destination. After that, the CPU 304 acquires GPS information indicating the current position of the mobile terminal 301 from the GPS 314 of the mobile terminal 301 in S2002. Then, in S2003, the CPU 304 calculates the route between the destination and the current location from the additional information and the GPS information. The map service on the Internet is used to calculate the route. When the user who owns the mobile terminal 301 moves and the GPS information of the current location changes, the route displayed on the map also changes. After that, the CPU 304 displays the calculated route on the display 308 of the mobile terminal 301 in S3004 and ends the process.

As a case assumed by this embodiment, a case where additional information is embedded in the concert ticket 2011 is assumed. By acquiring additional information and GPS information of the current location on the mobile terminal 301 and displaying the route from the current location to the concert hall, it is possible to guide the user from the user's location to the concert hall wherever the user is. can.

The GPS information of the destination and the GPS information of the current location may be transmitted to a route search service on the Internet, and the map and the route may be received and displayed.

(Third Embodiment)
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. Since the basic configuration of this embodiment is the same as that of the first embodiment, a characteristic configuration will be described below.

FIG. 21 (a) is a diagram showing an image of extracting an image in which additional information is embedded in the mobile terminal 301, and FIG. 21 (b) is a flowchart showing a process of extracting additional information. In this embodiment, the price of the product is acquired as additional information. Further, in addition to the extracted additional information, the user's country of residence information is acquired from the mobile terminal 301. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 21 (b). The process of FIG. 21B is realized, for example, by the CPU 304 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the extraction process of the additional information is started, the CPU 304 takes a picture of the printed matter by the mobile terminal 301 in S2102 and acquires the additional information embedded in the printed matter 2111. Here, the mobile terminal 301 acquires the price of the product, for example, "$ 2". After that, the CPU 304 acquires the user's country of residence information from the mobile terminal in S2102. In the present embodiment, the country of residence information of the user uses the set value of the mobile terminal 301, but other methods such as GSP history and user input may be used. Also, any country may be selected instead of the country of residence. Then, the CPU 304 uses the Internet exchange conversion service from the acquired price information, for example, "$ 2" and the user's country of residence information, for example, "Japan" in S2103, and sets the price of the price information in the currency of the country of residence. Get the price converted to. Specifically, if the acquired price is "$ 2" and the exchange rate is $ 1 = 100 yen, the converted price information will be "200 yen". The price information converted in S2104 is displayed and the process ends.

The case assumed by this embodiment is a case in which additional information is embedded in the product package 2011. For example, suppose you are traveling abroad from Japan to the United States and want to know the price of a local product converted to Japanese yen at the current exchange rate.

In the present embodiment, the price of the product obtained from the package of the product is converted into the price of the country of residence, but the present invention is not limited to this. For example, the present invention can be applied to receipts, quotations, financial statements, and other items that require price information.

(Fourth Embodiment)
Hereinafter, a fourth embodiment of the present invention will be described with reference to the drawings. Since the basic configuration of this embodiment is the same as that of the first embodiment, a characteristic configuration will be described below.

FIG. 22A is a diagram showing an image of extracting an image in which additional information is embedded in the mobile terminal 301, and FIG. 22B is a flowchart showing an extraction process of additional information. In the present embodiment, the URL is embedded as additional information in the explanatory display 2212 of the painting 2211 in the museum. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 22B. The process of FIG. 22B is realized, for example, by the CPU 304 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the additional information extraction process is started, the CPU 304 takes a picture of the explanatory display 2212 by the mobile terminal 301 in S2201 to acquire the additional information embedded in the explanatory display 2212. A brief description of the painting 2211 is described in the explanatory display 2212, and the URL is acquired as additional information by photographing the explanatory display 2212 with the mobile terminal 301. The same URL is embedded in the explanation display 2212 of a plurality of works of art, and "layout information" and "details of each work of art" indicating where in the museum the work of art is posted are embedded in the URL destination. Explanation ".

After that, the CPU 304 acquires GPS information and terminal orientation information from the mobile terminal 301 in S2202. GPS information is acquired from GPS314, and the position of the user in the museum can be acquired by this GPS information. Further, the terminal orientation information can be acquired from the accelerometer 312 and the gyroscope 313, and the orientation of the mobile terminal 301 can be acquired from this terminal orientation information. The CPU 304 is S2203, which identifies the work of art that the user is viewing in the museum from the layout information of the URL destination, the position of the user, and the orientation of the mobile terminal 301, and mobiles a detailed description of the specified work of art. Acquired by terminal 301. After that, in S2204, the CPU 304 displays a detailed description of the work of art that the user is viewing on the mobile terminal 301, and ends the process.

By embedding the same additional information in every work of art, even if the layout or description display of the work of art is changed, the detailed description of the work of art that the user is viewing is displayed on the user's mobile terminal 301. Will be done.

Although an example of a museum is shown in this embodiment, it can also be used in a zoo or an aquarium. Further, it may be applied to a treasure hunt game so that information can be acquired only when the article and the smartphone are in a predetermined position and a predetermined orientation. Further, the accelerometer 212 and the gyroscope 213 may be used to measure the number of steps of the user, and the content may be switched according to the number of steps of the user.

Further, the present embodiment can be applied more effectively by using a camera mounted on a glasses-type wearable terminal. For example, it is possible to acquire additional information from a printed matter in the direction viewed by a glasses-type wearable terminal in a museum and display the corresponding information.

(Fifth Embodiment)
Hereinafter, a fifth embodiment of the present invention will be described with reference to the drawings. Since the basic configuration of this embodiment is the same as that of the first embodiment, a characteristic configuration will be described below.

FIG. 23 (a) is a diagram showing an image of extracting an image in which additional information is embedded in the mobile terminal 301, and FIG. 23 (b) is a flowchart showing a process of extracting additional information. In the present embodiment, additional information is embedded in the printed matter 2311, and the mobile terminal 301 acquires the touch information of the printed matter 2311 after shooting. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 23 (b). The process of FIG. 23B is realized, for example, by the CPU 304 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the extraction process of the additional information is started, the CPU 304 takes a picture of the printed matter by the mobile terminal 301 in S2301 and acquires the additional information embedded in the explanatory display 2311. After that, in S2302, the CPU 304 acquires touch information, which is information corresponding to the user touching the display 308, from the display 308 of the mobile terminal 301. A push button is recorded on the printed matter 2311, and the mobile terminal 301 touches the printed matter 2311 when the user photographs the printed matter 2311 with the mobile terminal 301 and the user touches the push button of the printed matter 2311 displayed on the display 308 of the mobile terminal 301. Get information.

In S2303, the CPU 304 determines whether or not the mobile terminal 301 has acquired the touch information. If the touch information has been acquired, the process proceeds to S2304, and if the touch information has not been acquired, the process ends. When shifting to S2304, the CPU 304 displays the additional information acquired in S2301 on the display 308 of the mobile terminal 301 and ends the process.

The additional information is predetermined information and may be any information. For example, the printed matter is an advertisement of a product in which a push button is recorded, and is additional information by taking a picture and touching the push button image on the display. The screen of the URL in which the detailed information of the product is described may be displayed on the display. In this way, additional information can be displayed according to the touch operation of the user.

(Sixth Embodiment)
Hereinafter, a sixth embodiment of the present invention will be described with reference to the drawings. Since the basic configuration of this embodiment is the same as that of the first embodiment, a characteristic configuration will be described below.

FIG. 24 is a flowchart showing the extraction process of additional information in the present embodiment. In the present embodiment, additional information is embedded in the printed matter, and the mobile terminal 301 captures the printed matter so that the information α (browsing information) and the user ID of the user who is permitted to browse the information α can be obtained. Acquire the included additional information. Some mobile terminals are equipped with two cameras, one is a camera mounted on the back side of the mobile terminal for taking a picture of a landscape, and the other is a camera mounted on the front side for taking a picture of the user himself / herself. In the present embodiment, the mobile terminal 301 acquires additional information by the camera on the back side, and acquires face authentication information by photographing the user's face with the camera on the front side. Hereinafter, the extraction process of the additional information of the present embodiment will be described with reference to the flowchart of FIG. 24. The process of FIG. 24 is realized, for example, by the CPU 304 reading the program stored in the ROM 305 into the RAM 306 and executing the program.

When the extraction process of the additional information is started, the CPU 304 takes a picture of the printed matter by the mobile terminal 301 in S2401 and acquires the additional information embedded in the printed matter. The additional information includes the information α and the user ID of the user who is permitted to view the information α. After that, the CPU 304 takes a picture of the user with the mobile terminal 301 in S2402 and acquires the face authentication information. In the present embodiment, it is assumed that the printed matter is photographed by the first camera (imaging sensor 202) and the user himself is photographed by the second camera (second imaging sensor 215). For the face recognition information, a user ID indicating a specific user is calculated using the face image acquired by the second imaging sensor 215. Many methods have been proposed for face recognition, but any method may be used as long as the user can be identified. Then, the CPU 304 determines in S2403 whether or not the user ID acquired by face authentication matches the user ID of the user who is permitted to view the information α acquired in S2401. If they match, the process proceeds to S2404, and if they do not match, the process ends. When shifting to S2404, the CPU 304 unlocks the information α and displays the information α on the display 308 of the mobile terminal 301 in S2402.

In the present embodiment, the lock on the information α is released by using face authentication to identify the user, but another method may be used to identify the user. For example, a mobile terminal having a fingerprint authentication function may identify a user by using fingerprint authentication, or a mobile terminal having a voice authentication function may identify a user by using voice authentication.

102 Additional information multiplexing device 103 Recording device 301 Mobile terminal 304 CPU
308 display

Claims (10)

The acquisition process of capturing an image in which additional information is embedded as a digital watermark and acquiring the imaging information,
An extraction step of extracting the additional information from the imaging information, and
It is an information processing method that has
An imaging information acquisition step for acquiring imaging status information regarding the status in the imaging, and
An information processing method further comprising a processing step of performing a predetermined process based on the additional information and the imaging status information. The additional information is a part of the URL and date information regarding a predetermined date.
The imaging status information is date and time information relating to the date and time when the printed matter was photographed.
In the processing step, another part of the URL is created and connected to the network depending on whether the date and time when the printed matter was photographed is before or after the predetermined date. The information processing method according to claim 1. The additional information is GPS information indicating the position of a predetermined place, and is
The imaging status information is GPS information indicating the position of the place where the printed matter was photographed.
The information processing method according to claim 1, wherein in the processing step, a route from a position where the printed matter is photographed to the position of the predetermined place is displayed. The additional information is price information indicating the price of the product, and is
The imaging status information is the country of residence information indicating the country of residence of the user.
The information processing method according to claim 1, wherein in the processing step, a price obtained by converting the price of the price information into the currency of the country of residence is displayed. The additional information is a predetermined URL and detailed information of the article, and is
The imaging status information is GPS information indicating the position of the user and terminal orientation information indicating the orientation of the mobile terminal.
The processing step is characterized in that the detailed information of the article viewed by the user is displayed from the layout information and the detailed information of the article at the predetermined URL, the position of the user, the orientation of the mobile terminal, and the like. Item 1. The information processing method according to item 1. The additional information is predetermined information and is
The imaging status information is touch information which is information about the user touching the display of the mobile terminal.
The information processing method according to claim 1, wherein in the processing step, the predetermined information is displayed on the display based on the touch information. The additional information includes browsing information and a first user ID of a user who is permitted to browse the browsing information.
The imaging status information is the second user ID of the user who photographed the printed matter.
The information processing apparatus according to claim 1, wherein in the processing step, the browsing information is displayed when the first user ID and the second user ID match. The information processing device according to claim 7, wherein the second user ID is obtained by face authentication. An acquisition means for acquiring imaging information by capturing an image in which additional information is embedded as a digital watermark,
An extraction means for extracting the additional information from the imaging information, and
It is an information processing method that has
An imaging information acquisition means for acquiring imaging status information regarding the situation in the imaging, and
An information processing device further comprising a processing means for performing a predetermined process based on the additional information and the imaging status information. A program for operating a computer as each step in the information processing method according to any one of claims 1 to 8.

Images