JP6909055B2 - Information processing equipment, control methods and programs - Google Patents

Description

The present invention relates to an information processing device, a control method, and a program for capturing an image in which additional information is multiplexed.

A digital watermarking technique that multiplexes other information related to an image with respect to the image information is known. According to the digital watermarking technology, for example, in image information such as a photograph or a painting, additional information such as the author's name and permission to use is multiplexed so as to be difficult to visually discriminate. In addition, as a technology in the field of application, with the aim of preventing forgery of banknotes, stamps, securities, etc. with the improvement of image quality of image output devices such as copiers and printers, output devices and output devices from images output on paper A technique for embedding additional information in an image is known in order to specify the aircraft number.

In addition, in order to extract additional information embedded in the image output on paper, an imaging device such as a scanner or a built-in camera of a mobile terminal is used to take a picture of the target output object and analyze the taken image. A technique for extracting embedded information is known. Here, when shooting the target output, an image shot at a normal position with respect to the embedded information is generally required, but in the case of the built-in camera of the mobile terminal, the shooting posture is not stable and the image is rotated. Or trapezoidal distortion may occur and information may not be retrieved.

In Patent Document 1, when reading additional information, a watermarked image with a correction marker added, which is a mark representing a reference point or a reference line of the image, is used, so that a high-level geometry with respect to a digital watermark is used. It is described that conversion resistance is realized by a simple process.

Japanese Patent No. 3888549

In the method using the correction marker described above, it is necessary to take a picture of the entire printed matter in order to read the correction marker, so that the printed matter must be pulled until it enters the angle of view. In that case, if the printed matter is too far away, the resolution will be insufficient to read the embedded additional information. On the other hand, if the photograph is taken close to the printed matter in order to increase the resolution, the correction marker cannot be read. Therefore, even if you try to perform keystone correction or rotation correction by detecting the edge of the printed matter and estimating the tilt and rotation angle, the edge will not fit in the angle of view if the distance to the printed matter is short, so the above estimation is performed. Can't.

An object of the present invention is to solve such a conventional problem. In view of the above points, the present invention provides an information processing device, a control method, and a program for notifying the proper posture of the device for photographing the subject when the image in which the additional information is multiplexed is photographed as the subject. The purpose is to provide.

In order to solve the above problems, the information processing device according to the present invention is an information processing device, which is a first notification means for notifying the subject to set the information processing device so as to have a reference posture. an imaging means for capturing an image additional information is multiplexed as an image of the object, and acquisition means for acquiring orientation information of the information processing apparatus, based on the orientation information acquired by the acquisition unit, wherein A determination means for determining whether or not the attitude of the information processing device with respect to the subject is appropriate for extracting the additional information from the image captured by the imaging means, and an attitude of the information processing device with respect to the subject. display but if it is determined not to be appropriate to extract the additional information from the image captured by the imaging means, before photographing by the imaging means, a screen for proper the posture display unit The information processing means includes a display control means for information processing, and the acquisition means acquires the attitude information after the notification by the first notification means .

According to the present invention, when an image in which additional information is multiplexed is photographed as a subject, it is possible to give a notification to make the posture of the device for photographing the subject appropriate.

It is a block diagram which shows the structure of an image processing system. It is a figure which shows the block structure of the image processing apparatus and the mobile terminal with a camera. It is a block diagram which shows the structure which performs the embedding included in the additional information multiplexing part. It is a flowchart which shows the whole processing including the quantization condition control part. It is a figure which shows the image which the number of horizontal pixels is WIDTH, and the number of vertical pixels is HEIGHT. It is a figure for demonstrating the quantization conditions A and B. It is a block diagram which shows the structure of the additional information separation part. It is a figure which shows the spatial filter used for the additional information separation part. It is a figure which shows the two-dimensional frequency domain. It is a flowchart which shows the process from the thinning part to the determination part of FIG. It is a figure for demonstrating the posture at the time of shooting of the mobile terminal with a camera. It is a figure which shows the case which the tilt and rotation are improper. It is a figure which shows the state which a printed matter and a mobile terminal with a camera face each other. It is a figure which shows the message displayed on the display. It is a figure which shows the case where the posture of the mobile terminal with a camera exceeds a predetermined permissible range. It is a figure which shows the guidance screen which is displayed on the display. It is a figure which shows the example of API. It is a flowchart which shows the display control processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the present invention according to the claims, and not all combinations of features described in the present embodiment are essential for the means for solving the present invention. .. The same components are given the same reference numbers, and the description thereof will be omitted.

FIG. 1 is a block diagram showing a configuration of an image processing system. The image processing system in the present embodiment includes an image processing device 100 and a mobile terminal 101 with a camera. The image processing device 100 is a printing device having a printing function, and is, for example, an MFP (multifunctional peripheral device). The camera-equipped mobile terminal 101 is, for example, a camera-equipped mobile phone, a camera-equipped smartphone, or a tablet PC.

In the present embodiment, the image processing device 100 prints the printed matter 104 in which the additional information is multiplexed. Then, the mobile terminal 101 with a camera images the printed matter 104 as a subject by the image sensor 107 (camera). Therefore, in the present embodiment, the image processing device 100 and the camera-equipped mobile terminal 101 may be able to communicate with each other via a wired network or a wireless network, or are connected to each other via a network or the like. It does not have to be.

The image processing device 100 includes an additional information multiplexing unit 105 that embeds additional information in the printed matter 104, and the camera-equipped mobile terminal 101 includes an additional information separating unit 108 that reads the multiplexed additional information from the printed matter 104. The additional information multiplexing unit 105 is realized as, for example, printer driver software or application software that creates image information to be output to the print unit 106 (printer engine). Further, it may be realized in a form of being built in as hardware or software in a copying machine, a facsimile, a printer main body or the like.

Further, the additional information separation unit 108 may be realized by, for example, internal application software or hardware that separates additional information from an image taken by a digital still camera. In the present embodiment, the additional information multiplexing unit 105 and the printing unit 106 will be described as being included in the image processing device 100, but the additional information multiplexing unit 105 and the printing unit 106 are separated into separate devices. Is also good. For example, an information processing device such as a PC or a smartphone may include an additional information multiplexing unit 105, and an image processing device different from the information processing device may include a printing unit 106. The information processing device may include both the additional information multiplexing unit 105 and the additional information separation unit 108.

[Configuration for embedding additional information]
Multi-gradation image information is input from the input terminal 102, and additional information to be embedded in the image information is input from the input terminal 103. This additional information includes information other than image information, such as audio information, video information, text information, copyright related to the image, shooting date and time, shooting location, various information of the photographer, or completely different image information. be. Further, the additional information may be included as a part of the image information configured as an image file.

The additional information multiplexing unit 105 embeds (multiplexes) additional information in the image information so that it is difficult to visually discriminate. The print unit 106 executes the print process based on the information created by the additional information multiplexing unit 105. The print unit 106 is, for example, an inkjet printer, a laser printer, or the like, which is a printer that realizes gradation expression on a printed matter 104 by using pseudo gradation processing.

FIG. 2A is a diagram showing a block configuration of the image processing device 100. The CPU 200 is a processor that comprehensively controls the inside of the image processing device 100. The additional information multiplexing unit 105 of FIG. 1 is realized by, for example, the CPU 200. The ROM 201 and HDD 203 store basic programs, control programs, various applications, data, and the like of the image processing device 100. For example, the CPU 200 realizes the operation of each embodiment by reading the program stored in the ROM 201 into the RAM 202 and executing the program. The RAM 202 is also used as a working memory of the CPU 200.

The network interface (NW I / F) 204 has a configuration according to the form of the network, for example, wired or wireless. It is also possible to support multiple types of wireless networks with different communication ranges. For example, it is possible to communicate with the camera-equipped mobile terminal 101 by short-range wireless communication (NFC: Near Field Communication) with a communication distance of several cm. Is. The display 205 displays each setting screen, preview screen, and the like to the user. The operation unit 206 has, for example, a keyboard, a touch panel, or the like, and can receive operation instructions from the user.

The device I / F 207 connects the print unit 106 (print engine) and the system bus 209. Although the print unit 106 is shown in FIG. 2A, other blocks such as a scanner and a fax machine may be connected to the device I / F 207 depending on the executable function of the image processing device 100. The image processing unit 208 executes image processing according to the intended use on the image data acquired from the outside. For example, the image processing unit 208 executes processing such as color space conversion, binarization processing, and image enlargement / reduction / rotation according to the recording method of the printing unit 106.

The blocks shown in FIG. 2A are communicably connected to each other via the system bus 209. A configuration other than the configuration shown in FIG. 2A may be used. For example, the system bus 209 and the image data bus are connected via a bus bridge. In that case, for example, the device I / F 207 is connected to the image data bus. The input terminal 102 of FIG. 1 shows, for example, a configuration input from the network I / F 204, and the input terminal 103 shows, for example, a configuration input from the network I / F 204 or the operation unit 206.

[Configuration for reading additional information]
The reading application executed on the camera-equipped mobile terminal 101 of FIG. 1 reads the information on the printed matter 104 using the image sensor 107, and separates the additional information embedded in the printed matter 104 by the additional information separating unit 108. , Output to output terminal 109. The output terminal 109 is an interface for outputting the acquired additional information. For example, if it is image information or character information, it is output to the display 110 (display unit), and if it is link information such as a URL, a browser is started on the display 110 to display the link destination. It may also be output to an interface that outputs data to an external device.

FIG. 2B is a diagram showing a block configuration of the camera-equipped mobile terminal 101. The camera-equipped mobile terminal 101 includes a configuration of a general-purpose information processing device such as a CPU, ROM, and RAM. The CPU 210 is a processor that comprehensively controls the inside of the camera-equipped mobile terminal 101. The additional information separation unit 108 of FIG. 1 is realized by, for example, the CPU 210. The ROM 211 stores basic programs, control programs, various applications, data, and the like of the camera-equipped mobile terminal 101. For example, the CPU 210 realizes the operation of each embodiment by reading the program stored in the ROM 211 into the RAM 212 and executing the program. The RAM 212 is also used as a working memory for the CPU 210.

The network I / F 213 has a configuration according to the form of the network, such as wired or wireless. It is also possible to support multiple types of wireless networks with different communication ranges. For example, it is possible to communicate with the image processing device 100 by short-range wireless communication (NFC: Near Field Communication) with a communication distance of several centimeters. be. The display 110 displays each setting screen, preview screen, and the like to the user. Further, as will be described later, the display 110 displays an image pickup result by the image pickup sensor 107. The operation unit 214 has, for example, a hard key or the like, and can receive an operation instruction from the user. The acceleration sensor 111 detects the gravitational acceleration applied to the camera-equipped mobile terminal 101, and the geomagnetic sensor 112 detects the orientation of the camera-equipped mobile terminal 101. The blocks shown in FIG. 2B are communicably connected to each other via the system bus 215.

[Embedding of additional information]
FIG. 3 is a block diagram showing a configuration for embedding included in the additional information multiplexing unit 105 of FIG. The error diffusion processing unit 300 converts the image information input to the input terminal 102 into a quantization level smaller than the number of input gradations by performing pseudo gradation processing using a general error diffusion method. The gradation value is expressed in terms of area by the quantization value of a plurality of pixels.

The blocking unit 301 divides the image information input to the input terminal 102 into predetermined area (block) units. The blocking performed by the blocking unit 301 may be divided into rectangles or areas other than rectangles. The quantization condition control unit 302 controls to change the quantization condition in units of regions blocked by the blocking unit 301 based on the additional information input at the input terminal 103. Further, the quantization condition control unit 302 changes the quantization condition in block units based on the additional information input to the input terminal 103.

Next, the entire process including the quantization condition control unit 302 will be described with reference to the flowchart of FIG. An example in which the quantization value is binary will be described. The process of FIG. 4 is realized, for example, by the CPU 200 of the image processing device 100 reading the program stored in the ROM 201 into the RAM 202 and executing the program.

In S401, the CPU 200 initializes the variable i reserved in the RAM 202. Here, the variable i is a variable that counts addresses in the vertical direction. In S402, the CPU 200 initializes the variable j reserved in the RAM 202. Here, the variable j is a variable that counts the addresses in the horizontal direction. Subsequently, in S403, the CPU 200 determines whether or not the coordinates (i, j), which are the current processing addresses, belong to the area where the multiplexing process should be executed.

The multiplexing region will be described with reference to FIG. FIG. 5 shows one image in which the number of horizontal pixels is WIDTH and the number of vertical pixels is HEIGHT. In this embodiment, an example in which additional information is multiplexed in this image will be described. Blocking is performed with horizontal N pixels and vertical M pixels with the upper left of the image as the origin. In the present embodiment, blocking is performed with the origin as a reference point, but a point away from the origin may be set as a reference point. In order to multiplex the maximum amount of information in this image, N × M blocks will be arranged from the reference point. Assuming that the number of blocks that can be arranged in the horizontal direction is W and the number of blocks that can be arranged in the vertical direction is H, W and H are calculated from the equations (1) and (2).

W = INT (WIDTH / N) ・ ・ ・ (1)
H = INT (HEIGHT / M) ・ ・ ・ (2)
However, INT () indicates the integer part in ().

The number of surplus pixels that cannot be divided in the equations (1) and (2) corresponds to the end portion when a plurality of N × M blocks are arranged, and in the present embodiment, that portion is outside the code multiplexing region.

In S403 of FIG. 4, when it is determined that the pixel of interest currently being processed is not in the multiplexing region, that is, outside the multiplexing region, the CPU 200 sets the quantization condition C in S404. On the other hand, when it is determined that the information is within the multiplexing area, the CPU 200 reads the additional information to be multiplexed in S405. Here, for ease of explanation, the additional information is represented by one bit each using an array called code []. For example, assuming that the additional information is information for 48 bits, 1 bit each from code [0] to code [47] is stored in the array code [].

In S405, the CPU 200 substitutes the information in the array code [] into the variable bit secured in the RAM 202, as in the equation (3).

bit = code [INT (i / M) x W + INT (j / N)] ... (3)
In S406, the CPU 200 determines whether or not the assigned variable bit is "1". As described above, since the information in the array code [] is stored by 1 bit each, the value of the variable bit also indicates either "0" or "1".

Here, if it is determined to be "0", the CPU 200 sets the quantization condition A in S407, and if it is determined to be "1", the CPU 200 sets the quantum condition A in S408. The conversion condition B is set.

Next, in S409, the CPU 200 performs the quantization process based on the quantization conditions set in S404, S407, and S408. This quantization process is performed by the error diffusion method.

In S410, the CPU 200 counts up the horizontal variable j, and in S411, determines whether or not the counted up variable j is less than WIDTH, which is the number of horizontal pixels of the image. Here, when it is determined that the variable j is less than WIDTH, the process from S403 is repeated. On the other hand, when it is determined that the variable j is not less than WIDTH, that is, when the horizontal processing is completed for the number of WIDTH pixels, the CPU 200 counts up the vertical variable i in S412. Then, in S413, the CPU 200 determines whether or not the counted-up variable i is less than HEIGHT, which is the number of vertical pixels of the image. Here, when it is determined that the variable i is less than HEIGHT, the process from S402 is repeated. On the other hand, when it is determined that the variable i is not less than HEIGHT, that is, when the processing in the vertical direction is completed for the number of HEIGHT pixels, the processing of FIG. 4 is terminated. By the above processing, the quantization condition is changed in block units consisting of N × M pixels.

Next, examples of quantization conditions A, B, and C will be described. There are various factors in the quantization condition in the error diffusion method, but in the present embodiment, the factor of the quantization condition is the quantization threshold value. Since the quantization condition C set in S404 is used outside the multiplexing region, the quantization threshold value may be any condition. When one pixel is a gradation expression with 8 bits and the quantization level is binary, the maximum value "255" and the minimum value "0" are typical quantization values, but in between. The value "128" is often set as the quantization threshold. Therefore, in the present embodiment, the quantization condition C is a condition in which the quantization threshold value is a fixed value of "128".

Since the quantization condition A set in S407 and the quantization condition B set in S408 are used in the blocks in the multiplexing region, it is necessary to cause a difference in image quality due to the difference in the quantization condition. However, it is necessary that the difference in image quality is expressed so as to be difficult to visually distinguish and can be easily identified from the paper.

6 (a) and 6 (b) are diagrams for explaining the quantization conditions A and B. FIG. 6A is a diagram showing the period of change of the quantization threshold value under the quantization condition A. In the figure, one cell is assumed to be one pixel, a white cell is a fixed threshold value, and a gray cell is a variable threshold value. That is, in the example of FIG. 6A, a matrix of 8 pixels in the horizontal direction and 4 pixels in the vertical direction is assembled, and a value that protrudes only for the gray square is set as the threshold value.

FIG. 6B is also a diagram showing the period of change of the quantization threshold under the quantization condition B. In the example of FIG. 6B, unlike FIG. 6A, a matrix of 4 horizontal pixels and 8 vertical pixels is assembled, and a value that protrudes only for the gray square is set as a threshold value.

As described above, when one pixel has an 8-bit gradation value, for example, a fixed threshold value of "128" and a prominent threshold value of "10" are set. When the quantization threshold is lowered, the quantization value of the pixel of interest tends to be "1" (quantization representative value "255"). That is, in both FIGS. 6A and 6B, the quantized value "1" can be easily arranged in the arrangement of the gray cells in the figure. In other words, for each block of N × M pixels, there are a block in which dots are generated in the arrangement of gray cells in FIG. 6 (a) and a block in which dots are generated in the arrangement of gray cells in FIG. 6 (b). It will be mixed.

A slight change in the quantization threshold in the error diffusion method does not have a great effect on the image quality. In the systematic dither method, the image quality of gradation expression is greatly affected by the dither pattern used. However, in the error diffusion method in which the quantization threshold is regularly changed as described above, the gradation expression that determines the image quality is the error diffusion method, so that the arrangement of dots may change slightly or the texture may change. Changes in generation have almost no effect on the image quality of gradation expression. Even if the quantization threshold changes, the error that is the difference between the signal value and the quantization value is diffused to the surrounding pixels, so that the input signal value is stored macroscopically. That is, it can be said that the redundancy is extremely large with respect to the arrangement of dots and the generation of texture in the error diffusion method.

As described above, in the present embodiment, multiplexing is realized by superimposing a predetermined periodicity representing a sign on the quantization threshold of the error diffusion method. However, multiplexing may be realized by another superimposition method. For example, multiplexing may be realized by a method of directly superimposing periodicity on RGB values (luminance information). Alternatively, multiplexing may be realized by a method in which the RGB value is separated into other color space information (for example, CIE L * a * b *, YCrCb signal) such as luminance-color difference information and the periodicity is superimposed. .. Alternatively, multiplexing may be realized by a method in which RGB values are separated into ink colors (for example, CMYK signals) and periodicity is superimposed.

[Reading process of additional information]
Next, the processing of the additional information separation unit 108 in the image processing system of FIG. 1 will be described. This process is a process of separating additional information from the captured image by using the notification for adjusting the posture of the camera-equipped mobile terminal 101, which will be described later. FIG. 7 is a block diagram showing the configuration of the additional information separation unit 108. In order to facilitate the explanation, as in the case of the additional information multiplexing unit 105 described above, an example of separating the additional information from the printed matter 104 in which the additional information of 1 bit each is multiplexed in the divided blocks. explain. As a matter of course, the amount of additional information per block in the additional information multiplexing unit 105 and the amount of separated information per block in the additional information separation unit 108 are equal.

The image information read by the camera-equipped mobile terminal 101 is input to the input terminal 700. Here, the resolution (imaging resolution) of the image pickup sensor 107 of the mobile terminal 101 with a camera is preferably equal to or higher than the print resolution at the time of creating the printed matter 104. Of course, in order to accurately read the dotted information of the dots of the printed matter 104, the imaging sensor 107 side needs to have a resolution more than twice that of the printer side according to the sampling theorem. However, if it is equal to or higher than that, it is possible to determine that the dots are scattered to some extent even if it is not accurate. In the present embodiment, the print resolution and the resolution of the image sensor 107 are assumed to be the same resolution for ease of explanation.

The blocking unit 701 blocks in units of horizontal P pixels and vertical Q pixels. Here, each block is smaller than the N × M pixels blocked when the digital watermark is superimposed. That is, the relationship of equation (4) holds.

P ≦ N and Q ≦ M ・ ・ ・ (4)
Further, the block formation in units of P × Q pixels is performed by skipping at regular intervals. That is, the block is performed so that one block in units of P × Q pixels is included in the area assumed to be a block composed of N × M pixels at the time of multiplexing.

Spatial filters 702 and 703 indicate spatial filters A and B having different characteristics, and filtering unit 704 indicates a digital filtering unit that calculates the sum of products with peripheral pixels. Each coefficient of this spatial filter is set corresponding to the period of the fluctuation threshold of the quantization condition at the time of multiplexing. Here, it is assumed that the additional information is multiplexed by changing the quantization conditions in the additional information multiplexing unit 105 using the two types of periodicity shown in FIGS. 6A and 6B. Examples of the spatial filter 702 and the spatial filter 703 used for the additional information separation unit 108 in that case are shown in FIGS. 8A and 8B. In FIGS. 8A and 8B, the central portion of the 5 × 5 pixel is the pixel of interest, and the other 24 pixels are the peripheral pixels. In FIGS. 8A and 8B, the pixels in the blank portion indicate that the filter coefficient is "0". As shown in FIGS. 8A and 8B, FIGS. 8A and 8B are edge enhancement filters. Moreover, the directionality of the emphasized edge and the directionality of the fluctuation threshold value when multiplexing are the same in FIGS. 8 (a) and 8 (b) and FIGS. 6 (a) and 6 (b). That is, the spatial filter is created so that FIG. 8 (a) matches FIG. 6 (a) and FIG. 8 (b) matches FIG. 6 (b).

Each of the thinning units 705 and 706 thins out the filtered signals (hereinafter referred to as conversion values) in the block composed of P × Q pixels based on a certain regularity. In the present embodiment, the thinning process is performed separately for each regularity of periodicity and phase. That is, the thinning units 705 and 706 have different thinning periodicities, and a plurality of thinning processes having different phases are executed in each of the thinning units 705 and 706. The thinning process will be described later.

The conversion value addition unit 707 adds the conversion values thinned out by the thinning units 705 and 706 for each phase. The thinning process and the addition process of the conversion values of the thinned pixels correspond to extracting the power of the predetermined frequency vector emphasized by the spatial filter.

The variance value calculation unit 708 calculates the variance value of a plurality of added values added for each phase in each periodicity. The determination unit 709 determines the multiplexed code based on the variance value in each periodicity.

FIG. 9 is a diagram showing a two-dimensional frequency domain. The horizontal axis shows the frequency in the horizontal direction, and the vertical axis shows the frequency in the vertical direction. The origin at the center shows the DC component, and it is shown that the frequency becomes higher as the distance from the origin increases. The circle in FIG. 9 shows the cutoff frequency due to error diffusion. The filter characteristic of the error diffusion method shows the characteristic of the HPF (high-pass filter) in which the low frequency region is cut off, and the cutoff frequency changes according to the density of the target image.

In the present embodiment, the frequency characteristic generated after quantization changes by changing the quantization threshold, but the change of the quantization threshold according to FIG. 6A produces a large power spectrum on the frequency vector A of FIG. Further, in the change of the quantization threshold value according to FIG. 6B, a large power spectrum is generated on the frequency vector B of FIG. At the time of additional information separation, the multiplexing signal is determined based on the detection of the frequency vector in which this large power spectrum is generated. In this embodiment, each frequency vector is individually emphasized and extracted.

8 (a) and 8 (b) correspond to an HPF having a specific frequency vector directionality. That is, the spatial filter of FIG. 8A can emphasize the frequency vector on the straight line A of FIG. 9, and the spatial filter of FIG. 8B can emphasize the frequency on the straight line B of FIG. It becomes possible to emphasize the vector. For example, it is assumed that a large power spectrum is generated on the frequency vector of the straight line A of FIG. 9 due to the change of the quantization condition as shown in FIG. 6A. At that time, the amount of change in the power spectrum is amplified by the spatial filter of FIG. 8 (a), but is hardly amplified by the spatial filter of FIG. 8 (b). That is, when a plurality of spatial filters are filtered in parallel, amplification is performed only when the spatial filters have the same frequency vector, and there is almost no amplification when filtering by other filters. Therefore, it is possible to easily determine on what frequency vector a large power spectrum is generated.

FIG. 10 is a flowchart showing the processing of the thinning units 705 and 706 of FIG. 7, the conversion value addition unit 707, the dispersion value calculation unit 708, and the determination unit 709. The process of FIG. 10 is realized, for example, by the CPU 210 of the camera-equipped mobile terminal 101 reading the program stored in the ROM 211 into the RAM 212 and executing the program.

In FIG. 10, S1001 and S1002 indicate the initialization of the variables, and the CPU 210 initializes the values of the variables i and j secured in the RAM 212 to 0. In S1003, the CPU 210 determines two factors of the regularity of thinning by the thinning units 705 and 706, that is, "periodicity" and "phase". In this flowchart, the variable related to periodicity is i and the variable related to phase is j. The conditions of the periodicity and the phase are managed by the number (number), and here, the periodicity number (hereinafter abbreviated as No.) is i, and the phase No. Set the factor of the thinning method in which is j.

In S1004, the CPU 210 adds the converted values thinned out in the block, and stores the added values as an array of variables TOTAL [i] [j]. In S1005, the CPU 210 counts up the variable j and compares it with the fixed value J in S1006. The fixed value J stores the number of times the thinning process is performed by changing the phase. Here, if the variable j is less than J, the process returns to S1003, and the new phase No. As a result, the thinning process and the addition process of the conversion values of the thinned pixels are repeated.

When the phase-shifted thinning process and the thinning pixel conversion value addition process are completed a set number of times, in S1007, the CPU 210 calculates the variance value of the addition result TOTAL [i] [j]. That is, it is evaluated how much each addition result varies depending on the phase difference. Here, i is fixed and the variance values of J TOTAL [i] [j] are obtained. Here, let the variance value be B [i].

In S1008, the CPU 210 counts up the variable i and compares it with the fixed value I in S1009. The fixed value I stores the number of times the thinning process is performed by changing the periodicity. Here, if the variable i is less than I, the process returns to S1002, and the new periodicity No. Using the condition of, the thinning process and the addition process of the converted values of the thinned pixels are repeated again.

In S1009, when the CPU 210 determines that i has completed the set number of times, it means that I can calculate the variance value B [i]. In S1010, the maximum value of the variance value is detected from the set of I variance values, and the value of i at that time is substituted into the variable imax. In S1011, the CPU 210 has a periodicity No. A code in which is imax is determined to be a multiplexed code. After that, the process of FIG. 10 is completed.

In this way, by associating the quantization condition, the spatial filter characteristic, and the periodicity of the thinning condition, multiplexing and separation can be easily realized. In this embodiment, the multiplexing code in the block is 1 bit. However, the multiplexing code may be more than one bit.

In the present embodiment, the multiplexing information (additional information) is embedded invisible as described above, and further, the image is captured and read, so that it is not necessary to arrange the code portion on the printed matter. Therefore, for example, it is possible to avoid the arrangement of the code portion which is considered unnecessary from the viewpoint of viewing a photograph. In addition, there is no need for a two-action user to take an image of the code portion of the printed matter and then take an image of the photo portion, or to take an image of the photo portion after the user operates which feature amount to use. That is, in the present embodiment, since it can be executed by one action of capturing only the photo portion of the printed matter, the man-hours for the user can be reduced and the convenience can be improved.

[Operation when reading]
The image information transmitted from the input terminal 700 may not be separated from the image information before the printer output because the printed matter 104 by the print unit 106 is photographed by the mobile terminal 101 with a camera. The reason is that the posture of the mobile terminal 101 with a camera does not face the printed matter 104 at the time of shooting, so that the image is geometrically distorted.

FIG. 11 is a diagram for explaining the posture of the mobile terminal 101 with a camera at the time of shooting. FIG. 11A shows the side of the camera-equipped mobile terminal 101 where the display 110 is located, and FIG. 11B shows the opposite side. As shown in FIG. 11B, the camera-equipped mobile terminal 101 is provided with a camera lens 1100. 11 (c) is a view of FIG. 11 (a) viewed from the X direction, and FIG. 11 (d) is a view of FIG. 11 (a) viewed from the Y direction.

There are three types of postures of the camera-equipped mobile terminal 101: yaw, pitch, and roll, as shown in FIGS. 11A, 11C, and 11D. However, even if an attempt is made to correct these effects by geometric transformation, it is not possible to correct them because it is unknown at what tilt and angle of rotation the image was taken.

FIG. 12 is a diagram showing a case where the tilt and rotation are improper. The camera-equipped mobile terminal 101 has a tilt angle as shown in FIG. 12A and a rotation angle as shown in FIG. 12B with respect to the printed matter 104. In this case, the printed matter 104 seen from the camera-equipped mobile terminal 101 is distorted as shown in FIG. 12 (c), but the end portion of the printed matter 104 cannot be seen in the image 1200 acquired by shooting. Therefore, it is not known that the distortion is occurring. However, since distortion actually occurs, it becomes difficult to separate the multiplexing from the captured image.

As a countermeasure, in the present embodiment, as shown in FIG. 13 (a), first, in order to superimpose the camera-equipped mobile terminal 101 on the printed matter 104 facing directly, the display 110 is as shown in FIG. 14 (a). Display instructions. That is, by displaying the message as shown in FIG. 14A, the user is urged to set the camera-equipped mobile terminal 101 at a position facing the printed matter 104. Here, when the OK button 1401 is pressed, the values of the acceleration sensor 111 and the geomagnetic sensor 112 are acquired, a rotation matrix is created from both values, and a matrix conversion is performed on the coordinate axes of the system to rotate, tilt, and tilt. It is calculated and stored in the RAM 212 as the attitude information in the reference state. A known technique may be used for these calculations, and for example, the system standard API of Android (registered trademark) may be used as shown in FIG.

When the OK button 1401 is pressed, the posture information is calculated and at the same time, an instruction as shown in FIG. 14B is displayed on the display 110. With the display, the user lifts the terminal and moves it to a desired position to take a picture of the printed matter 104. In the present embodiment, shooting is performed by pressing the shooting button 1402, but shooting of a moving image may be started from the time when the OK button 1401 is pressed, and still images may be sequentially cut out to perform multiplexing separation processing. ..

When the OK button 1401 is pressed, the posture information in the reference state is calculated, and then the posture information is periodically calculated and compared with the posture information in the reference state stored in the RAM 212, and the difference is ROM 211. Determine if it is within the specified tolerance recorded in. At this time, if the user can lift the printed matter 104 while being parallel to the printed matter 104 as shown in FIG. 13 (c), the display 110 should maintain the posture as shown in FIG. 14 (c). Notice. However, as shown in FIG. 15, when the user lifts the terminal so that the difference between the current posture information and the posture information in the reference state exceeds the specified allowable range, the screen as shown in FIG. 16 is displayed. For example, as shown in FIG. 16A, a guidance message for returning a value exceeding a specified allowable range is notified. Alternatively, as shown in FIG. 16B, the state of the current posture is graphically notified.

FIG. 16B shows a display in which the spirit level and the compass are combined, and if the bubble-like symbol 1602 is in the center, the angle between tilt and inclination is the same as the reference state. Since the dotted arrow 1603 points in the direction when the OK button 1401 is pressed, if the dotted arrow 1603 overlaps with the solid arrow 1604, it indicates that the rotation angle is the same as the reference state.

FIG. 18 is a flowchart showing the procedure of the display control process in the present embodiment. The process of FIG. 18 is realized, for example, by the CPU 210 of the camera-equipped mobile terminal 101 reading the program stored in the ROM 211 into the RAM 212 and executing the program. The process of FIG. 18 is started when the function of the image sensor 107 of the mobile terminal 101 with a camera is executed, such as a camera shooting by a user. Alternatively, it may be executed when the user operates the mobile terminal 101 with a camera to start the application and inputs an instruction to read the multiplexing information. In this case, the CPU 210 executes the process described later by reading the application including the additional information separation unit 108 from the memory and executing the application.

In S1801, the CPU 210 displays the message shown in FIG. 14A on the display 110. As shown in FIG. 14A, the message 1400 "Please place the terminal facing the printed matter and press" OK "" and the OK button 1401 are displayed on the display 110. In S1802, the CPU 210 determines whether or not the OK button 1401 has been pressed. Here, if it is determined that the button has been pressed, the process proceeds to S1803, and if it is determined that the button has not been pressed, the process of S1802 is repeated.

In S1803, the CPU 210 acquires the attitude information of the camera-equipped mobile terminal 101 by the acceleration sensor 111 and the geomagnetic sensor 112. The CPU 210 stores the acquired posture information in a storage area such as the RAM 212 as the posture information in the reference state. That is, the posture information of the mobile terminal 101 with a camera when the user is conscious of facing the user is acquired as the posture information in the reference state.

In S1804, the CPU 210 displays the message shown in FIG. 14B on the display 110. As shown in FIG. 14B, the message 1403 "Please lift the terminal straight" and the shooting button 1402 are displayed on the display 110.

In S1805, the CPU 210 determines whether or not a predetermined time has elapsed since the attitude information in the reference state was acquired in S1803. Here, if it is determined that the predetermined time has elapsed, the process proceeds to S1806, and if it is determined that the predetermined time has not elapsed, the process of S1805 is repeated. In S1806, the CPU 210 acquires the posture information of the mobile terminal 101 with a camera at the present time by the acceleration sensor 111 and the geomagnetic sensor 112. The predetermined time here is a time interval during which the user can detect that he / she is no longer facing the camera-equipped mobile terminal 101 while he / she is lifting the camera-equipped mobile terminal 101 straight, for example, 0.05 seconds. .. That is, the attitude information is acquired from the acceleration sensor 111 and the geomagnetic sensor 112 in S1805 at predetermined time intervals.

In S1807, the CPU 210 determines whether or not the posture information acquired in S1805 is within the specified allowable range from the posture information in the reference state acquired in S1803. In other words, the determination process of S1807 corresponds to a process of determining whether or not it is appropriate to extract additional information from the image captured by the imaging means. Here, when it is determined that the range is within the specified allowable range, in S1808, the CPU 210 executes a display control process for displaying the message 1404 shown in FIG. 14C on the display 110. As shown in FIG. 14B, the message 1404 "Please maintain that posture" and the shooting button 1402 are displayed on the display 110. On the other hand, if it is determined that it is not within the specified allowable range, S1811 is executed. That is, the CPU 210 executes a display control process for displaying the message 1601 shown in FIG. 16A or the spirit level and compass shown in FIG. 16B on the display 110.

After displaying the message in S1808 or S1811, in S1809, the CPU 210 determines whether or not the shooting button 1402 is pressed. Here, when it is determined that the shooting button 1402 is pressed, in S1810, the CPU 210 takes a picture of the printed matter by the image sensor 107. By this shooting process, the mobile terminal 101 with a camera can be embedded in the printed matter to acquire additional information and execute a display process based on the additional information. For example, the camera-equipped mobile terminal 101 can display a printed matter being photographed while superimposing a message or the like based on additional information. In addition, the camera-equipped mobile terminal 101 can display a screen corresponding to the URL acquired from the additional information. After that, the process of FIG. 18 is completed. On the other hand, when it is determined that the shooting button 1402 is not pressed, the process from S1805 is repeated.

As described above, according to the present embodiment, before shooting, a notification is given using the difference between the proper imaging posture and the current imaging posture when the digital watermark is read by the built-in camera of the mobile terminal with a camera. It can be corrected to an imaging posture suitable for reading.

(Other Examples)
In the above-described embodiment, the message display process of S1808 or S1811 is executed using the execution result of the determination process of S1807 executed before S1809. That is, the message display process is executed before shooting, but the message display process may be executed at other timings. For example, when shooting is performing a plurality of shooting processes such as moving image shooting, a determination process such as S1807 may be executed after the start of shooting, and a message display process may be executed after the start of shooting.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image processing device: 101 Mobile terminal with camera: 200, 210 CPU: 201, 211 ROM: 202, 212 RAM

Claims (10)

It is an information processing device
A first notification means for notifying the subject to set the information processing device so as to have a reference posture.
Imaging means for taking an image additional information is multiplexed as an image of the object,
An acquisition means for acquiring the posture information of the information processing device, and
Based on the posture information acquired by the acquisition means, whether or not the posture of the information processing device with respect to the subject is appropriate for extracting the additional information from the image captured by the imaging means. Judgment means for judgment and
Attitude to the subject of the information processing apparatus, when it is determined not to be appropriate to extract the additional information from the image captured by the imaging means, before photographing by the imaging means, the position on the display unit Display control means to display the screen to make it appropriate,
With
The acquisition means acquires the posture information after the notification by the first notification means.
An information processing device characterized by this. Before After the acquisition of the pre Kisugata posture information by Quito resulting unit, according to claim 1, further comprising a second notifying means, for notifying to move the information processing apparatus to a position for photographing Information processing equipment. Before Quito resulting means, after acquisition of standards of orientation information, at predetermined time intervals, before the information processing apparatus according to claim 1 or 2, characterized in that to obtain the Kisugata biased information. When the determination means determines that the posture of the information processing device with respect to the subject is appropriate for extracting the additional information from the image captured by the imaging means, the determination means notifies that the posture is maintained. The information processing apparatus according to any one of claims 1 to 3, further comprising a third notification means. Before Kisugata posture information, the angle of rotation of the information processing apparatus, the angle of tilt, the information processing apparatus according to any one of claims 1 to 4, characterized in that it comprises one of the angle of the inclined. The information processing device according to any one of claims 1 to 5 , wherein the display control means causes the display unit to display a guidance message for optimizing the posture of the information processing device. Any of claims 1 to 6 , wherein the display control means causes the display unit to display a screen including either a spirit level or a compass as a screen for optimizing the posture of the information processing device. The information processing device according to item 1. The information processing device according to any one of claims 1 to 7 , wherein the display control means further displays a reception means for receiving a shooting instruction by the imaging means on the display unit. A control method executed in an information processing device.
A notification process for notifying the subject to set the information processing device so as to have a reference posture.
An imaging step of capturing an image of the additional information is multiplexed as an image of the object,
The acquisition process for acquiring the attitude information of the information processing device, and
Based on the posture information acquired in the acquisition step, whether or not the posture of the information processing device with respect to the subject is appropriate for extracting the additional information from the image captured in the imaging step. Judgment process and
Attitude to the subject of the information processing apparatus, when it is determined not to be appropriate to extract the additional information from the image captured in the imaging step, before photographing in the imaging step, the orientation on the display unit Display control process to display the screen to make the information processing and
Have,
In the acquisition step, after the notification in the notification step, the posture information is acquired.
A control method characterized by that. The computer program for functioning as a means of the information processing apparatus according to any one of claims 1 to 8.

Images