JP6406758B2 - Imaging apparatus, digital watermark extraction method, digital watermark and coding aperture optimization method - Google Patents

Description

  The present invention relates to an imaging apparatus, a digital watermark extraction method, a digital watermark, and a coding aperture optimization method.

  Two-dimensional codes such as boarding passes for airplanes or trains, admission tickets for events, electronic money, coupons, etc. are expected to play a role of authentication. A technique that can identify an illegally copied two-dimensional code by embedding a digital watermark in a two-dimensional code printed on paper has been disclosed (for example, see Patent Documents 1 and 2).

  On the other hand, realization of a technique for more accurately detecting a copy of a two-dimensional code displayed on a mobile phone screen is desired.

Japanese Patent No. 4713691 Japanese Patent No. 4742175

  However, when extracting a digital watermark embedded in a two-dimensional code, if image data captured due to defocusing is blurred, it is difficult to accurately extract a digital watermark that is a high-frequency component from the image data.

  The present invention has been made in view of the above circumstances, and an imaging apparatus, a digital watermark extraction method, a digital watermark, and a coding aperture optimization method that can accurately extract a digital watermark even when defocusing occurs. The purpose is to provide.

In order to achieve the above object, an imaging apparatus according to the first aspect of the present invention provides:
An image sensor that captures a two-dimensional image formed on a light receiving surface and acquires image data of the two-dimensional image;
An imaging optical system that forms an image on a light receiving surface of the image sensor with a two-dimensional image embedded with a digital watermark;
An aperture stop having a coded aperture disposed on an optical path of light incident on the image sensor via the imaging optical system;
An image processing unit that performs image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired by the imaging device, and extracts the digital watermark from the image data;
Is provided.

The coded aperture includes
Of a plurality of images decomposed from the two-dimensional image by frequency analysis, a code pattern having a spatial frequency component is formed in a direction corresponding to the spatial frequency component of the image in which the digital watermark is embedded.
It is good as well.

The image processing unit
Performing deconvolution using a blur function corresponding to the coded aperture to restore the image data of the two-dimensional image;
Extracting the digital watermark by performing frequency conversion processing on the restored image data of the two-dimensional image;
It is good as well.

The image processing unit
Performing deconvolution using a function in which a blur function corresponding to the coded aperture and a function for extracting the digital watermark are combined to extract the digital watermark;
It is good as well.

With a liquid crystal drive
The aperture stop is
It is a transmissive liquid crystal opening that forms the coded opening by being driven by the liquid crystal driving unit.
It is good as well.

The liquid crystal driving unit
Driving the liquid crystal aperture so that a first aperture for extracting the two-dimensional image and a second aperture as the encoding aperture for extracting the digital watermark are alternately formed;
The image processing unit
Performing image processing on the first image data acquired based on the light incident on the image sensor through the first aperture to extract the two-dimensional image;
Performing the image processing on the second image data acquired based on the light incident on the image sensor through the second aperture to extract the digital watermark;
It is good as well.

The aperture stop is
A first opening for extracting the two-dimensional image, and a second opening as the encoded opening for extracting the digital watermark;
The imaging optical system is
A light splitting unit configured to separately form a first image based on the light passing through the first aperture and a second image based on the light passing through the second aperture on the image sensor; And
The image processing unit
Performing image processing on the image data of the first image to extract the two-dimensional image;
Performing image processing on the image data of the second image to extract the digital watermark;
It is good as well.

The two-dimensional image is
It is a two-dimensional code displayed on the image of the information terminal.
It is good as well.

The aperture stop is disposed at a pupil position of the imaging optical system.
It is good as well.

A digital watermark extraction method according to a second aspect of the present invention includes:
An imaging step of acquiring image data by imaging image data of a two-dimensional image in which a digital watermark is embedded through an imaging optical system and an aperture stop having a coded aperture; and
An image processing step of performing image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired in the imaging step, and extracting the digital watermark from the image data;
including.

A digital watermark and coding aperture optimization method according to a third aspect of the present invention includes:
An imaging step of acquiring image data by imaging image data of a two-dimensional image in which a digital watermark is embedded through an imaging optical system and an aperture stop having a coded aperture; and
An image processing step of performing image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired in the imaging step, and extracting the digital watermark from the image data;
A calculation step of calculating an extraction rate of the digital watermark extracted in the image processing step;
Including
While changing the digital watermark pattern and the encoded aperture pattern, the imaging step, the image processing step, and the calculation step are repeated, and the extraction rate is maximized by using an optimization method. The watermark and the coded aperture are determined.

  According to the present invention, the image data of the two-dimensional image in which the digital watermark is embedded is imaged by the imaging element through the aperture stop having the encoded aperture. The image data obtained through the coded aperture is an image in which the blur function corresponding to the coded aperture and the two-dimensional image are convolved. This image is formed on the image sensor in a state where high spatial frequency components are not lost due to the coded aperture. For this reason, even if the image data of the two-dimensional code is blurred due to defocus, if image processing including deconvolution is performed on the image data with a blur function corresponding to the encoded aperture corresponding to the defocus, Image data of a two-dimensional code can be obtained without losing high spatial frequency components. As a result, the digital watermark can be accurately extracted even when the focus is deviated. In addition, it is possible to realize a digital watermark that can be decoded only by a device having a coding aperture.

It is a block diagram which shows the structure of the optical system of the imaging device which concerns on Embodiment 1 of this invention. FIG. 2A is an example of a cover image that is a two-dimensional image displayed on the screen of the information terminal. FIG. 2B is an example of a digital watermark embedded in a two-dimensional code. FIG. 2C is an example of a watermarked two-dimensional image in which a digital watermark is embedded. It is a figure which shows an example of an encoding opening. It is a block diagram which shows the hardware constitutions of the image process part of FIG. It is a block diagram which shows the software structure of the image process part of FIG. 2 is a flowchart illustrating an operation of the imaging apparatus in FIG. 1. FIG. 7A is an example of image data (part 1) of a captured two-dimensional code. FIG. 7B is digital watermark image data (part 1) extracted from the image data of FIG. FIG. 8A is an example of image data (part 2) of a captured two-dimensional code. FIG. 8B is digital watermark image data (part 2) extracted from the image data of FIG. FIG. 9A is an example of image data of a two-dimensional code imaged using a circular aperture stop. FIG. 9B is digital watermark image data extracted from the image data of FIG. FIG. 10A shows another example of the coded aperture. FIG. 10B is another example of a two-dimensional image. FIG. 10C is an example of a digital watermark embedded in a two-dimensional image. It is a schematic diagram which shows the structure of the design system of the digital watermark and coding opening based on Embodiment 2 of this invention. It is a block diagram which shows the structure of the design system of FIG. It is a flowchart which shows a part of operation | movement of the design system of FIG. It is a flowchart which shows the specific process of step S12 of FIG. FIG. 15A is a diagram illustrating a detection result of a check pattern encoding aperture and a digital watermark. FIG. 15B is a diagram illustrating a detection result of a circular opening and a digital watermark. FIG. 15C is a diagram illustrating the detection result of the coding aperture (Zhou code) and the digital watermark. FIG. 15D is a diagram showing the detection result of another coding aperture (Zhou code) and digital watermark. FIG. 16A shows a circular opening. FIG. 16B is a diagram showing a coded aperture. FIG. 16C is a diagram illustrating a first modification of the configuration of the imaging device. FIG. 17A is a diagram showing an aperture stop in which a circular aperture and a coded aperture are formed. FIG. 17B is a diagram illustrating a second modification of the configuration of the imaging device.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Embodiment 1 FIG.
First, a first embodiment of the present invention will be described.

  The imaging apparatus 100 includes a lens 1A, an aperture stop 2, a lens 1B, a drive unit 3, an imaging element 4, and an image processing unit 5. The information terminal 200 has a screen 10. The imaging apparatus 100 targets a two-dimensional image displayed on the screen 10 of the information terminal 200 as an imaging target. The imaging device 100 is, for example, a digital camera, a mobile phone, a smartphone, a tablet computer, a mobile computer, or the like. The information terminal 200 is a mobile phone, a smartphone, a tablet computer, a mobile computer, a barcode reader (installed type, handheld type), or the like.

  The two-dimensional image displayed on the screen 10 of the information terminal 200 uses, for example, a two-dimensional code (QR (Quick Response) code (registered trademark)) 11 as shown in FIG. 2A as an original image (cover image). . The QR code (registered trademark) is a matrix type two-dimensional code. The two-dimensional code (QR code (registered trademark)) 11 has information vertically and horizontally, and is characterized by square cutout symbols (position detection pattern, finder pattern) at three corners. In addition, a small square alignment pattern is fixed. And a code | symbol is recorded on a part other than that.

  A digital watermark is embedded in the two-dimensional code 11. As shown in FIG. 2B, the digital watermark 12 is also a two-dimensional code. The digital watermark technique is a kind of information hiding (data hiding) technique for embedding information in digital contents such as images and music. By embedding the digital watermark 12 shown in FIG. 2B in the two-dimensional code 11 shown in FIG. 2A, the entire two-dimensional image 13 shown in FIG. 2C is formed.

  The digital watermark 12 is embedded as follows. For example, the two-dimensional code 11 shown in FIG. 2A is separated into RGB color components, for example. Discrete wavelet transform is performed on the image data of each color component, and each image data has LL component (low frequency component), LH component (low frequency component in the vertical direction, high frequency component in the horizontal direction), HL component ( The vertical direction is decomposed into frequency components such as a high frequency component, the horizontal direction is a low frequency component, and an HH component (diagonal direction is a high frequency component). Then, watermark information is embedded in at least one of the components other than the LL component, the HH component, the LH component, and the HL component of the wavelet coefficient value (coefficient value), and each component is subjected to inverse discrete wavelet transform to reconstruct an image. As a result, a two-dimensional image 13 shown in FIG. 2C is generated.

  As described above, the two-dimensional code 11 of FIG. 2A is frequency-converted, and the digital watermark 12 component shown in FIG. 2B is embedded in the high spatial frequency component, and then the inverse transformation is performed to reconstruct the image. Then, a two-dimensional image 13 in which the digital watermark 12 shown in FIG. 2C is embedded is generated. In addition to the above-described discrete wavelet transform, various transform methods such as discrete cosine transform or Fourier transform can be applied to the frequency transform.

  Returning to FIG. 1, the lenses 1A and 1B are convex lenses. The imaging optical system 1 is configured by the lenses 1A and 1B. The imaging optical system 1 forms an image of the two-dimensional image 13 in which the digital watermark 12 is embedded on the light receiving surface of the image sensor 4. The configuration of the imaging optical system 1 is not limited to that shown in FIG. It may be an imaging optical system configured by arranging three or more lenses along the optical axis AX.

  The aperture stop 2 is disposed on the optical path of light incident on the image sensor 4 via the imaging optical system 1, such as the pupil position of the imaging optical system 1 (lenses 1A, 1B). The aperture stop 2 has a coded aperture 20 as shown in FIG. As shown in FIG. 3, square openings are arranged in a mosaic pattern in the coded opening 20 of the aperture stop 2. A coded opening pattern is formed in the coded opening 20. The encoded aperture pattern includes a wide band of spatial frequency components in a plurality of directions.

  Here, an in-plane position of the coded aperture 20 of the aperture stop 2 (a position through which light passes) is set as (X, Y). If the aperture shape of the aperture stop 2 is circular, the light incident on each pixel of the image sensor 4 is light in which the information regarding the position information (X, Y) in the aperture of the incident light is lost. It becomes. On the other hand, if the aperture pattern of the aperture stop 2 is an encoded aperture 20 that is encoded, the light incident on each pixel of the image sensor 4 has positional information (X, Y) will remain.

  The aperture stop 2 can be made of, for example, an OHP sheet or a metal plate, but a transmissive liquid crystal aperture can be used. In this embodiment, the drive unit 3 controls the transmissivity of a desired pixel in the liquid crystal opening (increasing the transmissivity of a portion that transmits light and decreasing the transmissivity of a portion that does not transmit light). Thus, the encoding opening 20 including a desired code pattern can be formed on the liquid crystal opening.

  The image sensor 4 captures two-dimensional image data formed on the light receiving surface. The image sensor 4 is, for example, a CCD (Charge Coupled Device). In the image sensor 4, a plurality of pixels are spread on the light receiving surface. In each pixel, a charge corresponding to light incident during an exposure time during photographing is accumulated, and two-dimensional digital image data is generated by the charge accumulated in each pixel.

  The image of the two-dimensional image 13 displayed on the screen 10 is imaged on the light receiving surface of the image sensor 4 by the imaging optical system 1 (lenses 1A, 1B). At that time, the light incident on the image sensor 4 passes through the encoding aperture 20. As a result, the image of the two-dimensional image 13 formed on the image sensor 4 is a convolution of the two-dimensional image 13 displayed on the screen 10 and the blur function corresponding to the encoded aperture 20. This blur function varies depending on the shift in the optical axis direction between the image formation position of the image data of the two-dimensional code and the light receiving surface of the image sensor 4, that is, the focus shift. This blur function is also called a kernel. This blur function corresponds to the optical transfer function (OTF) in Fourier optics.

  Therefore, even if the position of the image sensor 4 is out of focus with respect to the image formation position of the two-dimensional image 13, the image corresponds to the encoded aperture 20, and a blur function corresponding to the amount of focus deviation is expressed. The original image data can be restored by image processing including the used deconvolution.

  The image processing unit 5 performs image processing on the image data acquired by the image sensor 4. The image processing unit 5 is a computer. As shown in FIG. 4 showing the hardware configuration of the image processing unit 5 in FIG. 1, the image processing unit 5 includes a control unit 21, a main storage unit 22, an external storage unit 23, and an input / output unit 25. The main storage unit 22, the external storage unit 23, and the input / output unit 25 are all connected to the control unit 21 via the internal bus 28.

  The control unit 21 includes a CPU (Central Processing Unit) and the like. The CPU executes the program 29 stored in the external storage unit 23 and stored in the main storage unit 22, thereby realizing each component of the image processing unit 5 illustrated in FIG. 1.

  The main storage unit 22 includes a RAM (Random-Access Memory) or the like. The main storage unit 22 is loaded with a program 29 stored in the external storage unit 23. In addition, the main storage unit 22 is used as a work area (temporary data storage area) of the control unit 21.

  The external storage unit 23 includes a nonvolatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc Random-Access Memory), and a DVD-RW (Digital Versatile Disc ReWritable). In the external storage unit 23, a program 29 to be executed by the control unit 21 is stored in advance. Further, the external storage unit 23 supplies data used when executing the program 29 to the control unit 21 in accordance with an instruction from the control unit 21, and stores the data supplied from the control unit 21.

  The input / output unit 25 inputs image data from the image sensor 4. On the other hand, the input / output unit 25 outputs a calculation result.

  The various components of the image processing unit 5 shown in FIG. 1 are executed by the program 29 shown in FIG. 4 using the control unit 21, the main storage unit 22, the external storage unit 23, the input / output unit 25, and the like as hardware resources. To demonstrate its function.

  As shown in FIG. 5 (software configuration of the image processing unit 5 in FIG. 1), the image processing unit 5 includes a deconvolution unit 30 and an extraction unit 31.

  The deconvolution unit 30 performs, for example, deconvolution (deconvolution operation) using a blur function corresponding to the encoding aperture 20 on the image data acquired by the image sensor 4 to generate the two-dimensional image 13. Obtain image data. In the deconvolution unit 30, for example, various defocus amounts and blur functions corresponding to the defocus amounts are stored in association with each other. The deconvolution unit 30 performs deconvolution for each of the plurality of blur functions. Then, the deconvolution unit 30 performs matching with the original two-dimensional image 13 among the plurality of images obtained as a result of the deconvolution, and obtains the image having the highest degree of coincidence as the image data obtained as a result of the deconvolution. Output as.

  The extraction unit 31 extracts a digital watermark from the image data of the two-dimensional code obtained by the deconvolution unit 30. Specifically, after the image data is separated into RGB color components, the discrete wavelet transform is applied to the R, G, and B color-specific images, and the HH component, LH component, At least one of the HL components is extracted, and a digital watermark is extracted by performing inverse transformation on at least one of the extracted HH component, LH component, and HL component.

  Next, the operation of the imaging apparatus 100 according to this embodiment will be described.

  As shown in FIG. 6, the imaging apparatus 100 uses the encoded aperture 20 as the aperture stop 2 to convert the image data of the two-dimensional image 13 in which the digital watermark 12 is embedded via the imaging optical system 1 into the imaging device 4. (Step S1). Thereby, two-dimensional image data as shown in FIG. 7A is obtained.

  Subsequently, the deconvolution unit 30 performs deconvolution on the image data captured by the image sensor 4 using a blur function corresponding to the encoded aperture 20 to obtain image data of the two-dimensional image 13 ( Step S2).

  Subsequently, the extraction unit 31 extracts a digital watermark from the image data of the two-dimensional image 13 obtained by the deconvolution unit 30 (step S3). Specifically, the extraction unit 31 performs frequency conversion on the image data, extracts a high spatial frequency component in which the digital watermark is embedded, and extracts the digital watermark by performing inverse conversion. FIG. 7B shows an example of the extracted digital watermark.

  By using the coded aperture 20 as the aperture stop 2, even when the two-dimensional image 13 is tilted as shown in FIG. 8A, a digital watermark is extracted as shown in FIG. 8B. be able to. On the other hand, when a circular aperture is used for the aperture pattern of the aperture stop 2, the digital watermark extracted for the two-dimensional image shown in FIG. 9A is as shown in FIG. The image is significantly different from the original digital watermark 12 (see FIG. 2B).

  Moreover, the opening pattern of the encoding opening 20, the cover image, and the like are not limited to those described above. For example, a pattern can be adopted as shown in FIG. Further, an image as shown in FIG. 10B can be adopted as the cover image. Even when such an opening pattern of the encoded opening 20 and a background image are employed, it is possible to embed and extract a digital watermark as shown in FIG.

  As described above, if the encoded aperture 20 is used as the aperture stop 2 to accurately extract the digital watermark, it is possible to prevent the high spatial frequency component of the two-dimensional image in which the digital watermark is embedded from being damaged. Therefore, if image processing is performed on the obtained image data, a digital watermark can be accurately extracted.

  In the image processing unit 5, deconvolution and digital watermark extraction are performed separately. However, the blur function corresponding to the encoded aperture and the function for extracting the digital watermark (high spatial frequency in which the digital watermark is embedded) The digital watermark may be extracted by executing deconvolution using a function synthesized with the function for extracting the component.

  As described in detail above, according to this embodiment, image data of the two-dimensional image 13 in which the digital watermark 12 is embedded through the aperture stop 2 having the encoded aperture 20 is obtained. The image data obtained through the encoded aperture 20 is an image obtained by convolution of the blur function corresponding to the encoded aperture 20 and the image of the two-dimensional image. This image data is deconvolved using, for example, a blur function corresponding to the encoded aperture 20. If the image data is deconvolved with a blur function corresponding to the defocus, the image data of a two-dimensional image can be obtained without losing the high spatial frequency component of the digital watermark 12. As a result, even if the image of the two-dimensional image data is blurred due to defocusing, the digital watermark 12 can be accurately extracted. Thereby, for example, the digital watermark 12 can be accurately extracted even in a state where the image quality is deteriorated by shooting without matching the focus. In addition, it is possible to realize a digital watermark that can be decoded only by a device having a coding aperture.

  Further, when a 2D image displayed on the information terminal 200 is captured by another information terminal and displayed to reproduce the 2D image, image processing such as deconvolution is not performed on the obtained image data. Even when the image data is displayed on the screen, the image often becomes an image in which high spatial frequency components are lost, and it is difficult to accurately extract a digital watermark from the image. That is, according to this embodiment, by adopting convolution and deconvolution by the encoding aperture 20, it is possible to detect a copy of a two-dimensional image in which a digital watermark is embedded.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described. In the above embodiment, the coded aperture 20 is used as the aperture stop 2 for capturing a two-dimensional image in which a digital watermark is embedded. Also in this embodiment, the coded aperture 20 is used as the aperture stop 2 for capturing a two-dimensional image in which a digital watermark is embedded. A method of designing a digital watermark pattern and an aperture pattern of the coding aperture 20 that optimizes the digital watermark extraction rate will be described.

  In FIG. 11, a design system 500 for a watermark and encoding aperture 20 is shown. As shown in FIG. 11, the design system 500 includes a computer 300 in addition to the imaging device 100 and the information terminal 200. The computer 300 includes a CPU, a memory, and other hardware, and the CPU realizes its functions by executing a program stored in the memory.

  The configurations and operations of the imaging apparatus 100 and the information terminal 200 are the same as those in the first embodiment. The computer 300 can exchange data with the imaging device 100 and the information terminal 200 by wireless communication or wired communication. The information terminal 200 is mounted on the electric stage 201, and the distance between the imaging device 100 and the information terminal 200 can be adjusted. If the distance between the imaging device 100 and the information terminal 200 can be adjusted, the electric stage 201 is not necessary, but by using the electric stage 201, it is possible to efficiently design a robust watermark and coding aperture due to defocusing.

  As illustrated in FIG. 12, the computer 300 includes an extraction rate calculation unit 32, an optimization unit 33, a coded aperture generation unit 34, and an image generation unit 35.

  The extraction rate calculation unit 32 inputs the digital watermark image data extracted by the extraction unit 31 from the image processing unit 5 of the imaging apparatus 100. The extraction rate calculation unit 32 performs matching between the input digital watermark image data and the digital watermark reference image data, and uses the ratio of the number of pixels having the same luminance value to the total number of pixels as the digital watermark extraction rate. calculate.

  Based on the extraction rate calculated by the extraction rate calculation unit 32, the optimization unit 33 optimizes the aperture pattern of the encoded aperture 20 and the digital watermark pattern using an optimization method. In this embodiment, a genetic algorithm is adopted as an optimization method.

  The optimization unit 33 displays a two-dimensional image in which the digital watermark is embedded on the screen 10 of the information terminal 200 while changing the aperture pattern of the encoded aperture 20 and the digital watermark pattern on the screen 10 of the information terminal 200. Then, imaging using the aperture stop 2 of the encoding aperture 20 in the imaging device 100 is performed. The optimization unit 33 searches for the digital watermark pattern and the aperture pattern of the encoding aperture 20 that maximize the extraction rate using the optimization method while changing the digital watermark pattern and the aperture pattern of the encoding aperture 20. To do.

  The encoded aperture generation unit 34 generates an aperture pattern of the encoded aperture 20 in accordance with an instruction from the optimization unit 33. The encoded aperture generation unit 34 transmits the generated aperture pattern of the encoded aperture 20 to the drive unit 3. The drive unit 3 drives the liquid crystal aperture of the aperture stop 2 based on the received aperture pattern, and forms the encoded aperture 20 corresponding to the aperture pattern.

  The image generation unit 35 generates two-dimensional image data in which a digital watermark is embedded in accordance with an instruction from the optimization unit 33. The image generation unit 35 transmits the generated image data to the information terminal 200. The information terminal 200 displays the received two-dimensional image on the screen 10.

  FIG. 13 shows the flow of the optimization process performed with the optimization unit 33 as the center. As shown in FIG. 13, first, the optimization unit 33 selects a plurality of parent generation candidates (step S11). This candidate is a combination candidate of the pattern of the digital watermark and the pattern of the encoded aperture 20.

  Subsequently, the optimization unit 33 instructs the encoding aperture generation unit 34 and the image generation unit 35 to select a combination of the selected digital watermark pattern and the aperture pattern of the encoding aperture 20, and each of the combinations of the above patterns. Then, the imaging device 100 and the information terminal 200 are caused to execute imaging and digital watermark extraction (step S12). Here, the extraction rate of the digital watermark in each combination of the selected digital watermark pattern and the aperture pattern of the encoded aperture 20 is calculated by the extraction rate calculation unit 32 and output to the optimization unit 33.

  In step S12, specifically, as shown in FIG. 14, first, the optimization unit 33 sets the aperture pattern of the encoded aperture 20 in the drive unit 3 via the encoded aperture generator 34, and generates an image. The digital watermark is set by causing the unit 35 to generate the two-dimensional image 13 in which the digital watermark is embedded (step S1). The two-dimensional image 13 is transmitted to the information terminal 200, displayed on the screen 10 of the information terminal 200, and the encoded aperture 20 is formed in the aperture stop 2 by driving the driving unit 3.

  Subsequently, using the encoded aperture 20 as the aperture stop 2, image data of a two-dimensional image in which a digital watermark is embedded is captured by the image sensor 4 via the imaging optical system 1 (step S <b> 2).

  Subsequently, the imaging device 100 (the deconvolution unit 30 of the image processing unit 5) performs deconvolution on the image data captured by the imaging device 4 using a blur function corresponding to the encoding aperture 20, Image data of a two-dimensional image is obtained (step S3). As described above, clear image data deconvoluted by the blur function corresponding to the focus shift between the light receiving surface of the image sensor 4 and the imaging surface of the two-dimensional image 13 is acquired here.

  Subsequently, the image processing unit 5 (extraction unit 31) extracts a digital watermark from the image data of the two-dimensional image obtained by the deconvolution unit 30 (step S4). As described above, here, frequency conversion, extraction of a high frequency band, and inverse conversion are performed, and digital watermark image data is obtained.

  Subsequently, the computer 300 (extraction rate calculation unit 32) calculates the extraction rate of the extracted digital watermark (step S5). As described above, the extraction rate of the digital watermark is calculated based on the degree of coincidence between the obtained digital watermark image data and the reference digital watermark image data.

  Subsequently, the optimization unit 33 determines whether or not extraction has been completed at all distances (distance between the imaging device 100 and the information terminal 200) (step S6). If not completed yet (step S6; No), the optimization unit 33 drives the electric stage 201 to change the distance between the imaging device 100 and the information terminal 200 (step S7). Thereafter, the optimization unit 33 repeats imaging (step S2), deconvolution (step S3), digital watermark extraction (step S4), extraction rate calculation (step S5), and extraction completion determination (step S6).

  If it is determined that extraction of digital watermarks at all distances has been completed (step S6; Yes), the optimization unit 33 performs digital watermarking using all combinations of the aperture pattern of the encoded aperture 20 and the digital watermark pattern. It is determined whether or not extraction has been completed (step S8). If not completed (step S8; No), the optimization unit 33 sets a digital watermark pattern and an aperture pattern of the encoded aperture 20 that are the next candidates (step S1). Thereafter, steps S2 to S8 are repeated, and the process for the next candidate is performed. When all the combinations are completed (step S8; Yes), the optimization unit 33 ends the process of step S12.

  In this way, the optimization unit 33 changes the digital watermark pattern and the aperture pattern of the encoding aperture 20, changes the distance between the imaging device 100 and the information terminal 200, and performs imaging (step S2) and deconvolution. (Step S3), digital watermark detection (Step S4), and extraction rate calculation (Step S5) are repeated.

  Returning to FIG. 13, subsequently, the optimization unit 33 generates a plurality of candidate child generations by the genetic algorithm (GA) (step S13). In GA, a candidate is an individual, and crossover (recombination) and mutation are performed to generate a child generation candidate. It is to be noted that an individual that is the best in the parent generation is left as it is as an individual of the child generation.

  Subsequently, the optimization unit 33 determines whether or not the value of the digital watermark extraction rate has converged within a certain range (step S14). This is a so-called end-of-search determination. The determination condition for the end determination may be that the number of generations is a predetermined number, or that the extraction rate exceeds a threshold value.

  When it is determined that the value of the extraction rate has not converged (step S14; No), the optimization unit 33 returns to step S12, executes digital watermark extraction in the child generation (step S12), and selects candidate child generations. Generation (step S13) and extraction rate convergence determination (step S14) are repeated.

  When it is determined that the extraction rate has converged (step S14; Yes), the optimization unit 33 determines a candidate having the highest extraction rate as a final pattern (step S15). Subsequently, the optimization unit 33 outputs the determined candidate (step S16) and ends the process.

  Note that the optimization method is not limited to the genetic algorithm. A hill climbing method, annealing method, particle swarm optimization, differential evolution method, or the like may be used. A multi-objective optimization algorithm having a plurality of objective functions may be used. For example, by executing the multi-objective optimization algorithm with the first objective function as the watermark extraction rate and the second objective function as the defocus range from which the watermark can be extracted, both the ease and robustness of watermark extraction are achieved. It is also possible to optimize in consideration of the above.

  Since the digital watermark is embedded in the two-dimensional image as a higher frequency pattern, the aperture pattern of the encoded aperture 20 has a higher frequency compared to the encoded aperture that removes the blur of the license plate or barcode. It becomes a pattern.

  The direction corresponding to the spatial frequency component of the image in which the digital watermark is embedded among the plurality of images decomposed from the two-dimensional image by frequency analysis as a result of using the coding pattern optimization method as described above. It has been found that the coding aperture 20 in which a code pattern having a spatial frequency component is formed can extract a digital watermark well. For example, when a two-dimensional image is divided into LL component, LH component, HL component, and HH component images and a digital watermark is embedded in the HH component image, the two-dimensional image is obliquely crossed in the horizontal and vertical directions. It has been found that the coding aperture 20 in which the code pattern having the spatial frequency component, that is, the check pattern having the spatial frequency component in the oblique direction shown in FIG.

  In order to confirm that this encoded aperture 20 is good for extracting a digital watermark, it is used for the encoded pattern 20 of a check pattern that can extract a digital watermark well, a circular aperture, and blur removal. A comparative experiment of digital watermark extraction with a coded aperture (Zhou code) was performed. FIG. 15A shows a check pattern encoding opening 20 (left) and a digital watermark detection result (right) when the check pattern is used. In FIG. 15B. A circular opening 40 and a digital watermark detection result when the circular opening 40 is used are shown. FIG. 15C shows a coded aperture (Zhou code) 41 and a digital watermark detection result when the code is used. FIG. 15D shows another encoded aperture (Zhou code) 42 and the detection result of the digital watermark when that code is used. As can be seen from a comparison between FIG. 15A and FIG. 15B to FIG. 15D, the check pattern encoding aperture 20 can extract the digital watermark in the best manner.

  When the digital watermark is embedded in the LH component, the code pattern of the encoding aperture 20 may be a pattern having only a spatial frequency component in the vertical direction, or the digital watermark is embedded in the HL component. In such a case, the pattern may have only a spatial frequency component in the horizontal direction.

  The aperture stop 2 is not limited to the above. When the aperture stop 2 is a liquid crystal aperture, the aperture shape of the liquid crystal aperture is changed with time so that the circular aperture 40 shown in FIG. 16 (A) and the encoded aperture 20 shown in FIG. 16 (B) are alternately formed. You may make it make it. The circular aperture 40 is an aperture stop for extracting a two-dimensional image (cover image), and the encoded aperture 20 is an aperture stop for extracting a digital watermark. As shown in FIG. 16C, the driving unit 3 outputs a signal indicating whether the circular opening 40 is currently displayed or the encoded opening 20 is displayed to the image processing unit 5. Based on this signal, the image processing unit 5 applies image data acquired based on light incident on the image sensor 4 through the circular aperture 40 when the circular aperture 40 is displayed on the aperture stop 2. The image processing is executed to extract a two-dimensional image. On the other hand, the image processing unit 5 obtains based on light incident on the image sensor 4 through the encoded aperture 20 when the encoded aperture 20 is displayed on the aperture stop 2 based on the input signal. Image processing is executed on the obtained image data, and a digital watermark is extracted. In this way, both the two-dimensional image and the digital watermark can be extracted with high accuracy.

  As shown in FIG. 17A, the aperture stop 2 may be formed with a circular aperture 40 for extracting a two-dimensional image and an encoding aperture 20 for extracting a digital watermark. In this case, the imaging optical system 1 captures an image A based on light through the circular aperture 40 and an image B based on light through the encoding aperture 20, as shown in FIG. A half mirror 1C, mirrors 1D and 1E, and a half mirror 1F are provided to separately form an image on the element 4. The light beam emitted from the two-dimensional image on the screen 10 enters the half mirror 1C via the lens 1A. The half mirror 1C reflects a part of the incident light beam and transmits the remaining part. The light beam reflected by the half mirror 1C is reflected by the mirror 1D along the optical axis AX1, passes through the circular opening 40, is reflected by the mirror 1E, is further reflected by the half mirror 1F, and enters the imaging device 4 through the lens 1B. . The light beam transmitted through the half mirror 1C enters the half mirror 1F along the optical axis AX2 through the encoding aperture 20, and the light beam transmitted through the half mirror 1F enters the image sensor 4 through the lens 1B. .

  In this imaging optical system 1, an image A that is imaged by the light beam that has passed through the circular aperture 40 and an image B that is imaged by the light beam that has passed through the encoding aperture 20 are separately imaged on the image sensor 4. It is configured. The image processing unit 5 performs image processing on the image data of the image A to extract a two-dimensional image, and executes image processing on the image data of the image B to extract a digital watermark.

  In addition, instead of the configuration using the half mirrors 1C and 1F and the mirrors 1D and 1E, the imaging optical system 1 is configured by using a microlens array, a lenticular lens, or the like as a light dividing unit, and an image through the circular aperture 40 is used. And the image through the encoding aperture 20 may be separately formed on the image sensor 4.

  Similarly to the circular aperture 40 and the encoded aperture 20 described above, two encoded apertures that can remove blur from a watermark image that is a high-frequency component and a cover image that is a low-frequency component are temporal. Or it can be combined spatially.

  In the above embodiment, the case where the digital watermark 12 is extracted from the two-dimensional image 13 displayed on the screen 10 of the information terminal 200 has been described, but the present invention is not limited to this. The present invention can also be applied to extracting a digital watermark from a two-dimensional image printed on a paper medium.

  In the above embodiment, the program to be executed is readable by a computer such as a flexible disk, a CD-ROM (Compact Disc Read-Only Memory), a DVD (Digital Versatile Disc), or an MO (Magneto-Optical Disc). The computer 300 that executes the processes shown in FIGS. 6, 13, and 14 may be configured by storing and distributing the program on a simple recording medium and installing the program in a computer or the like.

  Further, the above-described program may be stored in a disk device or the like of a predetermined server device on a communication network such as the Internet, and may be downloaded, for example, superimposed on a carrier wave.

  In addition, when the processing shown in FIGS. 6, 13, and 14 is realized by each OS (Operating System) sharing, or when the processing shown in FIG. Only the part may be stored and distributed in a medium, or may be downloaded.

  Various embodiments and modifications can be made to the present invention without departing from the broad spirit and scope of the present invention. The above-described embodiments are for explaining the present invention and do not limit the scope of the present invention. In other words, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications within the scope of the claims and within the scope of the equivalent invention are considered to be within the scope of the present invention.

  The present invention can be applied to a system for extracting a digital watermark embedded in a two-dimensional code.

DESCRIPTION OF SYMBOLS 1 Imaging optical system, 1A, 1B lens, 1C half mirror, 1D, 1E mirror, 1F half mirror, 2 aperture stop, 3 Drive part, 4 Image sensor, 5 Image processing part, 10 screen, 11 Two-dimensional code, 12 Digital watermark, 13 Two-dimensional image, 20 Coding aperture, 21 Control unit, 22 Main storage unit, 23 External storage unit, 25 Input / output unit, 28 Internal bus, 29 Program, 30 Deconvolution unit, 31 Extraction unit, 32 Extraction Rate calculation unit, 33 optimization unit, 34 encoding aperture generation unit, 35 image generation unit, 40 circular aperture, 41, 42 encoding aperture (Zhou code), 100 imaging device, 200 information terminal, 201 electric stage, 300 computer 500 design system

Claims (11)

An image sensor that captures a two-dimensional image formed on a light receiving surface and acquires image data of the two-dimensional image;
An imaging optical system that forms an image on a light receiving surface of the image sensor with a two-dimensional image embedded with a digital watermark;
An aperture stop having a coded aperture disposed on an optical path of light incident on the image sensor via the imaging optical system;
An image processing unit that performs image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired by the imaging device, and extracts the digital watermark from the image data;
An imaging apparatus comprising: The coded aperture includes
Of a plurality of images decomposed from the two-dimensional image by frequency analysis, a code pattern having a spatial frequency component is formed in a direction corresponding to the spatial frequency component of the image in which the digital watermark is embedded.
The imaging device according to claim 1. The image processing unit
Performing deconvolution using a blur function corresponding to the coded aperture to restore the image data of the two-dimensional image;
Extracting the digital watermark by performing frequency conversion processing on the restored image data of the two-dimensional image;
The imaging device according to claim 1 or 2. The image processing unit
Performing deconvolution using a function in which a blur function corresponding to the coded aperture and a function for extracting the digital watermark are combined to extract the digital watermark;
The imaging device according to claim 1 or 2. With a liquid crystal drive
The aperture stop is
It is a transmissive liquid crystal opening that forms the coded opening by being driven by the liquid crystal driving unit.
The imaging device according to any one of claims 1 to 4. The liquid crystal driving unit
Driving the liquid crystal aperture so that a first aperture for extracting the two-dimensional image and a second aperture as the encoding aperture for extracting the digital watermark are alternately formed;
The image processing unit
Performing image processing on the first image data acquired based on the light incident on the image sensor through the first aperture to extract the two-dimensional image;
Performing the image processing on the second image data acquired based on the light incident on the image sensor through the second aperture to extract the digital watermark;
The imaging device according to claim 5. The aperture stop is
A first opening for extracting the two-dimensional image, and a second opening as the encoded opening for extracting the digital watermark;
The imaging optical system is
A light splitting unit configured to separately form a first image based on the light passing through the first aperture and a second image based on the light passing through the second aperture on the image sensor; And
The image processing unit
Performing image processing on the image data of the first image to extract the two-dimensional image;
Performing image processing on the image data of the second image to extract the digital watermark;
The imaging device according to any one of claims 1 to 6. The two-dimensional image is
It is a two-dimensional code displayed on the image of the information terminal.
The imaging device according to any one of claims 1 to 7. The aperture stop is disposed at a pupil position of the imaging optical system.
The imaging device according to any one of claims 1 to 8. An imaging step of acquiring image data by imaging image data of a two-dimensional image in which a digital watermark is embedded through an imaging optical system and an aperture stop having a coded aperture; and
An image processing step of performing image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired in the imaging step, and extracting the digital watermark from the image data;
A method for extracting a digital watermark including: An imaging step of acquiring image data by imaging image data of a two-dimensional image in which a digital watermark is embedded through an imaging optical system and an aperture stop having a coded aperture; and
An image processing step of performing image processing including deconvolution using a blur function corresponding to the encoded aperture on the image data acquired in the imaging step, and extracting the digital watermark from the image data;
A calculation step of calculating an extraction rate of the digital watermark extracted in the image processing step;
Including
While changing the digital watermark pattern and the encoded aperture pattern, the imaging step, the image processing step, and the calculation step are repeated, and the extraction rate is maximized by using an optimization method. Determining the watermark and the coded aperture;
Method for optimizing digital watermark and coded aperture.

Images