JP6794316B2 - Image processing device, image processing method and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Conventionally, shooting techniques called Time Slice, Bullet Time, and the like are known. In this photographing technique, a plurality of imaging devices (for example, cameras) are arranged around an imaging target (for example, a subject such as a human being). At this time, the plurality of imaging devices are arranged so that the optical axis of each imaging device passes through a predetermined point on the imaging target, and images the subject. An image generated by combining image data captured by such a photographing technique is called a free viewpoint image or the like. The user can select an arbitrary viewpoint in the free viewpoint image, and can visually recognize the subject from various angles. For example, a free-viewpoint image may exert a higher sales promotion effect than a normal image when used for sales promotion or the like.

Regarding shooting of a subject, there is known a technique of recognizing a subject by shooting the subject on a sheet on which a marker, which is a feature point, is printed (for example, Patent Document 1). In the literature relating to the prior art, it is disclosed that the conditions under which the marker can be used can be freely changed by recording the feature point display medium information which is information about the marker itself in the marker and specifying it by image recognition. ing. As a technique related to a free viewpoint image, a method of generating a free viewpoint image and the like are disclosed (for example, Patent Documents 2 and 3).

JP-A-2007-286715 Japanese Unexamined Patent Publication No. 2014-239384 JP 2015-127903

However, with the above-mentioned prior art (Patent Document 1), it has been difficult for a general user to effectively utilize a free viewpoint image. Specifically, in the prior art according to the above-mentioned Patent Document 1, it is necessary to use a marker to which an identifier and feature information are added. It is difficult for a general user to prepare such a marker and use it for image processing. Therefore, in the prior art according to the above-mentioned Patent Document 1, for example, a general user can easily generate a free-viewpoint image, or a third party can view the generated free-viewpoint image. Difficult to promote.

The technique disclosed in the present application has been made in view of the above, and an object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program capable of effectively utilizing a free-viewpoint image.

The image processing apparatus according to the present application includes a circle as a component, and is information about a marker on which a predetermined figure is drawn inside the circle, and is two points of pixels existing on radiation from the center of the circle to the outside. A reception unit that acquires information indicated by the difference in brightness of the marker as feature information of the marker, and an image data created by being imaged by a predetermined imaging device and accepting image data including an elliptical figure. The image data is projected and converted so that the elliptical figure included in the unit and the image data received by the reception unit becomes a perfect circle, and the feature information of the perfect circle after the projection conversion and the feature information acquired by the acquisition unit are acquired. It is characterized by having a specific part that specifies the perfect circle as a marker for using the image data in a predetermined process by collating the feature information with each other to obtain the similarity between the feature information. To do.

According to the image processing device which is one aspect of the embodiment, there is an effect that the free viewpoint image can be effectively utilized.

FIG. 1 is a diagram showing an example of image processing according to the first embodiment. FIG. 2 is a diagram showing a configuration example of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram showing an example of an image data storage unit according to the first embodiment. FIG. 4 is a diagram showing an example of a free viewpoint image storage unit according to the first embodiment. FIG. 5 is a diagram showing an example of an ideal arrangement of an imaging device for a free viewpoint image. FIG. 6 is a diagram showing an example of an imaging process relating to a free viewpoint image according to the first embodiment. FIG. 7 is a diagram showing an example of arrangement of an imaging device for a free viewpoint image according to the first embodiment. FIG. 8 is a diagram for explaining a marker used by the image processing apparatus according to the first embodiment. FIG. 9 is a diagram showing an example of the generation process by the generation unit according to the first embodiment. FIG. 10 is a flowchart showing an image processing procedure according to the first embodiment. FIG. 11 is a diagram showing an example of image processing according to the second embodiment. FIG. 12 is a diagram showing a configuration example of the image processing apparatus according to the second embodiment. FIG. 13 is a diagram showing an example of the marker information storage unit according to the second embodiment. FIG. 14 is a diagram (1) for explaining an example of the marker according to the second embodiment. FIG. 15 is a diagram (2) for explaining an example of the marker according to the second embodiment. FIG. 16 is a diagram (3) for explaining an example of the marker according to the second embodiment. FIG. 17 is a hardware configuration diagram showing an example of a computer that realizes the functions of the image processing device.

Hereinafter, each embodiment of the image processing apparatus, the image processing method, and the image processing program according to the present application will be described in detail with reference to the drawings. Note that this embodiment does not limit the image processing apparatus, image processing method, and image processing program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

[1. First Embodiment]
[1-1. An example of image processing]
First, an example of image processing according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of image processing according to the first embodiment. As shown in FIG. 1, the image processing system 1 according to the first embodiment includes a user terminal 10 and an image processing device 100. The user terminal 10 and the image processing device 100 are communicably connected to each other via a network N (for example, the Internet) (not shown). The number of each device included in the image processing system 1 is not limited to the example shown in FIG. For example, the image processing system 1 may include a plurality of user terminals 10.

The user terminal 10 is an information processing device used by a user who requests the image processing device 100 to generate a free viewpoint image. For example, the user terminal 10 is realized by a desktop PC (Personal Computer), a notebook PC, a mobile phone such as a smartphone, a PDA (Personal Digital Assistant), a tablet terminal, or the like. In the following, the user may be read as the user terminal 10. For example, "delivering a free viewpoint image to a user" may actually mean "delivering a free viewpoint image to a user terminal 10 used by a user".

In the first embodiment, the user who uses the user terminal 10 is, for example, a seller who sells a product owned by the user terminal 10 on an auction site or the like. The user takes a picture of the product from various angles and acquires a plurality of image data. Subsequently, the user transmits a plurality of image data to the image processing device 100 via the user terminal 10. Then, the user acquires a free viewpoint image generated based on the plurality of image data from the image processing device 100. The user, for example, uses a free-viewpoint image as one of the information for introducing the product and puts it up on an auction site or the like.

The image processing device 100 is a server device that generates a free viewpoint image which is an image capable of continuously displaying an image corresponding to an arbitrary viewpoint position. Specifically, the image processing device 100 according to the first embodiment accepts submission of a plurality of image data from a user who desires distribution of a free viewpoint image, and generates a free viewpoint image from the received plurality of image data. To do. Then, the image processing device 100 distributes the generated free viewpoint image to the user terminal 10.

The image processing device 100 generates a free viewpoint image by performing a predetermined conversion process on the original image data submitted from the user terminal 10. The image processing device 100 converts a plurality of images captured from multiple viewpoints into a free-viewpoint image as if the images were taken with the optical axes of the respective image pickup devices (cameras) overlapping at one point at a three-dimensional position. A projective transformation matrix is derived for the transformation. In this case, the image processing device 100 refers to a camera parameter including the position and orientation of the image pickup device as a parameter of the image pickup device that captures each image. In other words, in the generation of the free viewpoint image, it is desirable that the imaging device at the time of acquiring each image data is strongly calibrated.

A known method of estimating camera parameters based on captured image data when the camera parameters cannot be specified is also known. However, when imaging is not performed using commercial equipment or equipment, it is not possible to accurately estimate camera parameters with known methods, and problems may occur in the generation of free-viewpoint images. is there. For example, an image taken for the purpose of introducing a product to be put up on an auction site is often taken by the seller himself, and is rarely taken by a professional photographer or the like. Therefore, with known methods, it may be difficult to estimate the camera parameters with high accuracy with respect to the image data that is the source of the free-viewpoint image.

Therefore, the image processing apparatus 100 according to the first embodiment uses a visual marker (hereinafter, simply referred to as a “marker”) included in the image data to perform predetermined calibration with respect to a plurality of submitted image data. Perform (camera calibration). As the marker, a figure drawn so that each can be identified is used. When a marker is used, the image processing device 100 can perform the calibration process by obtaining the rotation angle R and the translation amount T between the marker in the image and the image pickup device. As a result, the image processing apparatus 100 acquires an appropriate parameter (for example, a projective transformation matrix) for projective conversion from the original image to an image for forming a free viewpoint image. Then, the image processing device 100 generates a free viewpoint image capable of smoothly transitioning the display of a plurality of images by connecting the projected images. Hereinafter, the image processing performed by the image processing apparatus 100 will be described along the flow.

First, the user prepares image data as a source for generating a free viewpoint image. At this time, the user shall prepare a plurality of image data including the marker in the image together with the subject (step S11). The user can use the circle drawn on the paper (sheet) as an example of the marker. Specifically, the user prepares a sheet 70 provided by the image processing device 100 and having predetermined circles drawn at the four corners. For example, the user acquires the image data of the sheet 70 via the network N. Then, the user acquires the sheet 70 by printing out the image data of the sheet 70 on a predetermined paper (for example, A4 size paper).

Circle markers M01, M02, M03 and M04 represented by circles are drawn on the sheet 70. The circle adopted as a circle marker shall be a perfect circle (perfect circle) with the same distance from any point on the circumference to the center. Further, it is assumed that the image processing device 100 has information on the positional relationship (for example, the relationship on coordinates) of the circular markers M01, M02, M03, and M04 on the sheet 70. In addition, the circle marker is recognized as two concentric circles, that is, an outer circle drawn in black and an inner circle drawn in the inner white part, not as one circle when the constituent circle has a width. It shall be.

Further, it is assumed that the circular markers M01, M02, M03 and M04 have different sizes to the extent that the image processing apparatus 100 can recognize them. Therefore, when the image data includes two circle markers, the image processing device 100 can identify the circle markers by collating the correct circle size after the projective conversion. Although the details will be described later, the image processing apparatus 100 may identify the circle markers M01, M02, M03, and M04 by using the ratio of the radii of the outer circle and the inner circle. In this case, the image processing device 100 can identify the circle marker even when the image data includes only one circle marker.

The user places the subject 60 on the center of the sheet 70 and performs imaging from various angles. For example, the user uses the imaging function of the user terminal 10 to image the subject 60. Alternatively, the user captures the subject 60 using a device such as a digital camera capable of acquiring image data by imaging. Then, the user stores the image data acquired by the imaging in the user terminal 10.

In this way, the user prepares a plurality of image data of the subject 60 captured while being placed on the sheet 70. As shown in FIG. 1, the user prepares original image data P01, P02, P03, ..., P0N (N is an arbitrary number) captured so as to surround the subject 60. In the example shown in FIG. 1, the original image data P01 is an image of the subject 60 taken from the right side. Further, the original image data P02 is an image obtained by capturing the subject 60 from diagonally right front. Further, the original image data P03 is an image obtained by capturing the subject 60 from the diagonally right front side closer to the front side as compared with the original image data P02. As described above, the original image data P01 to P0N are, for example, image data captured in order so as to surround the subject 60 in a semicircular shape.

As described above, the circular markers M01, M02, M03 and M04 are drawn on the sheet 70. However, when the subject 60 is placed on the center of the sheet 70 for imaging, in many image data, one or two circular markers are not included in the image data because they are shielded by the subject 60. In the example of the original image data P01, P02 and P03 shown in FIG. 1, since the circular marker M04 is shielded by the subject 60, it is not completely included in the image data. In other words, when the subject 60 is placed on the center of the sheet 70 and the image is taken, the image data includes two or three circular markers in most of the image data. For example, as shown in FIG. 1, the original image data P01, P02 and P03 are image data including the circle markers M01, M02 and M03 together with the subject 60. Although details will be described later, in the image processing performed by the image processing apparatus 100, the number of markers that need to be included in the image data changes depending on the type of marker. For example, when the marker included in the image is a circle marker, the image processing device 100 can generate a free viewpoint image from a plurality of image data including at least two circle markers in each image.

Subsequently, the user terminal 10 submits a plurality of original image data P01 to P0N to the image processing device 100 according to the user's operation (step S12). For example, the user terminal 10 uploads image data by uploading a plurality of original image data P01 to P0N on a web page that provides an image generation service provided by the image processing device 100.

The image processing device 100 accepts submission of original image data P01 to P0N in which each image data includes at least two circular markers together with the subject 60. Then, the image processing device 100 generates a free viewpoint image based on the submitted image data (step S13). Specifically, the image processing apparatus 100 performs predetermined calibration on a plurality of submitted image data based on at least two circular markers included in one image data.

Generally, the calibration process for image data using markers is realized by calculating the parameters of the angle of rotation R and the amount of translation T between the marker in the image data and the image pickup device (camera) that captured the image data. To.

Here, the calibration process performed by the image processing apparatus 100 according to the first embodiment will be described by taking the original image data P01 as an example. First, the image processing device 100 detects each circular marker in the original image data P01. In this example, the image processing apparatus 100 shall detect the circular markers M01 and M02. The image processing device 100 identifies the circle markers M01 and M02 by the difference in the ratio of the outer circle and the inner circle of the circle markers M01 and M02, respectively. As described above, each circle marker drawn on the sheet 70 is a perfect circle. Therefore, in the image data acquired by imaging, each circular marker becomes an ellipse.

The image processing apparatus 100 estimates the normal of each ellipse based on the detected parameters of each ellipse. Two ellipse normals are calculated for each ellipse. Therefore, the image processing apparatus 100 can estimate the plane including the circle markers M01 and M02 by combining the solutions of the normals calculated from the two circle markers M01 and M02. Further, the image processing apparatus 100 can estimate the normal of the estimated plane.

The image processing device 100 uses the estimated plane normal as the world coordinate system (coordinate system for indicating the position of an object in space. “World coordinate system” or “global coordinate system” in the original image data P01. It is assumed to be the Y-axis in (sometimes written). Then, the image processing apparatus 100 uses a vector connecting appropriate circle centers (for example, circle markers M01 and M02) as a temporary X-axis in the world coordinate system. Based on this information, the image processing apparatus 100 can determine the relationship between the rotation angle R of the circular marker M01 or M02 in the image and the imaging apparatus. Further, the image processing apparatus 100 can calculate a temporary translation amount T by assuming that the distance between the circle centers is 1. The image processing apparatus 100 can determine which circle can be detected as an ellipse by obtaining the original magnitude relationship of the detected circles by using the tentative parameters R and the translation amount T. it can. Since the position information of the circle on the sheet 70 is known, the image processing apparatus 100 can estimate the correct X-axis and the translation amount T based on the position of the detected circle in the world coordinate system. As a result, the image processing apparatus 100 completes the strong calibration.

Then, the image processing device 100 acquires appropriate parameters for projective conversion from the submitted image data to an image for forming a free viewpoint image. Then, the image processing apparatus 100 projects and transforms the image data using the acquired parameters. The image processing device 100 generates a free viewpoint image F01 capable of displaying each submitted image data from an arbitrary viewpoint by connecting the projected images.

The image processing device 100 distributes the generated free viewpoint image F01 to the user terminal 10 (step S14). As shown in FIG. 1, the free viewpoint image F01 includes the converted image data RP01 to RP0N obtained by converting the original image data P01 to P0N. Further, the free viewpoint image F01 can smoothly transition the display of the converted image data RP01 to RP0N according to, for example, a user operation. Further, as shown in FIG. 1, each subject 60 included in the converted image data RP01 to RP0N has a uniform size as a result of calibration as compared with the subject 60 included in the original image data P01 to P0N. It is aligned.

The user terminal 10 acquires the free viewpoint image F01 distributed from the image processing device 100. For example, when listing on an auction site, the user uses the free viewpoint image F01 as one of the information for introducing his / her own product (step S15).

As described above, in the image processing apparatus 100 according to the first embodiment, the circular markers M01 to M04, which are a plurality of image data having different viewpoint positions and have predetermined parameters and can be identified, are used for each image. It accepts a plurality of original image data P01 to P0N including at least two in the data. Then, the image processing device 100 calibrates the received plurality of original image data P01 to P0N based on the parameters of the circular marker. Further, the image processing device 100 generates a free viewpoint image F01 capable of continuously displaying an image corresponding to an arbitrary viewpoint position by performing a predetermined conversion process on a plurality of calibrated image data.

As a result, the image processing device 100 according to the first embodiment provides the free viewpoint image to a user who promotes product sales by utilizing the free viewpoint image, for example, a user of the user terminal 10. Can be done. That is, by using the free viewpoint image for sales promotion or the like, the user can exert a higher sales promotion effect than presenting a normal image. Further, the image processing device 100 performs a proofreading process using a marker when generating a free viewpoint image. By using a marker, the image processing device 100 can robustly calibrate even from the original image data that has not been acquired in a shooting environment equipped with specialized equipment and devices. That is, the image processing device 100 can generate a free viewpoint image that can be naturally displayed from an arbitrary viewpoint even from the original image data acquired by the image pickup device that has not been strongly calibrated. As a result, the image processing device 100 has an effect that the free viewpoint image can be effectively utilized.

[1-2. Image processing device configuration]
Next, the configuration of the image processing apparatus 100 according to the first embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the image processing device 100 according to the first embodiment. As shown in FIG. 2, the image processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

(About communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to a network N (not shown) by wire or wirelessly, and transmits / receives information to / from the user terminal 10.

(About storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 2, the storage unit 120 includes an image data storage unit 121 and a free viewpoint image storage unit 122.

(About the image data storage unit 121)
The image data storage unit 121 stores information regarding image data submitted from the user terminal 10. Here, FIG. 3 shows an example of the image data storage unit 121 according to the first embodiment. In the example shown in FIG. 3, the image data storage unit 121 has items such as "submission ID" and "image data".

The “traffic ID” indicates identification information for identifying the submitted image data. The information indicated by the submission ID may include, for example, information that identifies the date and time of submission and the user terminal 10 that is the submission source.

"Image data" indicates the submitted image data. As shown in FIG. 1, the user terminal 10 submits a plurality of image data as the image data that is the source of the free viewpoint image. A series of image data submitted from the user terminal 10 is associated and stored in the item of "image data". Further, identification information by a serial number such as "P01, P02, P03, P04, P05 ..." may be added to each image data. Based on such serial numbers, the image processing apparatus 100 recognizes the connection order of the images in the free viewpoint image.

That is, in FIG. 3, in the submission of the image data identified by the submission ID "A01", the image data "P01, P02, P03, P04, P05 ..." Is the source image for generating the free viewpoint image. Indicates that the data was submitted as data.

(About the free viewpoint image storage unit 122)
The free viewpoint image storage unit 122 stores information about the free viewpoint image generated by the image processing device 100. Here, FIG. 4 shows an example of the free viewpoint image storage unit 122 according to the first embodiment. In the example shown in FIG. 4, the free viewpoint image storage unit 122 has items such as "free viewpoint image ID", "original image data", "calibration data", "converted image data", and "initial display screen". ..

The "free viewpoint image ID" indicates the identification information of the free viewpoint image. The "original image data" indicates the image data from which the free viewpoint image is generated. Since the free viewpoint image is generated from a plurality of image data, a plurality of original image data are stored for one free viewpoint image.

"Calibration data" indicates predetermined calibration data (camera calibration) for generating a free viewpoint image. In FIG. 4, the calibration data is shown by a concept such as “R01”. The calibration data includes various data for the image processing apparatus 100 to appropriately project and transform the original image data in order to acquire an image constituting the free viewpoint image. For example, the calibration data includes the rotation angle R between the marker and the imaging device and the translation amount T. The calibration data may include the focal length of the image pickup device, the aspect ratio of the image generated by the image pickup device, the skew, and the like as camera parameters of the image pickup device for each image data.

The "converted image data" indicates the image data constituting the free viewpoint image. The converted image data is, for example, image data after a predetermined calibration is performed on the original image data and the image data is projected and converted according to the parameters calculated by the calibration.

The "initial display screen" shows an image corresponding to the initial display of the free viewpoint image. On the initial display screen, one of the converted image data after conversion is selected.

That is, in FIG. 4, the original image data of the free viewpoint image identified by the free viewpoint image ID "F01" is "P01, P02, P03, P04, P05, ...", And the respective calibration data is ". R01, R02, R03, R04, R05, ... ”, the converted image data is“ RP01, RP02, RP03, RP04, RP05, ... ”, and the initial display screen is“ RP03 ”. Is shown.

As the free viewpoint image, a free viewpoint image having a different rotation center (gaze point) of the free viewpoint image may be generated by arbitrary selection by the user. In this case, the free viewpoint image storage unit 122 may store information about the gazing point in association with the free viewpoint image ID.

(About control unit 130)
The control unit 130 is, for example, a controller, and various programs (of an image processing program) stored in a storage device inside the image processing device 100 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. (Corresponding to one example) is realized by executing RAM as a work area. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2, the control unit 130 includes a reception unit 131, a calibration unit 132, a generation unit 133, and a distribution unit 134, and realizes or executes an information processing function or operation described below. .. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 2, and may be another configuration as long as it is a configuration for performing information processing described later. Further, the connection relationship of each processing unit included in the control unit 130 is not limited to the connection relationship shown in FIG. 2, and may be another connection relationship.

(About reception desk 131)
The reception unit 131 receives a plurality of image data having different viewpoint positions. Specifically, the reception unit 131 according to the first embodiment has a plurality of image data having different viewpoint positions, and at least a marker having a predetermined parameter and capable of identifying each of them is included in each image data. One accepts submissions of multiple image data included.

As described above, in the image processing performed by the image processing apparatus 100, the number of markers that need to be included in the image data changes depending on the type of markers. For example, when the marker included in the image is a circle marker, the calibration unit 132 and the generation unit 133, which will be described later, can generate a free viewpoint image from a plurality of image data including at least two circle markers in each image. That is, when the marker is a circle marker, the reception unit 131 accepts the submission of a plurality of image data in which at least two circle markers that can identify each of them are included in each image data.

Further, the parameters of the marker used by the image processing apparatus 100 may also differ depending on the type of marker. For example, in the case of a circular marker, the image processing device 100 uses the length of the diameter of the concentric circles constituting the circular marker, the position information of each circular marker on the sheet 70, and the like as parameters. Further, for example, when the image processing device 100 is a marker having two sets of line segments parallel to the identification information, such as a quadrangular two-dimensional code (for example, a QR code (registered trademark)), the identification information is used. Two sets of parallel line segments are used as parameters. In other words, the image processing apparatus 100 uses two sets of parallel line segments, which are the line segments constituting the quadrangle, as parameters in the case of the quadrangular marker as a parameter instead of the estimated normal in the case of the circular marker. ..

The reception unit 131 assigns a submission ID to a series of image data for which submission has been received, and stores the submission ID in the image data storage unit 121.

(About calibration unit 132)
The calibration unit 132 calibrates a plurality of image data received by the reception unit 131 based on the parameters of the marker. Specifically, the calibration unit 132 according to the first embodiment is used as a transformation matrix for converting the submitted original image data into an image constituting a free viewpoint image by performing predetermined calibration. Calculate the camera parameters. The generation unit 133, which will be described later, generates a free viewpoint image by performing a predetermined conversion process on the original image data based on the result of calibration by the calibration unit 132.

Here, the free viewpoint image handled by the calibration unit 132 and the generation unit 133 will be described with reference to FIGS. 5, 6 and 7. First, the arrangement of the imaging device for the free viewpoint image will be described with reference to FIG. FIG. 5 is a diagram showing an example of an ideal arrangement of an imaging device for a free viewpoint image. FIG. 5 shows an example of an ideal arrangement of an imaging device for capturing a plurality of image data that are the source of a free viewpoint image. Note that FIG. 5 shows an example of the subject 60 viewed from above.

In FIG. 5, the image pickup devices C11 to C16 are, for example, digital cameras. Further, the imaging devices C11 to C16 are arranged around the subject 60 to be imaged so that the optical axis of each imaging device passes through a predetermined position. In the example of FIG. 5, the image pickup apparatus C11 is arranged so that the optical axis E11 ₁ passes through the three-dimensional position D1 corresponding to the center of the subject 60. Further, the image pickup devices C12 to C16 are arranged so that their respective optical axes E12 _{1 to} E16 ₁ pass through the three-dimensional position D1. In an ideal situation, the imaging devices C11 to C16 are installed so that at least the bottom of the projection surface is substantially parallel to the ground. Then, the imaging devices C11 to C16 can acquire image data that is a source for generating a free viewpoint image of the subject 60 by simultaneously imaging the subject 60 in the state shown in FIG.

In the example of FIG. 5, an example in which the subject 60 is imaged using six image pickup devices C11 to C16 is shown, but the number of image pickup devices is not limited to this example. Further, the arrangement positions of the image pickup devices C11 to C16 are not limited to the example shown in FIG. For example, the imaging devices C11 to C16 may be arranged at equal intervals on the circumference so that a plurality of imaging devices surround the subject 60, instead of surrounding the subject 60 in a semicircular shape.

However, in the imaging method of FIG. 5, appropriate arrangement of the imaging devices C11 to C16 and appropriate calibration of the imaging devices C11 to C16 are required. That is, it is difficult for a general user who is not an imaging expert such as a user to perform the imaging process as shown in FIG.

Here, with reference to FIGS. 6 and 7, an imaging process in which the subject 60 is imaged by a general user or the like (hereinafter, simply referred to as “user”) who is not an imaging expert will be described. FIG. 6 is a diagram showing an example of an imaging process relating to a free viewpoint image according to the first embodiment. FIG. 6 shows a situation in which a user uses the imaging device C20 to image a subject 60 placed on a sheet 70.

As shown in FIG. 6, the user continuously images the subject 60 so as to rotate around the subject 60 placed on the sheet 70. In this case, it is assumed that the optical axis and focal length of the camera are not constant for each captured image data as compared with FIG.

As described in FIG. 1, in each image data obtained by the user capturing the subject 60 from the surroundings, one of the circle markers drawn on the sheet 70 is shielded and hidden, but at least two circle markers are present. It is expected to be included in each image data.

Next, a case where the imaging situation shown in FIG. 6 is viewed from above will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of arrangement of an imaging device for a free viewpoint image according to the first embodiment. FIG. 7 shows an arrangement of an imaging device and the like that are assumed when a user actually images a subject 60 for capturing a plurality of image data that are the basis of a free viewpoint image. In the following description, when it is not necessary to distinguish the imaging devices C20 _{1 to} C20 _N , the term "imaging device C20" is used.

The image pickup devices C20 _{1 to} C20 _N are, for example, digital cameras. Ideally, the image pickup devices C20 _{1 to} C20 _N are arranged around the subject 60 as individual image pickup devices as described with reference to FIG. However, in practice, the user can move the image pickup device C20 ₁ one at a location indicated by the imaging device _C20 1 -C20 _N in FIG. 7, it is assumed for imaging a subject 60. Specifically, the user captures a subject 60 from the imaging device C20 ₁ location shown in FIG. Then, the user, along a circle around the object 60, and moves to the image pickup device C20 ₂ locations shown in FIG. 7, for imaging a subject 60. By repeating this work, the user acquires a plurality of image data for the subject 60.

In this case, even if the user optical axis E20 ₁ is carefully imaged so as to pass through the center of the object 60, exactly as the original image of the free viewpoint image (for example, as shown in FIG. 5) for imaging the subject 60 It's difficult. Further, since the image pickup device C20 handled by the user is not always camera-calibrated, it may not be possible to acquire camera parameters or the like from the image data.

Therefore, the calibration unit 132 makes it possible to convert the image data received by the reception unit 131 into an image constituting a free viewpoint image by performing predetermined calibration using a marker. This point will be described with reference to FIG.

FIG. 8 is a diagram for explaining a marker used by the image processing apparatus 100 according to the first embodiment. FIG. 8 describes the sheet 70 in which the circular markers M01, M02, M03 and M04 used by the calibration unit 132 for calibration are drawn at the four corners.

As shown in FIG. 8, the sheet 70 includes circular markers M01, M02, M03 and M04, and a central display 75. The central display 75 is, for example, a display that serves as a guide when the subject 60 is placed on the sheet 70. The user acquires the sheet 70 shown in FIG. 8 and images the subject 60 placed on the sheet 70.

Further, as described above, the calibration unit 132 acquires the position information of the circular markers M01, M02, M03 and M04 drawn on the sheet 70 in advance. As a result, if the calibration unit 132 can identify each circle marker, the positional relationship of each of the circle markers M01, M02, M03 and M04 can be estimated, so that the calibration process using the markers can be performed. it can. In addition, the calibration unit 132 may give the position information of the marker as an absolute value in the world coordinate system in the image data. That is, the calibration unit 132 performs a mapping process for mapping a plurality of markers in advance in the world coordinate system. As a result, the calibration unit 132 can estimate the position of the shielded marker and perform the calibration process even when the marker shielded by the subject 60 or another object is present in the image data.

That is, the calibration unit 132 calibrates a plurality of image data based on the correspondence between the position information of each of the plurality of markers as the parameters of the markers. Further, when the marker is a circle marker, the calibration unit 132 includes a plane information including the normal of the circle marker estimated using the parameters of the circle marker and two circle markers included in the plane corresponding to the plane information. Calibrate a plurality of image data based on the correspondence of the position information of.

(About generator 133)
The generation unit 133 generates a free viewpoint image capable of continuously displaying an image corresponding to an arbitrary viewpoint position by performing a predetermined conversion process on a plurality of image data calibrated by the calibration unit 132.

As described with reference to FIG. 7, the original image data group submitted from the user terminal 10 is imaged with the optical axis directed to the subject 60, but strictly speaking, the optical axes are scattered vertically and horizontally. It is assumed. Therefore, even if the original image data group to be submitted is simply displayed continuously, it is presumed that the discontinuity is conspicuous in such a continuous image.

Therefore, in the generation unit 133, the optical axis of the imaging device C20 when each original image data is imaged passes through a certain point (for example, the three-dimensional position D1 shown in FIG. 5) in the three-dimensional space. The original image data is converted using a projective transformation matrix that transforms the image as if it had been done. Such a projective transformation matrix can be created by parameters derived by the calibration process performed by the calibration unit 132.

For example, the generation unit 133 generates the converted image data constituting the free viewpoint image by the projective transformation. For example, assuming that a predetermined imaging device (camera) corresponding to the converted image data is a camera k, the projection transformation matrix H _k in such a camera k is the rotation angle (rotation matrix) R of each camera and light. It can be obtained by the following equation (1) based on the three-dimensional point to be passed through the axis.

In the above formula (1), the inverse matrix of R _k, is the original rotation angle R in the camera k, and _R'k is a new angle of rotation R, and A _k is an internal parameter. Internal parameters are given, for example, by the image center position and focal length. Incidentally, the new rotation angle _R'k by known techniques, and the camera position, it is possible to obtain from the ideal optical axis determined from the center you want to rotate, the original and the optical axis in the world coordinate system.

The generation unit 133 generates a free viewpoint image by converting each original image data using the projective transformation matrix generated as in the above equation (1). Here, an example of the generation process by the generation unit 133 according to the first embodiment will be described with reference to FIG. FIG. 9 is a diagram showing an example of the generation process by the generation unit 133 according to the first embodiment.

As shown in FIG. 9, the generation unit 133 converts the original image data P01 to P05 into the converted image data RP01 to RP05 by performing a predetermined conversion process together with the calibration process by the calibration unit 132. Generate a free viewpoint image.

Taking the original image data P01 as an example, the calibration unit 132 detects at least two circular markers from the original image data P01. For example, the calibration unit 132 detects the circle marker M02 and the circle marker M03 from the original image data P01. Then, the calibration unit 132 calibrates the original image data P01 by the above-described processing. That is, the calibration unit 132 derives the camera parameters of the image pickup apparatus C20 that has captured the original image data P01 by the calibration process. In other words, the calibration unit 132 acquires the parameters for generating the projective transformation matrix.

Then, the generation unit 133 generates a projective transformation matrix in response to the result of the calibration process by the calibration unit 132. Subsequently, the generation unit 133 generates the converted image data RP01 by projecting and transforming the original image data P01. Similarly, the generation unit 133 generates the converted image data RP02 to RP05 based on the original image data P02 to P05, respectively (step S20).

After the calibration process using the marker is performed, the calibration unit 132 or the generation unit 133 may perform the calibration process or the projective conversion by using various known methods. For example, the calibration unit 132 or the generation unit 133 may use a known method called Structure From Motion (hereinafter referred to as “SFM”). According to SFM, it is possible to restore the three-dimensional shape of an image based on a plurality of image data captured from various viewpoint positions. Specifically, the SFM calculates the movement of the image pickup apparatus C20 by expressing the movement of the corresponding points with a basic matrix using the corresponding points associated with each other in the image data. Since the position information of the image pickup apparatus C20 can be restored according to the SFM, the calibration unit 132 may acquire the camera parameters by appropriately using the SFM method in a predetermined process.

As described above, the calibration unit 132 performs the camera parameters (for example, relative positions) of the image pickup apparatus C20 that acquired the image data based on the relationship of the corresponding markers among the plurality of image data as a predetermined calibration process. Information etc.) is acquired. Then, the generation unit 133 calculates the projection transformation matrix corresponding to each image data by using the calibration data such as the parameters acquired by the calibration unit 132. Then, the generation unit 133 converts each original image data using the calculated projective transformation matrix. Then, the generation unit 133 generates a free viewpoint image based on the converted converted image data. The generation unit 133 stores the generated free viewpoint image in the free viewpoint image storage unit 122. The processes disclosed in Patent Documents 2 and 3 of the prior art document may be appropriately applied to the processes executed by the calibration unit 132 and the generation unit 133.

The generation unit 133 extracts the image data to be handled in the generation process by determining whether or not the plurality of image data have a predetermined continuity, and generates a free viewpoint image using the extracted image data. You may. In other words, the generation unit 133 extracts the image data suitable for generating the free viewpoint image from the submitted image data and generates the free viewpoint image. For example, in the generation process, the generation unit 133 can exclude image data that has been erroneously submitted and image data that is inappropriate for generating a free-viewpoint image, out of a plurality of image data that have been submitted. Such processing can be performed, for example, when the marker cannot be detected or the corresponding marker is not detected in the original image data when performing the calibration process using the marker. In addition, when the free viewpoint image cannot be generated from the plurality of submitted image data, the generation unit 133 indicates the reason why the free viewpoint image cannot be generated, or the image data for enabling the generation of the free viewpoint image. Notification data including information related to replacement may be generated.

(About distribution unit 134)
The distribution unit 134 distributes the free viewpoint image generated by the generation unit 133. Specifically, the distribution unit 134 according to the first embodiment uses the free viewpoint image generated by the generation unit 133 and stored in the free viewpoint image storage unit 122 as the source of the original image data. Deliver to terminal 10.

In addition, the distribution unit 134 indicates that the original image data is not appropriate and the generation unit 133 cannot generate the free viewpoint image, and that the distribution unit 134 requests the submission of additional image data for generating the free viewpoint image. When the indicated notification data is generated, the notification data may be distributed to the source of the original image data.

[1-3. Image processing procedure]
Next, the procedure of image processing by the image processing apparatus 100 according to the first embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing an image processing procedure according to the first embodiment.

As shown in FIG. 10, the reception unit 131 of the image processing device 100 determines whether or not the submission of image data has been received from the user terminal 10 (step S101). At this time, if the reception unit 131 has not accepted the submission of the image data (step S101; No), the reception unit 131 waits until the submission of the image data is accepted.

On the other hand, when the reception unit 131 receives the submission of the image data (step S101; Yes), the reception unit 131 stores the received image data in the image data storage unit 121. Then, the proofreading unit 132 identifies (detects) a marker included in each image data (step S102).

Then, the proofreading unit 132 performs a proofreading process on the image data using the marker (step S103). Subsequently, the generation unit 133 converts the image data based on the calibration data of the calibration process performed by the calibration unit 132 (step S104).

Then, the generation unit 133 generates a free viewpoint image based on the converted image data (step S105). Subsequently, the distribution unit 134 distributes the free viewpoint image to the user terminal 10 (step S106).

[1-4. Modification example]
The image processing apparatus 100 according to the first embodiment described above may be implemented in various different forms other than the above-described embodiment. Therefore, another embodiment of the above-mentioned image processing apparatus 100 will be described below.

[1-4-1. Markers using figures other than circles]
In the above embodiment, it has been described that the image processing apparatus 100 performs calibration processing on each image data by using a circular marker. However, the image processing device 100 may use a marker other than the circular marker.

For example, the image processing device 100 can use a two-dimensional code having identification information as described above. In this case, since the two-dimensional code itself contains the identification information and one two-dimensional code contains two sets of parallel line segments (not parallel to each other), the image processing apparatus 100 uses each image data. The calibration process can be performed by detecting at least one marker from. It should be noted that the line segments need not be parallel in the calibration process as long as the line segments can define a plane in the image data.

As described above, the image processing apparatus 100 accepts a plurality of image data in which at least one two-dimensional code having a quadrangular shape and identification information is included in each image data as a marker. Then, the image processing device 100 uses the plane information estimated by using the information of the line segments of the two sides of the two-dimensional code and the position information of at least one two-dimensional code included in the plane corresponding to the plane information. Based on this, calibrate a plurality of image data. As described above, the image processing apparatus 100 can perform appropriate calibration even if the marker used for calibration is a figure other than a circle.

[Marker using 1-4-2.4 points]
Further, the image processing apparatus 100 can also use as a marker four or more points whose positional relationship with each other is known. In this case, the image processing device 100 accepts the submission of image data in which each image data contains at least four points.

That is, the image processing device 100 can connect two points out of the four points and define two line segments by detecting four points as markers in the image data. As a result, the image processing apparatus 100 can perform the same processing as the processing using the circle marker as described in the above embodiment.

Specifically, the image processing device 100 is a plurality of points in which the correspondence relationship of the mutual position information is predetermined, and a plurality of images including a plurality of points including at least four in each image data. Accept data. Then, the image processing device 100 extracts two line segments from the line segment connecting the two points out of the plurality of points, and the plane information including the plurality of points estimated using the extracted two line segments. , The calibration of a plurality of image data is performed based on the correspondence relationship of the position information of a plurality of points included in the plane corresponding to the plane information. As described above, the image processing apparatus 100 uses the points as markers by using the correspondence of each point based on the position information and coordinates of each point even if the marker itself does not have the identification information. Can be done.

[1-4-3. Utilization example (1)]
In the above embodiment, the image processing device 100 shows, as an example, a process of generating a free viewpoint image based on a plurality of image data submitted by a user who wants to put up an auction. In this case, the image processing device 100 uses the generated free-viewpoint image as identification information for identifying the user terminal 10 or the user who uses the user terminal 10 that has submitted a plurality of image data that are the source of the free-viewpoint image, or It may be stored in a predetermined storage unit in association with at least one of the exhibition information in the service used by the user.

That is, the image processing device 100 not only distributes the generated free viewpoint image to the user, but also has the free viewpoint together with the user terminal 10 and the user's identification information (for example, the user ID) and the exhibition information in the service used by the user. It may be stored in the image storage unit 122 or the like. The listing information in the service used by the user is, for example, identification information indicating the item for sale on the auction site, identification information for identifying the auction being held on the auction site, or the item on the shopping site. It is information that can identify an exhibit (for example, it is assumed that it is a subject of the original image data of a free viewpoint image), such as identification information for identifying.

In this case, the image processing device 100 receives, for example, a request for distribution of a free viewpoint image from a web server or the like that manages and operates the service. Then, when the image processing device 100 receives the request, the free viewpoint image is specified based on the user terminal 10 and the user's identification information and the exhibition information. Then, the image processing device 100 distributes the specified free viewpoint image to the service side (for example, a web server). The web server that has acquired the free-viewpoint image posts the free-viewpoint image as an introduction image of the product at the auction.

In this way, the image processing device 100 not only distributes the free viewpoint image to the user, but also stores the free viewpoint image in the storage unit 120 together with the information identifying the user, and requests the service side to store the free viewpoint image. The free viewpoint image may be delivered. As a result, the image processing device 100 can deliver the free viewpoint image in response to various requests, so that the free viewpoint image can be further utilized. In this case, the image processing device 100 stores information such as a user ID and a user's exhibition information ID in the free viewpoint image storage unit 122 shown in FIG. 4 in association with the free viewpoint image ID. It may be.

[1-4-4. Utilization example (2)]
In the above embodiment, as an example in which the free-viewpoint image is used by the user, listing of a product on an auction site or the like is shown. However, the image processing device 100 may distribute the generated free viewpoint image to various targets. For example, the image processing device 100 may provide a service of directly delivering to a web server or the like instead of the user terminal 10. Further, the image processing device 100 mainly distributes the generated free-viewpoint image to UGM (User-Generated Media), CGM (Consumer-Generated Media), or the like, which is a site for posting UGC (User-Generated Contents). May be good. The UGM includes a posted video sharing site, a photo sharing site, an illustration posting site, an SNS (Social Networking Service), and the like.

[1-5. effect〕
As described above, the image processing apparatus 100 according to the first embodiment has a reception unit 131, a calibration unit 132, and a generation unit 133. The reception unit 131 receives a plurality of image data having different viewpoint positions, and a marker having a predetermined parameter and capable of identifying each of the image data includes at least one of the image data. The calibration unit 132 calibrates a plurality of image data received by the reception unit 131 based on the parameters of the marker. The generation unit 133 generates a free viewpoint image capable of continuously displaying an image corresponding to an arbitrary viewpoint position by performing a predetermined conversion process on a plurality of image data calibrated by the calibration unit 132.

As described above, the image processing apparatus 100 according to the first embodiment performs the calibration process using the marker when generating the free viewpoint image. For example, the image processing device 100 uses image data obtained by capturing a marker together with a subject from original image data as if it was captured by a general user, which is not acquired in a shooting environment equipped with specialized equipment or devices. Even so, the calibration can be performed robustly. That is, the image processing device 100 can generate a free viewpoint image that can be naturally displayed from an arbitrary viewpoint even from the original image data acquired by the image pickup device that has not been strongly calibrated. As a result, the image processing device 100 has an effect that the free viewpoint image can be effectively utilized.

Further, the calibration unit 132 calibrates a plurality of image data based on the plane information estimated by using the parameters of the markers and the position information of the markers included in the plane corresponding to the plane information in the image data. Do.

As described above, the image processing apparatus 100 according to the first embodiment performs the calibration process by using the marker having the parameter (for example, the information defining the two line segments) capable of defining the plane information. Do. As a result, the image processing device 100 can perform appropriate calibration processing even for image data captured by a general user.

Further, the reception unit 131 receives a plurality of image data including a plurality of markers in each image data. As a parameter, the calibration unit 132 calibrates a plurality of image data based on the correspondence between the position information of each of the plurality of markers.

As described above, the image processing apparatus 100 according to the first embodiment can calibrate a plurality of image data based on the correspondence (mapping information) of the marker position information in the image data. As a result, the image processing apparatus 100 can appropriately identify the markers of each other, that is, the corresponding points between the images, so that robust calibration can be performed.

Further, the reception unit 131 receives a plurality of image data in which at least two circular markers, which are markers composed of two concentric circles, are included in each image data. The calibration unit 132 is based on the correspondence between the plane information including the normal of the circle marker estimated using the parameters of the circle marker and the position information of the two circle markers included in the plane corresponding to the plane information. , Perform calibration for multiple image data.

As described above, the image processing apparatus 100 according to the first embodiment can perform the calibration process using the circular marker. Since the circular marker has an elliptical parameter when projected as image data, the normal can be estimated. Therefore, the image processing apparatus 100 can easily estimate the plane information based on the normal, and can further estimate the normal of the plane. As a result, the image processing device 100 can correspond to the world coordinate system in the image data, so that appropriate calibration processing can be performed.

Further, the reception unit 131 receives a plurality of image data including a plurality of points in which the correspondence relationship of the position information with each other is predetermined and at least four are included in each image data. The calibration unit 132 extracts two line segments from a line segment connecting two points out of a plurality of points, and plane information including a plurality of points estimated using the extracted two line segments and plane information. Based on the correspondence between the position information of the plurality of points included in the plane corresponding to, the calibration is performed on the plurality of image data.

As described above, the image processing apparatus 100 according to the first embodiment can use at least four points as markers. Therefore, the image processing apparatus 100 can perform appropriate proofreading processing without using a figure such as a circle as long as the information can be distinguished from each other. That is, the image processing device 100 can perform highly versatile image processing.

Further, the reception unit 131 receives a plurality of image data in which at least one two-dimensional code having a quadrangular shape and identification information is included in each image data. The calibration unit 132 is based on the plane information estimated using the information of the line segments of the two sides of the two-dimensional code and the position information of at least one two-dimensional code included in the plane corresponding to the plane information. Calibrate for multiple image data.

As described above, the image processing apparatus 100 according to the first embodiment may use the two-dimensional code as a marker. Since the two-dimensional code has identification information for identifying the two-dimensional code itself, it can be identified by one marker without specifying two circles by size like a circle marker. Therefore, the image processing apparatus 100 can perform the calibration process based on at least one two-dimensional code. That is, the image processing device 100 can reduce the processing load such as calculating the correspondence between a plurality of markers.

[2. Second Embodiment]
[2-1. An example of image processing]
Subsequently, a second embodiment according to the present application will be described. In the first embodiment described above, the image processing apparatus 100 performs a process of specifying each marker based on, for example, the relative size and mapping information of the two markers included in the image data after the projective transformation. explained. Alternatively, the process of identifying the marker by the image processing apparatus 100 based on the identification information included in the two-dimensional code, the coordinate information at four or more points, and the like has been described.

Here, the image processing apparatus 200 (see FIG. 12) according to the second embodiment acquires and acquires in advance the characteristic information that the marker has and that does not change even after the projective transformation. The marker may be identified based on the feature information. According to such processing, the image processing apparatus 200 can identify the markers, that is, uniquely identify the markers even if the image data does not include two or more markers. This point will be described with reference to FIG. FIG. 11 is a diagram showing an example of image processing according to the second embodiment. The image processing device 200 according to the second embodiment performs the same processing as the image processing device 100 according to the first embodiment except for the process of acquiring information about the marker in advance and the process of specifying the marker. ..

In FIG. 11, the image processing apparatus 200 shall acquire information regarding the circular markers M01, M02, M03, and M04 included in the sheet 70, as shown in FIG. The image processing device 200 is derived from the lengths of the diameters of the outer circle and the inner circle in each circle marker and the lengths of the diameters of the outer circle and the inner circle as information regarding the circle markers M01, M02, M03, and M04. Get the ratio. Further, the image processing apparatus 200 acquires the ratio of the ratio derived from the length of each diameter of the outer circle and the inner circle, that is, the cross ratio in the circle marker.

Here, the size of the circular marker M01 drawn on the sheet 70 and the circular marker M01 in the image data created by being imaged by the image pickup apparatus are different depending on the imaging situation even if the image data is projected and converted. It is expected to be different. However, even if the size of the circle marker itself changes, the value shown as the double ratio of the diameters of the outer circle and the inner circle remains unchanged even after the projective transformation. In the second embodiment, the image processing apparatus 200 utilizes this property to identify a circular marker.

In FIG. 11, the circle marker M01 will be described as an example. In the example shown in FIG. 11, the image processing apparatus 200 recognizes the circle marker M01 as two concentric circles of the inner circle 80 and the outer circle 85. In the example of FIG. 11, the center of the inner circle 80 and the outer circle 85 is set as the point C, the intersection of the straight line passing through the center and the inner circle 80 is the point B and the point D, and the intersection of the straight line passing through the center and the outer circle 85 is set. Let it be point A and point E. That is, the diameter of the inner circle 80 is shown as the line BD, and the diameter of the outer circle 85 is shown as the line AE. The radius of the inner circle 80 is shown as line BC or line CD, and the radius of the outer circle 85 is shown as line AC or line CE.

As described above, the image processing apparatus 200 utilizes the property that the cross ratio (ratio of length ratios) of line segments does not change in the projective space with respect to the identification of each circular marker. The image processing apparatus 200 may adopt any of the points A to E as long as the ratio of the ratios of the lengths of the two concentric circles can be obtained. For example, the image processing apparatus 200 utilizes the fact that the ratio of (line AC / line AD) and (line BC / line BD) is invariant. In other words, the image processing apparatus 200 utilizes the fact that the values of (line AC × line BD) / (line AD × line BC) are invariant.

In this case, the image processing device 200 acquires the sheet 70 as image data in advance, and acquires the length between each point of the circular marker M01 which is the target to be specified as the marker. As an example, the image processing apparatus 200 acquires a ratio of (line AC / line AD), (line BC / line BD), or the like as the ratio of the diameters of the inner circle 80 and the outer circle 85. Then, the image processing device 200 registers the value of the double ratio, which is the feature information calculated between the points A to E, in association with the marker ID, which is the identification information given to the circular marker M01. The image processing device 200 also registers the circular markers M02, M03, and M04 in the same manner. In this case, the circle markers M01, M02, M03, and M04 drawn on the sheet 70 have the image processing apparatus 200 at the intersection of the straight line passing through the center, the outer circle, and the inner circle, and the length between the points. It is assumed that there is a recognizable difference. That is, it is assumed that the image processing apparatus 200 acquires the double ratio indicated by the line segment between the intersection and the center as different values in each circular marker.

Then, when the image processing device 200 acquires the image data including any one of the circle markers M01, M02, M03, and M04, the image data so that the circle marker indicated by the ellipse can be recognized from the front. Is projected. Note that the projective transformation in this case does not require the projective transformation after calculating the camera feature information common to the plurality of images through the calibration process as described in the first embodiment. In one image data, it is only necessary to perform a projective transformation that replaces an ellipse with a perfect circle. As described in the first embodiment, when the marker is a perfect circle, the marker in the image data is represented by an ellipse. That is, when the marker is known to be a perfect circle, the image processing apparatus 200 may detect an ellipse in the image data and perform a projective transformation such that the detected ellipse is returned to a perfect circle.

Then, the image processing apparatus 200 collates the double ratio of the circle markers acquired in advance with the double ratio of the circle markers included in the projected image data, so that the circle markers included in the image data can be obtained. Circular marker M01, M02, M03 or M04 is specified as a circular marker. The image processing device 200 can accurately identify the marker included in each image data by performing the above processing on a plurality of image data received from the user, for example.

As described above, the image processing device 200 according to the second embodiment receives image data that is created by being imaged by a predetermined imaging device and includes image data including an elliptical figure. Then, the image processing device 200 projects the image data so that the elliptical figure included in the received image data becomes a perfect circle, and based on the feature information of the perfect circle after the projection conversion, the perfect circle. Is specified as a marker for use in a predetermined process of image data.

Specifically, the image processing apparatus 200 acquires information about a marker having an inner circle and an outer circle having a common center as markers, and as feature information, a straight line passing through the center of the marker, and the inner circle and the outer circle. Obtain a cross-ratio value that indicates the ratio of the ratios of the lengths of the circle to any of the intersections and centers of the circles. Then, the image processing device 200 collates the double ratio between the points corresponding to the acquired double ratio and the double ratio of the previously acquired marker in the marker obtained by projecting and transforming the image data. To identify the marker included in the image data.

As described above, the image processing apparatus 200 according to the second embodiment acquires the feature information that does not change even in the image data after the projective transformation with respect to the marker, and identifies the marker by collating the feature information. To do. As a result, the image processing apparatus 200 can uniquely identify the marker even when the original image data contains only one marker. In other words, the image processing apparatus 200 can identify the markers included in the image data without using the relative information of the two markers. As a result, the image processing device 200 can accurately specify whether the circle marker included in each image data is the circle marker M01 or the circle marker M02, so that the marker and the subject between a plurality of images can be accurately specified. It is possible to accurately grasp the positional relationship of. That is, the image processing device 200 can generate a free-viewpoint image that does not give a sense of discomfort even from a plurality of original image data that have not been accurately calibrated, such as those captured by a general user. Further, since the image processing device 200 can acquire information about the marker such as the sheet 70 in advance, it is not necessary to perform a process that burdens a general user in the marker identification process. As a result, the image processing device 200 can make a general user effectively utilize the free viewpoint image.

[2-2. Image processing device configuration]
Next, the configuration of the image processing apparatus 200 according to the second embodiment will be described with reference to FIG. FIG. 12 is a diagram showing a configuration example of the image processing device 200 according to the second embodiment. As shown in FIG. 12, the image processing device 200 includes a communication unit 210, a storage unit 220, and a control unit 230. The same processing as the processing performed by the image processing apparatus 100 described in the first embodiment and the processing unit that performs the same processing will not be described.

(About marker information storage unit 221)
The marker information storage unit 221 stores information about the marker. Here, FIG. 13 shows an example of the marker information storage unit 221 according to the second embodiment. In the example shown in FIG. 13, the marker information storage unit 221 has items such as “marker ID” and “feature information”.

The "marker ID" indicates identification information for identifying the marker. "Characteristic information" is information used for specifying a marker, and indicates information that does not change even after a projective transformation. In the example shown in FIG. 13, the feature information is conceptually described as "L01" or the like, but in reality, a numerical value or the like for specifying a marker is stored in the feature information item. .. As an example of the characteristic information, it is a value of a cross ratio derived based on the ratio of diameters of two concentric circles.

That is, in FIG. 13, it is shown that the marker identified by the marker ID “M01” has the information “L01” as the feature information.

(About control unit 230)
As shown in FIG. 12, the control unit 230 has an acquisition unit 231, a reception unit 232, a specific unit 233, a calibration unit 234, a generation unit 235, and a distribution unit 236, and the information described below. Realize or execute the function or action of processing. The internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 12, and may be another configuration as long as it is a configuration for performing information processing described later. Further, the connection relationship of each processing unit included in the control unit 230 is not limited to the connection relationship shown in FIG. 12, and may be another connection relationship.

(About acquisition unit 231)
The acquisition unit 231 acquires information about the marker. Specifically, the acquisition unit 231 acquires information about a marker including a circle as a component, and character information for specifying the marker. For example, the acquisition unit 231 acquires the image data of the circular marker M01 drawn on the sheet 70 provided to the user. Then, the acquisition unit 231 attaches a marker ID, which is identification information, to the circular marker M01 and stores it in the marker information storage unit 221.

Further, the acquisition unit 231 acquires information about a marker having an inner circle and an outer circle having a common center as markers, and as feature information, a straight line passing through the center of the marker and the inner circle and the outer circle. Obtain a cross-ratio that indicates the ratio of the lengths between each intersection and any of the points in the center. The acquisition unit 231 acquires the double ratio indicated by the line segment relating the two circles of the inner circle and the outer circle. In the example of FIG. 11, the acquisition unit 231 acquires the ratio between the line segment AD and the line segment AC and the ratio between the line segment BC and the line segment BD to acquire the cross ratio, while the line segment AB. And the line segment DE are not adopted as the line segment for obtaining the cross ratio.

(About reception desk 232)
The reception unit 232 receives image data that is created by being imaged by a predetermined imaging device and includes an elliptical figure. For example, the reception unit 232 according to the second embodiment acquires image data captured by placing the subject 60 on the sheet 70 and includes at least one of the circular markers M01 to M04. To do. Since the circular markers M01 to M04 are perfect circles, the imaged circular markers M01 to M04 are figures on an ellipse. That is, the image data received by the reception unit 232 includes the figure on the ellipse. The reception unit 232 stores the received image data in the image data storage unit 222.

Further, the reception unit 232 receives a plurality of image data having different viewpoint positions, and includes at least one figure specified as a marker in each image data. Similar to the first embodiment, the calibration unit 234 and the generation unit 235 generate a free viewpoint image based on the plurality of image data received by the reception unit 232.

In addition, the reception unit 232 accepts submission of a plurality of image data from the user terminal 10 and also receives information for identifying the user terminal 10. For example, the distribution unit 236 distributes the free viewpoint image generated by the generation unit 235 to the user terminal 10 based on the information that identifies the user terminal 10.

(About specific part 233)
The specific unit 233 projects the image data so that the elliptical figure included in the image data received by the reception unit 232 becomes a perfect circle, and based on the feature information of the perfect circle after the projection conversion, the true The circle is specified as a marker for use in a predetermined process of image data.

For example, the specific unit 233 is a marker included in the image data by collating the feature information of the perfect circle after the image data is projected and transformed by the acquisition unit 231 with the feature information acquired by the acquisition unit 231. To identify. Specifically, the specific unit 233 is a cross ratio acquired by the acquisition unit 231 when the two elliptical figures included in the image data are projected and transformed into two concentric circles. By collating the cross-ratio in two perfect circles calculated by using the length between the points in the corresponding relationship with the points used in calculating the cross-ratio, the image data is obtained. Identify the markers involved.

Here, the double ratio in two true circles calculated by using the length between the points used in calculating the double ratio and the points having a corresponding relationship is calculated by the circle marker M01. It means that the cross ratio is calculated for two concentric circles after the projective transformation by using the line segment having a corresponding relationship with the line segment used at the time of performing. For example, it is assumed that the acquisition unit 231 calculates the value of (line segment AC × line segment BD) / (line segment AD × line segment BC) as the cross ratio of the circular marker M01 in FIG. In this case, the specific portion 233 has a relationship corresponding to the case of the circle marker M01 even in the two concentric circles after the projection conversion ((line segment between the outer circle and the center) × (diameter of the inner circle)) / It means to obtain the cross ratio calculated by ((the intersection of the outer circle passing through the center and intersecting the inner circle) × (radius of the inner circle)).

(About calibration unit 234)
The calibration unit 234 calibrates a plurality of image data received by the reception unit 232 based on the parameters of the marker. Specifically, the calibration unit 234 according to the second embodiment is used as a transformation matrix for converting the submitted original image data into an image constituting the free viewpoint image by performing predetermined calibration. Calculate the camera parameters. As described above, according to the process according to the second embodiment, the proofreading unit 234 can accurately identify the marker included in each image data, so that the proofreading unit 234 serves as a corresponding point between the image data. It is possible to perform strong proofreading for each image data by using.

(About generator 235)
The generation unit 235 generates a free viewpoint image capable of continuously displaying an image corresponding to an arbitrary viewpoint position by performing a predetermined conversion process on a plurality of image data calibrated by the calibration unit 234.

(About distribution department 236)
The distribution unit 236 distributes the free viewpoint image generated by the generation unit 235. Specifically, the distribution unit 236 according to the second embodiment uses the free viewpoint image generated by the generation unit 235 and stored in the free viewpoint image storage unit 223 as the source of the original image data. Deliver to terminal 10.

[2-3. Modification example]
The image processing apparatus 200 according to the second embodiment described above may be implemented in various different forms other than the above-described embodiment. Therefore, another embodiment of the above-mentioned image processing apparatus 200 will be described below.

[2-3-1. Marker including design]
The image processing apparatus 200 may specify the marker by using the feature information different from the cross ratio calculated from the ratio of the diameter of 2 circles as the feature information. For example, the image processing device 200 may specify the marker based on the design drawn in the circular marker. This point will be described with reference to FIGS. 14 to 16.

FIG. 14 is a diagram (1) for explaining an example of the marker according to the second embodiment. In FIG. 14, a circular marker M10 will be shown and described as an example of the marker according to the second embodiment. As shown in FIG. 14, the circle marker M10 is a marker whose constituent elements are the inner circle 81, the outer circle 86, and the design 90. Design 90 is a design with the Chinese character "Shinobu" as a motif. It is assumed that the image processing device 200 acquires, for example, that the circle marker M10 is composed of a perfect circle, pixel data that constitutes the circle marker M10, and the like as information regarding the circle marker M10.

Subsequently, the state in which the circular marker M10 is imaged by the imaging device will be described with reference to FIG. FIG. 15 is a diagram (2) for explaining an example of the marker according to the second embodiment. As shown in FIG. 15, the circular marker M10 is represented as an elliptical figure in the image data captured by the image pickup apparatus. The image processing device 200 detects an elliptical figure from the image data, and projects and transforms the detected elliptical figure (step S30). As a result, the image processing apparatus 200 acquires the circular marker M10 represented by a perfect circle.

For example, when the image processing device 200 can recognize the design 90 "Shinobu" at the same angle as the acquired information about the circular marker M10, the image processing apparatus 200 can easily image the circular marker M10 by collating with pixel data or the like. Can be identified in the data. However, in the image data created by the image pickup apparatus, the design 90 related to the circular marker M10 is usually shown tilted in an oblique direction. Therefore, even when the elliptical circular marker M10 in the image data is projected and transformed, as shown in FIG. 15, the design 90 "Shinobu" is shown at an oblique angle. In this case, the image processing device 200 cannot identify the circle marker M10 simply by collating the pixel data. Further, rotating all the pixel data of the circular marker M10 acquired from the image data and detecting the angle to collate with the pixel data of the circular marker M10 acquired in advance increases the processing load.

Therefore, the image processing device 200 acquires characteristic information that does not change the rotation of the circular marker M10, and performs marker identification processing using such information. For example, the image processing apparatus 200 uses the feature information indicated by the brightness difference in the direction from the center of the circle to the outside in each pixel on the radiation extending from the center of the circle to the outside to display the marker. Identify.

Specifically, the image processing apparatus 200 provides a plurality of sample points at a location separated by a first predetermined distance from the center of the circle of the marker. Then, the image processing device 200 arranges as many numerical values as the number of sample points, which are obtained by binarizing the brightness difference between the sample points and the points separated from the sample points by a second predetermined distance outward. Obtain binary data, which is data, as feature information. This point will be described with reference to FIG.

FIG. 16 is a diagram (3) for explaining an example of the marker according to the second embodiment. As shown in FIG. 16, the image processing apparatus 200 provides a pixel at a position separated from the center of the circular marker M10 by a first predetermined distance as a sample point. In other words, the image processing apparatus 200 provides a sample point on the circumference of the circumference 91 having a radius of the first predetermined distance. In the example of FIG. 16, the image processing apparatus 200 provides sample points SA01 to SA06 on the circumference 91.

Then, the image processing apparatus 200 obtains the difference between the brightness of each of the sample points SA01 to SA06 and the brightness of each pixel at a position separated by a second predetermined distance on the radiation. That is, the image processing apparatus 200 obtains the brightness difference between the sample points SA01 to SA06 and the sample points SA11 to SA16 on the circumference 92 separated by a second predetermined distance.

The image processing device 200 acquires, for example, the luminance difference between points as a binary value of "0" and "1". For example, the image processing apparatus 200 gives a numerical value of "1" to the sample point SA11 when the brightness of the sample point SA11 is higher than the brightness of the sample point SA01. On the other hand, when the brightness of the sample point SA11 is lower than the brightness of the sample point SA01, the image processing device 200 gives the sample point SA11 a numerical value of “0”. Similarly, the image processing apparatus 200 gives a numerical value of "0" or "1" for the sample points SA12 to SA16.

Then, the image processing apparatus 200 arranges the numerical values given to the sample points SA11 to SA16 along the circumference 92. As a result, the image processing apparatus 200 can obtain binary data in which the numerical values at the sample points SA11 to SA16 on the circumference 92 are arranged. Specifically, the image processing apparatus 200 acquires a list of numerical values such as "00100" and "10100" as binary data on the circumference 92. The image processing apparatus 200 repeats this process, and obtains a brightness difference between the circumference 93 and the immediately preceding sample points SA11 to SA16 at the intersection with the line extending radially outward from the sample points SA01 to SA06. As a result, the image processing apparatus 200 acquires the binary data in the circular marker M01. The image processing apparatus 200 can be arbitrarily designed with respect to the number of sample points and the number of circumferences. For example, the image processing apparatus 200 may acquire the brightness difference between the sample point SA11 and the sample point SA21 which is a sample point on the circumference assumed to be further outside. As the image processing apparatus 200 increases the circumference and sample points for acquiring the binarized data, the accuracy of the marker identification processing improves.

As described above, the image processing apparatus 200 acquires binary data in advance as feature information for the circular marker M10. Further, as shown in FIG. 16, the image processing apparatus 200 also acquires binary data for the circular marker M10 after the projective transformation. Then, the image processing device 200 identifies the marker by obtaining the similarity between the binary data.

As described above, when the circle marker M10 is imaged, the direction of the circle marker M10 after the projective transformation is detected as a direction different from the circle marker M10 which is the correct answer data acquired in advance. Therefore, the image processing apparatus 200 may use binary data to create a histogram showing the characteristics of the marker.

The image processing apparatus 200 determines a bin based on the number of virtual circumferences on which sample points are provided in the histogram. Then, the image processing device 200 creates a histogram by increasing the frequency of the corresponding bin by 1 degree when the binarized data at the sample points on each circumference is “1”. Then, the image processing device 200 inputs the data at each circumference and each sample point, and then normalizes the total number of bins to 1.

The image processing apparatus 200 uses, for example, the following equation (2) to determine the degree of similarity between the histogram of the circular marker M10, which is the correct answer data acquired in advance, and the histogram of the circular marker M10 after the projective transformation.

In the above formula (2), "Similarity" indicates a numerical value of similarity. “Input _i ” indicates the frequency of the i-th bin in the circular marker M10 after the projective transformation. “Reference _i ” indicates the frequency of the i-th bin in the circle marker M10, which is the correct answer data. Further, "n" indicates the number of bins. “Min” indicates that the minimum value of any of the numerical values is taken.

As shown in the above equation (2), when the frequency of the i-th bin in the circle marker M10 after the projective transformation and the frequency of the i-th bin in the circle marker M10 which is the correct answer data match in all the bins. The similarity is "1", which is the maximum value. The minimum value that the similarity can take is "0".

In this way, the image processing apparatus 200 calculates the similarity of the markers based on the binary data of the circle marker M10 acquired in advance and the binary data of the circle marker M10 after the projective conversion. Then, the image processing device 200 identifies that the elliptical figure included in the image data is a figure indicating the circle marker M10 when the similarity exceeds a predetermined threshold value.

As described above, the image processing apparatus 200 can perform a process of specifying the marker by a method of comparing the brightness difference of the designs on the radiation based on the design inside the circle. According to such a method, the image processing apparatus 200 can specify a circular marker as rotation-invariant feature information. Further, in such a method, by determining the luminance difference, the accuracy can be improved as compared with simply comparing the pixel data of the markers. That is, it is conceivable that the brightness of the pixels of the marker may change depending on the camera parameters of the imaging device, the lighting condition at the time of imaging, and the like. Therefore, even if the numerical value of the marker brightness in the correct answer data and the numerical value of the marker brightness after the projective transformation are simply compared, the marker may not be identified accurately. On the other hand, in such a method, since the data obtained by binarizing the difference in brightness is used instead of the numerical value of the brightness itself, it is necessary to identify the marker with relatively high accuracy even if it is a marker in the image data after imaging. Can be done.

The image processing device 200 may employ the following method in order to further improve the specific accuracy of the marker. For example, when the image processing apparatus 200 binarizes the brightness difference between the sample point and the point separated from the sample point by a second predetermined distance outward, the image processing apparatus 200 is located around the pixel serving as the sample point. Binary data may be acquired by comparing the average luminance including the pixels with the average luminance including the pixels around the pixel as a point. That is, the image processing device 200 does not use the brightness of only the pixel (1 pixel) extracted as the sample point, but for example, the average value of the brightness of several pixels including the periphery of the pixel and the next circumference. The average value of the brightness for several pixels including the periphery of the sample point in is used for processing. As a result, the image processing apparatus 200 can suppress noise in the processing such as erroneously sampling a peculiar point in the image data, so that the accuracy of the specific processing can be improved.

Further, the image processing apparatus 200 may adopt the following method in order to further improve the specific accuracy of the marker. For example, in the image processing apparatus 200, after the similarity is calculated, only a predetermined angle is set along the assumed circle (for example, circumference 91) from the position of the sample point on which the similarity is calculated. A new sample point is set by shifting, and new binary data based on the new sample point is acquired. Further, the image processing apparatus 200 repeats obtaining the similarity between the binary data of the circular marker after the projective transformation and the acquired new binary data a predetermined number of times. Then, the image processing apparatus 200 may specify the marker included in the image data based on the maximum similarity among the obtained similarities.

In this way, the image processing apparatus 200 provides a sample point as a sample, performs a process of calculating the similarity, and then shifts the sample point by a predetermined angle (for example, 2 degrees) from the provided sample point. It is provided as a sample point, and processing is performed using a new sample point. As a result, the image processing apparatus 200 can suppress an error due to the distribution of the sample points provided, so that the accuracy of the specific processing can be improved.

Further, in the above, an example is shown in which the image processing apparatus 200 assumes a virtual circumference by using concepts such as a first predetermined distance and a second predetermined distance. Here, it is assumed that the size of the figure is different between the marker included in the image data and the marker acquired in advance. Therefore, the image processing device 200 may use a position where the diameter of the largest circle, which is a component of the marker, is equally divided by a predetermined ratio as the first predetermined distance or the second predetermined distance. For example, the image processing device 200 divides the diameter of the largest circle (outer circle 86 in the example of the circle marker M10 in FIG. 14), which is a component of the marker, into 10 equal parts, and virtually from the position closest to the center. You may assume the circumference. Further, when the image processing apparatus 200 specifies based on the above-mentioned design, the marker does not necessarily have two concentric circles as a component, and one circle is a component, and the design is drawn inside the marker. You may use a marker.

[2-3-2. Identifying markers]
In the second embodiment, the image processing apparatus 200 has described the process of identifying the marker in the image data by acquiring the rotation-invariant feature information of the circular marker in advance. Such a process is not necessarily used for generating a free viewpoint image, but may be simply used as a process for specifying a marker in the image data. That is, the image processing device 200 not only identifies the markers commonly included in the plurality of image data for the generation of the free viewpoint image, but also identifies the visual markers in one image data. The method may be adopted.

[2-4. effect〕
As described above, the image processing apparatus 200 according to the second embodiment has a reception unit 232 and a specific unit 233. The reception unit 232 receives image data that is created by being imaged by a predetermined imaging device and includes an elliptical figure. The specific unit 233 projects the image data so that the elliptical figure included in the image data received by the reception unit 232 becomes a perfect circle, and based on the feature information of the perfect circle after the projection conversion, the true The circle is specified as a marker for use in a predetermined process of image data.

As described above, the image processing apparatus 200 according to the second embodiment can uniquely identify the figure in the image data as a marker by using the invariant feature information even after the projective transformation. As a result, the image processing device 200 can generate a free-viewpoint image that does not give a sense of discomfort even from a plurality of original image data that have not been accurately calibrated, such as those captured by a general user. Can make effective use of free-viewpoint images.

Further, the image processing apparatus 200 according to the second embodiment further includes an acquisition unit 231 for acquiring information on a marker including a circle as a component and acquiring feature information for identifying the marker. The identification unit 233 identifies the marker included in the image data by collating the feature information of the perfect circle after the projective transformation with the feature information acquired by the acquisition unit 231.

In this way, the image processing apparatus 200 according to the second embodiment acquires information about the marker in advance, and identifies the marker by collating the acquired information with the feature information in the figure after the projective transformation. As a result, the image processing device 200 can accurately identify the figure used as the marker by the user.

Further, as feature information regarding a marker having an inner circle and an outer circle having a common center, the acquisition unit 231 includes an intersection of a straight line passing through the center of the marker, the inner circle and the outer circle, and any one of the centers. Obtain a cross-ratio that indicates the ratio of the length ratios between the points. The specific unit 233 sets the double ratio acquired by the acquisition unit 231 and the double ratio when the two elliptical figures included in the image data are projected and transformed so as to be two concentric circles. The marker included in the image data is specified by collating the cross ratio in two perfect circles calculated by using the length between the points used in the calculation and the length between the points having a corresponding relationship. ..

As described above, the image processing apparatus 200 according to the second embodiment uses the double ratio of the lengths of the two concentric circles as the invariant feature information in the projective space. As a result, the image processing device 200 can accurately identify the marker in the image data. Further, since the image processing device 200 can obtain information for identifying the marker only by changing the lengths of the diameters of the two concentric circles as the feature information of the marker acquired in advance, an identifiable marker can be obtained. It can be easily created.

Further, the acquisition unit 231 acquires the feature information indicated by the brightness difference between the two pixels existing on the radiation from the center of the circle to the outside as the feature information regarding the marker on which a predetermined figure is drawn inside the circle. To do. The identification unit 233 identifies the marker included in the image data by collating the feature information of the perfect circle after the projective transformation with the feature information acquired by the acquisition unit 231.

As described above, the image processing apparatus 200 according to the second embodiment does not use the brightness itself of the pixel of the marker but uses the brightness difference of the pixel on radiation in the specific processing. Therefore, the image processing device 200 can perform accurate specific processing that is not easily affected by the camera parameters of the image pickup device, the lighting state at the time of imaging, and the like.

Further, the acquisition unit 231 is provided with a plurality of sample points at a position separated by a first predetermined distance from the center of the circle of the marker, and is separated from the sample point by a second predetermined distance outward from the sample point. Binary data, which is data obtained by arranging the numerical values shown by binarizing the brightness difference between the points and the number of the sample points, is acquired as the feature information. The specific unit 233 calculates the binary data of the perfect circle after the projective conversion by executing the same process as the process of acquiring the binary data on the perfect circle after the projective conversion, and the true circle after the projective conversion. The marker included in the image data is specified by obtaining the degree of similarity between the binary data possessed by the data and the binary data acquired by the acquisition unit 231.

As described above, the image processing apparatus 200 according to the second embodiment identifies the marker by obtaining the similarity between the binary data of the marker which is the correct answer data and the binary data of the figure after the projective conversion. As a result, the image processing apparatus 200 can accurately identify the marker without performing a burdensome process such as comparing all the pixel data of the marker.

Further, when the acquisition unit 231 binarizes the brightness difference between the sample point and the point separated from the sample point by a second predetermined distance outward, the pixels around the pixel serving as the sample point are shown. Binary data is acquired by comparing the average luminance including the above and the average luminance including the pixels around the pixel as a point.

As described above, the image processing apparatus 200 according to the second embodiment can suppress noise in processing such as erroneously sampling a peculiar point in the image data, so that the accuracy of the specific processing can be improved. Can be done.

Further, after the similarity is calculated by the specific unit 233, the acquisition unit 231 shifts a new sample point by a predetermined angle along the circle from the position of the sample point on which the similarity is calculated. Set up and get new binary data based on new sample points. The specific unit 233 repeatedly obtains the similarity between the binary data of the perfect circle after the projective transformation and the new binary data acquired by the acquisition unit 231 a predetermined number of times, and determines the maximum of the obtained similarity. The markers contained in the image data are identified based on the similarity.

As described above, the image processing apparatus 200 according to the second embodiment may adopt a method of calculating the similarity several times while shifting the sample points. As a result, the image processing apparatus 200 can suppress an error due to the distribution of the sample points provided, so that the accuracy of the specific processing can be improved.

Further, the image processing apparatus 200 according to the second embodiment performs a predetermined conversion process on the proofreading unit 234 that calibrates the image data and the image data calibrated by the proofreading unit 234 based on the parameters of the marker. This further includes a generation unit 235 that generates a free viewpoint image capable of continuously displaying an image corresponding to an arbitrary viewpoint position. In this case, the reception unit 232 receives a plurality of image data having different viewpoint positions and including at least one figure specified as a marker in each image data. The calibration unit 234 calibrates a plurality of image data received by the reception unit 232. The generation unit 235 generates a free viewpoint image by performing a predetermined conversion process on a plurality of image data calibrated by the calibration unit 234.

As described above, the image processing apparatus 200 according to the second embodiment performs calibration processing using the specified marker to generate a free viewpoint image. As a result, the image processing device 200 can generate a free-viewpoint image that can be naturally displayed from an arbitrary viewpoint even from the original image data acquired by the image pickup device that has not been strongly calibrated. ..

In addition, the image processing device 200 according to the second embodiment further includes a distribution unit 236 that distributes the free viewpoint image generated by the generation unit 235. The reception unit 232 accepts the submission of a plurality of image data from the user terminal 10 and also receives the information for identifying the user terminal 10. The distribution unit 236 distributes the free viewpoint image to the user terminal 10 based on the information that identifies the user terminal 10 received by the reception unit 232.

As described above, the image processing device 200 according to the second embodiment distributes the generated free viewpoint image to the user terminal 10 used by the user. As a result, the image processing device 200 can promote the utilization of the free viewpoint image of a general user.

In addition, the generation unit 235 uses the generated free-viewpoint image as identification information for identifying the user terminal 10 or the user who uses the user terminal 10 that has submitted a plurality of image data that is the source of the free-viewpoint image, or the user. It is stored in a predetermined storage unit in association with at least one of the exhibition information in the service used by.

As described above, the image processing device 200 according to the second embodiment stores the free viewpoint image in the storage unit 220 together with the information for identifying the user. Then, the image processing device 200 may, for example, deliver a free viewpoint image to the service side in response to a request. As a result, the image processing device 200 can use the free viewpoint image in response to various requests, so that the user can use the free viewpoint image more effectively.

[3. Configuration change]
Further, among the processes described in each of the above-described embodiments, all or a part of the processes described as being automatically performed may be manually performed, or may be described as being performed manually. It is also possible to automatically perform all or part of the treatment by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. It can be integrated and configured. For example, the calibration unit 132 and the generation unit 133 shown in FIG. 2 may be integrated. Further, the image processing device 100 (including the image processing device 200; the same applies hereinafter) is distributed between a front-end server that mainly exchanges data with users such as users and a back-end server that performs generation processing and the like. It may be an embodiment. In this case, the front-end server has at least a reception unit 131 and a distribution unit 134. Further, the back-end server has at least a calibration unit 132 and a generation unit 133. Further, the image processing device 100 may be in a mode in which an external storage server is used instead of having the storage unit 120 inside.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[4. Hardware configuration]
Further, the image processing apparatus according to each of the above-described embodiments is realized by, for example, a computer 1000 having a configuration as shown in FIG. Hereinafter, the image processing apparatus 100 will be described as an example. FIG. 17 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the image processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, a program that depends on the hardware of the computer 1000, and the like.

The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from another device via the communication network 500 (corresponding to the network N of the embodiment) and sends the data to the CPU 1100, and the data generated by the CPU 1100 via the communication network 500 is transmitted to another device. Send to the device.

The CPU 1100 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Further, the CPU 1100 outputs the data generated via the input / output interface 1600 to the output device.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

For example, when the computer 1000 functions as the image processing device 100, the CPU 1100 of the computer 1000 realizes the function of the control unit 130 by executing the program loaded on the RAM 1200. Further, each data in the storage unit 120 is stored in the HDD 1400. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be acquired from another device via the communication network 500.

[5. Others]
Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other improved forms.

Further, the image processing device 100 described above may be realized by a plurality of server computers, and depending on the function, it may be realized by calling an external platform or the like by API (Application Programming Interface) or network computing. Can be changed flexibly.

Further, the "means" described in the claims can be read as "section, module, unit", "circuit" and the like. For example, the generation means can be read as a generation unit or a generation circuit.

1 Image processing system 10 User terminal 100 Image processing device 110 Communication unit 120 Storage unit 121 Image data storage unit 122 Free viewpoint image storage unit 130 Control unit 131 Reception unit 132 Calibration unit 133 Generation unit 134 Distribution unit 200 Image processing unit 210 Communication unit 220 Storage unit 221 Marker information storage unit 222 Image data storage unit 223 Free viewpoint image storage unit 230 Control unit 231 Acquisition unit 232 Reception unit 233 Specific unit 234 Calibration unit 235 Generation unit 236 Distribution unit

Claims (10)

Information about a marker that includes a circle as a component and has a predetermined figure drawn inside the circle, and binarizes the difference in brightness between two pixels existing on the radiation from the center of the circle to the outside. An acquisition unit that acquires the information indicated by the indicated numerical values as the feature information of the marker, and
An image data created by being imaged by a predetermined imaging device, and a reception unit that accepts image data including an elliptical figure.
The image data is projected and transformed so that the elliptical figure included in the image data received by the receiving unit becomes a perfect circle, and the feature information of the perfect circle after the projecting conversion and the features acquired by the acquisition unit By collating the information and obtaining the similarity between the feature information, when performing a predetermined process based on one marker on the image data, the perfect circle can be used as a marker for the predetermined process. Specific part to identify and
An image processing device characterized by being equipped with. The acquisition unit
As characteristic information about a marker having an inner circle and an outer circle having a common center, the length between the intersection of the straight line passing through the center of the marker, the inner circle and the outer circle, and any point of the center. Obtain a cross-ratio that indicates the ratio of
The specific part is
When the two elliptical figures included in the image data are projected and transformed into two concentric circles, the cross ratio acquired by the acquisition unit and the cross ratio are calculated. By collating the cross-ratio in the two perfect circles calculated by using the length between the points used and the length between the points having a corresponding relationship, the marker included in the image data is specified.
The image processing apparatus according to claim 1. The acquisition unit
A plurality of sample points are provided at a position separated by a first predetermined distance from the center of the circle of the marker, and the sample point and a point separated from the sample point by a second predetermined distance outward. Binary data, which is data obtained by arranging the numerical values shown by binarizing the brightness difference by the number of the sample points, is acquired as the feature information.
The specific part is
By executing the same process as the process of acquiring the binary data on the perfect circle after the projection conversion, the binary data of the perfect circle after the projection conversion is calculated, and the perfect circle after the projection conversion has. By obtaining the similarity between the binary data and the binary data acquired by the acquisition unit, the marker included in the image data is specified.
The image processing apparatus according to claim 1 or 2. The acquisition unit
When the brightness difference between the sample point and the point separated from the sample point by a second predetermined distance outward is shown as a binary, the average brightness including the pixels around the pixel serving as the sample point. And the average brightness including the pixels around the pixel at the point are compared to obtain the binary data.
The image processing apparatus according to claim 3. The acquisition unit
After the similarity is calculated by the specific unit, a new sample point is provided by shifting the position of the sample point from which the similarity is calculated by a predetermined angle along the circle. Get new binary data based on sample points,
The specific part is
The similarity between the binary data of the perfect circle after the projective transformation and the new binary data acquired by the acquisition unit is repeatedly obtained a predetermined number of times to obtain the maximum similarity among the obtained similarity. Identify the markers contained in the image data based on
The image processing apparatus according to claim 3 or 4. A proofreading unit that calibrates image data based on the parameters of the marker,
A generation unit that generates a free viewpoint image capable of continuously displaying an image corresponding to an arbitrary viewpoint position by performing a predetermined conversion process on the image data calibrated by the calibration unit.
With more
The reception department
A plurality of image data having different viewpoint positions and having at least one figure specified as the marker included in each image data are accepted.
The calibration unit
Proofreading the multiple image data received by the reception section
The generator
The free viewpoint image is generated by performing a predetermined conversion process on a plurality of image data calibrated by the calibration unit.
The image processing apparatus according to any one of claims 1 to 5. A distribution unit that distributes a free-viewpoint image generated by the generation unit,
With more
The reception department
In addition to accepting submissions of the plurality of image data from the terminal device, it also accepts information that identifies the terminal device.
The distribution unit
Based on the information that identifies the terminal device received by the reception unit, the free viewpoint image is distributed to the terminal device.
The image processing apparatus according to claim 6. The generator
The generated free-viewpoint image is exhibited in a terminal device that has submitted a plurality of image data that are the source of the free-viewpoint image, identification information that identifies a user who uses the terminal device, or a service used by the user. Stored in a predetermined storage unit in association with at least one of the information.
The image processing apparatus according to claim 6 or 7. An image processing method performed by a computer
Information about a marker that includes a circle as a component and has a predetermined figure drawn inside the circle, and binarizes the difference in brightness between two pixels existing on the radiation from the center of the circle to the outside. The acquisition process of acquiring the information indicated by the indicated numerical values as the characteristic information of the marker, and
An image data created by being imaged by a predetermined imaging device, and a reception process for receiving image data including an elliptical figure,
The image data is projected and transformed so that the elliptical figure included in the image data received by the receiving step becomes a perfect circle, and the feature information of the perfect circle after the projecting conversion and the features acquired by the acquisition step. By collating the information to obtain the similarity between the feature information, when performing a predetermined process based on one marker on the image data, the perfect circle can be used as a marker for the predetermined process. Specific process to identify and
An image processing method characterized by including. Information about a marker that includes a circle as a component and has a predetermined figure drawn inside the circle, and binarizes the difference in brightness between two pixels existing on the radiation from the center of the circle to the outside. The acquisition procedure for acquiring the information indicated by the indicated numerical values as the characteristic information of the marker, and
The reception procedure for accepting image data including an elliptical figure, which is image data created by being imaged by a predetermined imaging device, and
The image data is projected and converted so that the elliptical figure included in the image data received by the reception procedure becomes a perfect circle, and the feature information of the perfect circle after the projection conversion and the features acquired by the acquisition procedure. By collating the information and obtaining the similarity between the feature information, when performing a predetermined process based on one marker on the image data, the perfect circle can be used as a marker for the predetermined process. Specific steps to identify and
An image processing program characterized by having a computer execute.

Images