JP2020091718A - Photographed image classification device and photographed image classification method - Google Patents

Abstract

To provide a photographed image classification device capable of associating, with each target object, and classifying each piece of photographed image data photographed while changing the target object and an imaging location, when photographing the photographed image data in a multi-viewpoint image to be used for restoring a three-dimensional shape model of the target object.SOLUTION: A photographed image classification device associates, with each target object, and classifies a multi-viewpoint image for restoring a three-dimensional shape model of each target object. The photographed image data classification device includes a photographed image classification part configured to classify, corresponding to metadata added to photographed image data and related to a photographing condition when photographing the photographed image data, each piece of the photographed image data to an image data set photographing each target object.SELECTED DRAWING: Figure 1

Description

The present invention relates to a captured image classification device and a captured image classification method for classifying multi-view images used when restoring a three-dimensional shape model of a target object.

3D reconstruction technology based on multi-view images has been attracting attention not only in computer vision research communities but also in a wide range of fields such as digital archiving of cultural assets and entertainment industry.

The three-dimensional restoration technique based on the multi-viewpoint image described above uses a triangulation technique, and finally, the three-dimensional coordinate point group is generated by integrating the depth maps at each viewpoint.
Therefore, when a three-dimensional shape model of a target object in the world coordinate system is restored using images at each viewpoint (viewpoint image that is captured image data), three or more points in the world coordinate system between two or more different viewpoint images are restored. The process of detecting the corresponding pixel of each viewpoint image corresponding to the coordinate point of the dimensional coordinate point is performed.

Therefore, in order to accurately restore the three-dimensional shape model, it is necessary to move the imaging position of the imaging device and image the entire surface of the target object.
Therefore, the number of viewpoint images in which the user images the target object becomes very large, and when restoring the three-dimensional shape models of the plurality of target objects, it takes time to classify the multi-viewpoint images of each target object. It will be.
Therefore, the target objects are arranged and imaged as an image sequence so that it is not necessary to perform classification (see, for example, Patent Documents 1 and 2).

JP 05-35854 A JP, 04-62676, A

However, in the above-mentioned Patent Documents 1 and 2, the movement of the image pickup device is known, and it is premised that adjacent image pickups are performed, and the input of the image pickup position indicating that the adjacent image pickups have been performed. Is required.
In addition, when there is no boundary information such as when the target objects are different and the image capturing positions for capturing the target objects are not adjacent to each other, it is impossible to determine whether the images are multi-viewpoint images of the same target object. In this case, the user confirms the picked-up images one by one and classifies the picked-up images for each target object, which imposes a sorting load on the user.

The present invention has been made in view of such a situation, and when capturing imaged image data in a multi-viewpoint image used to restore a three-dimensional shape model of a target object, the captured image is captured while changing the target object and the imaging position. Provided are a captured image classification device and a captured image classification method capable of classifying each piece of image data in association with each target object.

In order to solve the above-mentioned problem, a captured image classification apparatus of the present invention is a captured image classification apparatus for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects. In addition, in correspondence with the metadata added to the captured image data regarding the capturing conditions when capturing the captured image data, each of the captured image data is an image data set obtained by capturing each of the target objects. It is characterized by including a captured image data classification unit for classifying into.

In the captured image classification apparatus of the present invention, the metadata is each of resolution, aperture value, focal length, shutter speed, ISO (International Organization for Standardization) sensitivity, image capturing date and time, and GPS (Global Positioning System) information. The image data classifying section classifies each of the captured image data by at least one of the metadata.

In the captured image classification device of the present invention, when performing classification using any of the metadata, the captured image data classification unit sorts each of the captured image data according to a numerical value of the metadata, It is characterized in that each of the captured image data is classified corresponding to the target object at a portion where a numerical distance of a parameter between the captured image data exceeds a predetermined separation distance.

In the captured image classification apparatus of the present invention, the metadata is each of resolution, aperture value, focal length, shutter speed, ISO (International Organization for Standardization) sensitivity, image capturing date and time, and GPS (Global Positioning System) information. The image data classifying unit generates a multidimensional vector for each of the captured image data based on a plurality of parameters in the metadata, and classifies each of the captured image data according to a similarity between the multidimensional vectors. ..

The captured image classification apparatus of the present invention is a captured image classification apparatus that classifies a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects. To each of the captured image data that has been captured, an identification information assigning unit that assigns identification information that identifies the target object, and the identification information that is assigned to each of the captured image data. And a captured image data classifying unit that classifies each of the target objects into an image data set in which each of the target objects is captured.

The captured image classification device of the present invention further includes an image capturing start detection unit that detects a time point when the image capturing of the target object is started, and the identification information assigning unit determines a time point when the image capturing start detection unit starts image capturing. When detected, the same identification information is added to each of the captured image data.

The captured image classification apparatus of the present invention is characterized by further comprising an image capturing end detection unit that detects a time point when the image capturing of the target object is completed.

The picked-up image classification method of the present invention is a picked-up image classification method for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects. , Each of the captured image data is classified into an image data set in which each of the target objects is captured, corresponding to the metadata added to the captured image data and related to the shooting condition when the captured image data is captured. It is characterized by including the process of doing.

The captured image classification method of the present invention is a captured image classification method for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects. An identification information assigning process of assigning identification information for identifying the target object to each of the captured image data obtained by capturing each of the target objects, and a captured image data classification unit is assigned to each of the captured image data. A captured image data classification process of classifying each of the captured image data into an image data set of each of the target objects, corresponding to the identification information.

As described above, according to the present invention, when the captured image data in the multi-viewpoint image used to restore the three-dimensional shape model of the target object is captured, each of the captured image data captured while changing the target object and the capturing position. It is possible to provide a captured image classification device and a captured image classification method capable of classifying the images according to respective target objects.

It is a block diagram which shows the structural example of the imaging system provided with the captured image classification apparatus by the 1st Embodiment of this invention. It is a figure explaining classification|category to the data set of the captured image data which used the imaging date and time as a parameter. FIG. 9 is a diagram showing an example of a user control input screen displayed on the display screen of the image display unit 300 by the image display control unit 104. 6 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification apparatus 100 according to the first embodiment of the present invention. 9 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification device 100 according to the second embodiment of the present invention. It is a block diagram which shows the structural example of the imaging system provided with the captured image classification apparatus by the 3rd Embodiment of this invention. It is a figure explaining GUI (Graphical User Interface) in classification processing of the picked-up image data of picked-up image classification device 100A. 6 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification apparatus 100A according to the present embodiment.

<First Embodiment>
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an imaging system including a captured image classification device according to the first embodiment of the present invention. The captured image classification apparatus 100 classifies, as a multi-view image used to restore a three-dimensional shape model, captured image data obtained by capturing a plurality of target objects at different image capturing positions in association with each target object. It is a device.

In FIG. 1, the imaging system 1 includes a captured image classification device 100, an imaging unit 200, and an image display unit 300. The imaging system 1 is, for example, a smartphone, installs an application of the function of the captured image classification device 100, and supports capturing of a multi-viewpoint image described below. In the present embodiment, as an example, a configuration that supports the capturing of a still image as captured image data of a multi-viewpoint image will be described, but the process itself is the same for a moving image.
The three-dimensional shape model restoration device (server) 2 uses the data set of the captured image data for each target object supplied from the imaging system 1 to, for example, use the SfM (Structure from Motion) algorithm to determine the corresponding target object 3 Restore the 3D model.

The captured image classification device 100 is a device that supports capturing of multi-viewpoint images, and includes an image data input unit 101, a captured image data classification unit 102, an imaging end detection unit 103, an image display control unit 104, a user edit input unit 105, Each of the restoration start notification unit 106 and the storage unit 107 is provided.
The image capturing section 200 is, for example, a camera function section having an image capturing element incorporated in a smartphone.
The image display unit 300 is a display device provided in the smartphone and includes, for example, a liquid crystal display.

The image data input unit 101 sequentially inputs captured image data captured by the image capturing unit 200, writes the captured image data in the storage unit 107, and stores the image data. The captured image data includes an image of the target object that is the target for restoring the three-dimensional shape model.

The picked-up image data classification unit 102 sets each of the picked-up image data obtained by picking up a plurality of target objects, which are randomly written and stored in the storage unit 107, as an image data set (corresponding to the picked-up image data Group). Here, when the captured image data classification unit 102 classifies the captured image data, the resolution (the number of pixels in the vertical direction (short side), the number of pixels in the horizontal direction (long side), the number of pixels in the vertical direction×the number of pixels in the horizontal direction= (Total number of pixels), aperture value, focal length, shutter speed, ISO (International Organization for Standardization) sensitivity, imaging date and time, and Exif (Exchangeable) which is metadata added to captured image data such as GPS (Global Positioning System) information. Each of the captured image data is classified into the corresponding image data set of the target object using the embedded information of the (image file format).

In the present embodiment, when the captured image data of the target object is captured, each of the resolution, aperture value, focal length, shutter speed, and ISO sensitivity is usually fixed in the metadata, and the date and time when the image is captured in the metadata. The fact that the numerical values of GPS and GPS do not change significantly is used for classification of captured image data into image data sets.

FIG. 2 is a diagram for explaining classification of captured image data into a data set using the capturing date and time as a parameter.
FIG. 2A shows, as an example of classification, a data table showing the respective time differences between the image capture dates and times of the captured image data of the image data sets #1, #2, and #3. The image data set #1 is a group of 7 pieces of captured image data, the image data set #2 is a group of 8 pieces of captured image data, and the image data set #3 is a group of 10 pieces of captured image data.

FIG. 2B shows a histogram created for each difference time of captured image data and each difference time between image data sets in the data table shown in FIG. 2A. In the histogram of FIG. 2B, the horizontal axis represents the difference time, and the vertical axis represents the number (frequency) of captured image data included in the predetermined time difference range and the difference time of the image capturing intervals between the image data sets.

As can be seen from this histogram, two frequency peaks are generated across a valley with a frequency of 0, and one peak is included in the range of up to about 30 seconds in the imaging date/time difference between the captured image data. However, the other mountain becomes an outlier from the one mountain, indicating the time difference between the image capture date and time of the captured image data between the image data sets, and it may take a few minutes due to movement to change the target object to be imaged or preparation for image capture. It is included in the range of. Therefore, for example, if the difference time is up to 1 minute and 30 seconds, it can be estimated that the same target object is being imaged. It can be used as a delimiter of the image data set indicating that the target object to be changed is changed. Further, when three or more peaks are formed, the time difference between the leftmost valleys (time difference between the center of the valleys) is used as a delimiter.

Returning to FIG. 1, the captured image data classification unit 102 sorts the captured image data written and stored in the storage unit 107 in order of the date and time of image capturing, and calculates the time difference between the respective image capturing times. Further, the captured image data classification unit 102 creates a histogram of the difference time, and extracts the difference time of the above-mentioned division as the classification difference time.
Then, the captured image data classification unit 102 extracts a combination of captured image data in which the time difference of the sorted image capturing dates and times is the classification time difference or more, and sets the captured image data as a delimiter of the image data set.

That is, in the data table of the captured image data shown in FIG. 2A, the time difference between the captured image data of the captured image data in each image data set is 30 seconds or less. On the other hand, the time difference between the last captured image data of image data set #1 and the first captured image data of image data set #2 is 270 seconds, and the captured image data in the image data set is 270 seconds. It shows that it can be used as a large break compared to the time difference between them.

As a result, the captured image data classification unit 102 detects the boundaries of the captured image data having imaging dates and times that are different from each other by the classification difference time and extracts each of the captured image data between the boundaries as an image data set. To do.
Then, the captured image data classification unit 102 writes and stores the captured image data extracted by the delimiters as a predetermined image data set provided for each target object, and stores each of the captured image data written as the image data set. There is a group for each type of target object.

The image capturing end detection unit 103 detects information indicating the image capturing end input by the user by an input unit (not shown) provided in the image capturing system 1, and outputs an image capturing end signal to the image data input unit 101 and the captured image data classification unit 102. Output to.
Accordingly, the captured image data classification unit 102 starts classification of the captured image data in the storage unit 107 into image data sets.
Here, the detection of the end of imaging may be detected by detecting that the user has pressed the end button displayed on the image display unit 300, or by shaking the imaging system 1 by the user and detecting it by the acceleration sensor.

The image display control unit 104 displays a user control input screen on the display screen of the image display unit 300.
The user edit input unit 105 performs processing on the image data set and the captured image data corresponding to the control information input by the user operating the user control input screen.

FIG. 3 is a diagram showing an example of a user control input screen displayed on the display screen of the image display unit 300 by the image display control unit 104.
FIG. 3A shows an image of a data set table showing a list of data sets stored in the storage unit 107.
This data set table has columns for each record, that is, the data set number of the image data set corresponding to the target object, the data set name, image reading, restoration execution, and deletion.

Here, the data set number is, for example, a number serially assigned in the order in which images are taken. The data set name is, for example, a name by which the imaged target object can be identified.
On the other hand, the fields of image reading, restoration execution and deletion are fields in which the user inputs control information. In the image reading column, by selecting the icon 501 in this column, a list of captured image data in the corresponding image data set is displayed as shown in FIG. 3B.
In FIG. 3B, each of the picked-up image data of the multi-viewpoint image of the data set name “Monument #1” stored in the storage unit 107 is thumbnailed and arranged and displayed.

When the image reading icon 501 in the same record as the data set name “Monument #1” is selected, the user edit input unit 105 displays the captured image data of this image data set on the display screen of the image display unit 300. A control signal to be output is output to the image display control unit 104.
As a result, the image display control unit 104 sequentially reads the captured image data of the data set name “Monument #1” in the storage unit 107, converts it into thumbnails, and displays the display screen of the image display unit 300 as illustrated in FIG. 3B. To display.

In addition, when the restoration execution icon 502 in the same record as the data set name “Monument #1” is selected, the user edit input unit 105 sets the captured image data of this image data set as a multi-viewpoint image in three dimensions. A control signal for restoring the shape model is output to the restoration start notification unit 106.
As a result, the restoration start notification unit 106 outputs to the 3D geometric model restoration device 2 a control signal for restoring the 3D geometric model together with the data set name “Monument #1”.

When the delete icon 503 in the same record as the data set name “Monument #1” is selected, the user edit input unit 105 deletes the captured image data of this image data set.
As a result, the user edit input unit 105 deletes all the captured image data in this image data set together with the data set name “Monument #1” in the storage unit 107.

FIG. 4 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification apparatus 100 according to the present embodiment. The metadata used to classify the captured image data will be described using the date and time when the captured image data was captured.
Step S101: The image data input unit 101 sequentially inputs the imaged image data of the target object imaged by the imaging unit 200 of the imaging system 2 by the user, and writes and stores the imaged data in the storage unit 107. At this time, the user continuously images a plurality of target objects in time series. Therefore, the storage unit 107 randomly stores each of the captured image data corresponding to the plurality of target objects. In addition, Exif information is embedded as metadata in each of the captured image data.

Step S102: The image capturing end detection unit 103 determines whether or not the user has performed an operation to terminate the image capturing process on a predetermined input unit.
At this time, when the user performs an operation to end the image capturing process, the image capturing end detecting unit 103 outputs an image capturing end signal to the captured image data classifying unit 102, and advances the process to step S103.
On the other hand, when the user does not perform the operation of ending the image capturing process, the image capturing end detecting unit 103 advances the process to step S101.

Step S103: The captured image data classification unit 102 sequentially reads the captured date and time from the storage unit 107 as metadata of each of the captured image data, one by one.

Step S104: The captured image data classification unit 102 sorts the captured image dates of the read captured image data in time series.

Step S105: The captured image data classification unit 102 determines whether or not the sorting of the captured date and time of the metadata in time series is completed for all the captured image data in the storage unit 107.
At this time, the captured image data classification unit 102 advances the process to step S106 when the sorting of the captured date and time of all the captured image data in the storage unit 107 is completed.
On the other hand, the captured image data classification unit 102 advances the processing to step S103 when the sorting of the captured date and time of all the captured image data in the storage unit 107 is not completed.

Step S106: The captured image data classification unit 102 calculates a difference time which is a time difference between the captured date and time of the captured image data sorted in the order of captured date and time and the captured date and time of the captured image data immediately after. Here, the captured image data classification unit 102 calculates the time difference between the captured date and time of the captured image data of all adjacent combinations that are sorted and arranged.
Then, the captured image data classification unit 102 creates a histogram of the difference time and extracts the classification difference time from the histogram as described above.

Step S107: The captured image data classification unit 102 divides the captured image date and time based on the extracted classification difference time, and the captured image data of the storage unit 107 for each target object based on the divided image capture date and time. Classify into image datasets.

Further, in the present embodiment, the image capture date and time of the captured image data is used as the metadata, but similarly in the GPS, the captured image data classification unit 102 sorts the position of the position information in the order of latitude and longitude, and obtains at that time. The distance that is the difference between the latitude and longitude coordinates in the position information for each captured image data is extracted. Then, the picked-up image data classification unit 102 performs the same process as the above-described time difference for this distance, obtains the classified distance, and performs the classified process of the picked-up image data using the GPS position information. Further, the position information of each captured image data may be plotted on a two-dimensional space composed of latitude and longitude, and the classification processing may be performed by cluster analysis.

When the resolution, aperture value, focal length, shutter speed, and ISO sensitivity are used as the metadata for classification, the captured image data classification unit 102 creates a histogram of each metadata. That is, the captured image data classification unit 102 creates a histogram for each type of metadata with the horizontal axis representing each numerical value range of the metadata and the vertical axis representing the number of captured image data falling within this numerical value range.
Then, the captured image data classification unit 102 selects metadata in which the same number of mountains as the number of target objects is formed, and classifies each of the captured image data based on the numerical range of the mountains in this histogram.

As described above, according to the present embodiment, when the same target object is imaged using the Exif metadata, the captured image data classification unit uses the fact that the numerical values of the metadata in this data set are close to each other. By obtaining the difference in metadata by 102 and grouping the captured image data having a small difference between the metadata arranged in the sort order, the captured image data in the multi-viewpoint image used to restore the three-dimensional shape model of the target object can be obtained. When picked up, each of the picked-up image data picked up while changing the target object and the picked-up position can be easily classified corresponding to each target object.

<Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. The captured image classification apparatus according to the second embodiment has the same configuration as the captured image classification apparatus according to the first embodiment of the present invention. Only the operation different from that of the first embodiment will be described below.
When the captured image data classification unit 102 classifies the captured image data, a combination of two or more of each of resolution, aperture value, focal length, shutter speed, ISO sensitivity, image capturing date and GPS information, or all multidimensional. A vector is generated in the evaluation vector space for each captured image data.

Then, the captured image data classification unit 102 calculates the degree of similarity between each multidimensional vector of the captured image data and the multidimensional vector of different captured image data. As a method of calculating the similarity, a method based on a distance such as Euclidean distance or a method based on an inner product of vectors such as cosine similarity can be appropriately used. Hereinafter, in the present embodiment, a case of using the cosine similarity will be described.
At this time, the captured image data classification unit 102 classifies captured image data having a multidimensional vector whose cosine similarity is equal to or higher than a preset similarity threshold as the same image data set.

FIG. 5 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification apparatus 100 according to the present embodiment. The metadata used to classify the captured image data will be described as all the Exif metadata added to the captured image data.
Further, since steps S101 to S102 are the same as those in the first embodiment, the description thereof will be omitted.

Step S203: The captured image data classification unit 102 sequentially obtains the Exif information (resolution, aperture value, focal length, shutter speed, ISO sensitivity, image capture date and GPS information) embedded in the captured image data from the storage unit 107. read out.

Step S204: The captured image data classification unit 102 generates a multidimensional vector for each captured image data from the numerical values of the resolution, aperture value, focal length, shutter speed, ISO sensitivity, image capturing date and GPS information.

Step S205: The captured image data classification unit 102 determines whether or not the reading of the Exif information from all the captured image data in the storage unit 107 and the generation of the multidimensional vector of the read Exif information are completed. To do.
At this time, when the captured image data classification unit 102 finishes reading the Exif information of all the captured image data in the storage unit 107 and generating the multidimensional vector of the read Exif information, the process proceeds to step S206. Proceed.
On the other hand, if the reading of the Exif information of all the captured image data in the storage unit 107 and the generation of the multidimensional vector of the read Exif information have not been completed, the captured image data classification unit 102 performs processing in step S203. Proceed to.

Step S206: The picked-up image data classification unit 102 generates each pair of picked-up image data such that all picked-up image data are combined without omission.
Then, the captured image data classification unit 102 calculates the cosine similarity between the multidimensional vectors of the Exif information of the captured image data for all pairs of combinations.

Step S207: The captured image data classification unit 102 classifies each captured image data of a combination of multidimensional vectors whose cosine similarity is equal to or higher than a preset similarity threshold as the same image data set.
Then, the captured image data classification unit 102 writes and stores each of the classified captured image data as a data set for each classification in the storage unit 107, for example.

As described above, according to the present embodiment, when the same target object is imaged using the Exif metadata, the captured image data classification unit uses the fact that the numerical values of the metadata in this data set are close to each other. In 102, a multidimensional vector of the metadata of the captured image data is calculated, a pair of captured image data is created so that all the captured image data in the storage unit 107 are combined, and a cosine similarity is obtained for each pair, Imaging in a multi-viewpoint image used for reconstruction of a three-dimensional shape model of a target object by grouping captured image data of multidimensional vectors of combinations that are equal to or greater than a preset similarity threshold value as a data set of the same imaging target. When the image data is captured, each of the captured image data captured while changing the target object and the capturing position can be easily classified corresponding to each target object.

<Third Embodiment>
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. FIG. 6 is a block diagram showing a configuration example of an image pickup system including a picked-up image classification device according to the third embodiment of the present invention. The captured image classification apparatus 100A classifies, as a multi-viewpoint image used to restore a three-dimensional shape model, captured image data obtained by capturing a plurality of target objects at different image capturing positions in association with each target object. It is a device.

In FIG. 6, the imaging system 1A includes a captured image classification device 100A, an imaging unit 200, and an image display unit 300. The imaging system 1A is, for example, a smartphone as in the first embodiment, installs an application having the function of the captured image classification apparatus 100A, and supports the capturing of multi-view images described below. Further, in the present embodiment, as in the first embodiment, a configuration that supports capturing of a still image as captured image data of a multi-viewpoint image will be described, but the processing itself is the same for a moving image. The same components as those in the first embodiment are designated by the same reference numerals.

The captured image data classification unit 102A classifies each of the captured image data into the data set for each target object based on the set identification information added to the captured image data when input.
The image capturing start detecting unit 108 detects information indicating the image capturing start input by the user by an input unit (not shown) provided in the image capturing system 1A, and outputs an image capturing start signal to the image data input unit 101 and the identification information adding unit. It outputs to 109.
The image capturing end detecting unit 103 detects information indicating the image capturing end input by the user by an input unit (not shown) included in the image capturing system 1, and outputs an image capturing end signal to the image data input unit 101, the captured image data classifying unit 102A, and It is output to the identification information addition unit 109.

FIG. 7 is a diagram illustrating a GUI in the image capturing and classifying process of the captured image classifying apparatus 100A.
FIG. 7A shows a target object displayed on the display screen 300A of the image display unit 300 by pressing the Start (imaging start) button 550 (input means for inputting to start imaging of captured image data). FIG. 8 is a diagram showing that image pickup of imaged image data of image 800 of FIG. That is, when the Start button 550 is pressed, the imaging start detection unit 108 detects the imaging start and outputs an imaging start signal to the image data input unit 101 and the identification information addition unit 109.

FIG. 7B shows that when the Start button 550 is pressed, a Finish (imaging end) button 552 is newly displayed at the position of the Start button 550. Also, it is indicated that by pressing the image capture button 551, an image (preview image) 800 of the target object displayed on the display screen 300A is captured as captured image data. That is, when the image capturing button 551 is pressed, the image capturing unit 200 captures an image of the target object displayed on the display screen 300A and supplies it to the image data input unit 101.

Further, when the Preview button 553 is pressed, the image display control unit 104 displays the user control input screen on the display screen 300A as described in FIG. 3, or the target object imaged after the imaging is started. A thumbnail image of the captured image data of is displayed on the display screen 300A.
Further, as illustrated in FIG. 7B, the image display control unit 104 displays the number of captured image data captured since the start of image capturing on the display screen 300A as a character string image 554 indicating the number of captured images. To do.

In FIG. 7C, the image 800 of the target object displayed on the display screen 300A of the image display unit 300 is displayed by pressing the Finish button 552 (input means for inputting to finish capturing the captured image data). It is a figure which shows that imaging of captured image data is complete|finished. That is, when the Finish button 552 is pressed, the image capturing end detection unit 103 detects the image capturing end and outputs an image capturing end signal to the image data input unit 101 and the identification information addition unit 109.

When the image capture start signal is supplied from the image capture start detection unit 108, the identification information addition unit 109 adds the captured image data supplied from the image data input unit 101 until the image capture end signal is supplied from the image capture end detection unit 103. The same identification information is given to them. Here, the identification information assigning unit 109 is first input after the random number generated by the random number generator, the serial number indicating the order of the data set, the date and time when the Start button 550 is pressed, and the imaging start signal are supplied. Data such as the shooting date and time in the Exif of the captured image data is used as the identification information.
Then, the identification information addition unit 109 sequentially writes the captured image data to which the identification information has been added to the storage unit 107 and stores the same.

When reading the picked-up image data stored in the storage unit 107, the picked-up image data classification unit 102A extracts picked-up image data to which the same identification information is added as the data set of the picked-up object, and the data set unit read out.
Further, the captured image data classification unit 102A may be configured to refer to the storage unit 107 and classify each of the captured image data to which the same identification information is added as an image data set into a type for each target object. ..

The identification information addition unit 109 may be configured to generate a folder of image data sets when the imaging start signal is supplied from the imaging start detection unit 108.
At this time, when the image capture start signal is supplied from the image capture start detection unit 108, the captured image data classification unit 102A creates a folder generated by the identification information addition unit 109 until the image capture end signal is supplied from the image capture end detection unit 103. On the other hand, the captured image data supplied from the image data input unit 101 is input as an image data set to perform classification processing.

FIG. 8 is a flowchart showing an operation example of classification processing of captured image data performed by the captured image classification apparatus 100A according to the present embodiment. A description will be given using a random number as identification information used for classification of captured image data. When the application for classifying the captured image data is activated, the captured image classification device 100A performs the following classification processing.

Step S301: The imaging start detection unit 108 determines whether or not the Start button 550 has been pressed, that is, whether or not an input for starting imaging has been made. At this time, if the Start button 550 is pressed, the imaging start detection unit 108 advances the process to step S302. Here, the imaging start detection unit 108 outputs an imaging start signal to the image data input unit 101 and the identification information addition unit 109.
On the other hand, when the Start button 550 is not pressed, the imaging start detection unit 108 repeats the process of step S301.

Step S302: The identification information adding unit 109 generates a random number by the random number generator provided therein, and uses this random number as the identification information to be added to the captured image data.

Step S303: The image data input unit 101 outputs the captured image data supplied from the image capturing unit 200 to the identification information adding unit 109.
As a result, the identification information adding unit 109 adds the generated identification information to the captured image data supplied from the image data input unit 101, and writes and stores the identification information in the storage unit 107.

Step S304: The image capturing end detection unit 103 determines whether or not the Finish button 552 has been pressed, that is, whether or not an input for ending the image capturing has been performed. At this time, when the Finish button 552 is pressed, the imaging end detection unit 103 advances the process to step S305. Here, the imaging end detection unit 103 outputs an imaging end signal to the image data input unit 101 and the identification information addition unit 109.
On the other hand, if the Finish button 552 is not pressed, the imaging end detection unit 103 advances the process to step S303.

Further, each of the following steps S305 and S306 surrounded by a broken line in FIG. 8 is not an essential process in the present embodiment, but after the classification of the captured image data into the image data set is completed, the image data set This is a process that can be arbitrarily added as a journalizing process for removing the captured image data that is predicted to be unusable for the restoration of the three-dimensional shape model.
By adding each of step S305 and step S306, it is possible to save the labor of manual sorting when restoring the three-dimensional shape model.

Step S305: The captured image data classification unit 102A calculates a multidimensional vector of metadata in each Exif of the captured image data to which the same identification information is added.
Then, the captured image data classification unit 102A generates two combinations of each of the captured image data to which the same identification information is added (similar to the first embodiment, all the captured image data are combined. To generate a pair).
The captured image data classification unit 102A calculates the cosine similarity of each combination.

Step S306: The captured image data classification unit 102A determines whether or not each of the calculated cosine similarities is equal to or greater than a preset similarity threshold.
Then, the captured image data classification unit 102A extracts captured image data that is a combination whose cosine similarity is less than the similarity threshold value, deletes this captured image data from the storage unit 107, or changes identification information. Processing is performed, and sorting processing is performed to exclude captured image data that is expected to be unusable for restoration of the three-dimensional geometric model from the data set.

As described above, according to the present embodiment, when the imaging of the captured image data is started for each target object, the identification information assigning unit 109 assigns the identification information to the captured image data as a data set corresponding to the target object. Is generated, and the same identification information is added to the captured image data from the start of image capturing of each target object to the end of image capturing, even if the storage unit 107 randomly stores the captured image data, this identification information is used. The user can easily extract each of the captured image data of each data set of the target object, and the user can classify each of the captured image data in the multi-view image used to restore the three-dimensional shape model captured while changing the target object and the imaging position. It is possible to easily classify the objects by associating them with each other without taking the trouble of.

Further, according to the present embodiment, the sorting processing is performed so that only the captured image data that can be used when the three-dimensional shape model is restored from the captured image data obtained by capturing the same target object using the Exif metadata. Therefore, it is possible to easily obtain a data set of captured image data capable of reconstructing a three-dimensional geometric model with high accuracy.

Further, in the present embodiment, as an example of an imaging system, the functions of each of the captured image classification apparatus 100 or 100A are installed as, for example, an application in a smartphone, and multi-viewpoints of a target object used for restoration of a three-dimensional shape model are used. It has been described as the configuration for classifying the captured image data in the image.
However, the imaging system may be constructed as, for example, a server client system including a digital camera (client) and a captured image classification device (server). In this case, the server is configured to perform a process that increases the calculation load of the captured image data classification unit 102 or 102A. The other image data input unit 101, the imaging end detection unit 103, the user edit input unit 105, the restoration start notification unit 106, and the like are installed in the digital camera as applications.

Note that a program for realizing the functions of the captured image classification apparatus 100 of FIG. 1 and the captured image classification apparatus 100A of FIG. 6 according to the present invention is recorded in a computer-readable recording medium, and the program recorded in this recording medium is recorded. May be loaded into a computer system and executed to perform a process of classifying the captured image data for restoring the three-dimensional image shape model. The “computer system” mentioned here includes an OS (Operating System) and hardware such as peripheral devices. Further, the "computer system" also includes a WWW (World Wide Web) system having a homepage providing environment (or display environment). The “computer-readable recording medium” is a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), and a CD-ROM (Compact Disc-Read Only Memory), and is incorporated in a computer system. A storage device such as a hard disk. Further, the "computer-readable recording medium" means a volatile memory (RAM (Random Access) in a computer system which is a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory)), such as holding a program for a certain period of time.

Further, the program may be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be a program for realizing a part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

The embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes design changes and the like without departing from the scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Imaging system 2... Three-dimensional geometric model restoration device 100, 100A... Captured image classification device 101... Image data input part 102, 102A... Captured image data classification part 103... Imaging end detection part 104... Image display control part 105... User Edit input unit 106... Restoration start notification unit 107... Storage unit 108... Imaging start detection unit 109... Identification information addition unit 200... Imaging unit 300... Image display unit

Claims (9)

A captured image classification device for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects.
Each of the captured image data is classified into an image data set in which each of the target objects is captured, corresponding to the metadata added to the captured image data and related to the shooting condition when the captured image data was captured. A captured image classification apparatus comprising: a captured image data classification unit. The metadata is each of resolution, aperture value, focal length, shutter speed, ISO (International Organization for Standardization) sensitivity, imaging date and time, and GPS (Global Positioning System) information,
The captured image classification apparatus according to claim 1, wherein the captured image data classification unit classifies each of the captured image data according to at least one of the metadata. When performing classification using any of the above metadata,
The captured image data classification unit sorts each of the captured image data in accordance with the numerical value of the metadata, and in a portion where the numerical distance of the parameter between the respective captured image data exceeds a predetermined separation distance. The captured image classification apparatus according to claim 2, wherein each of the captured image data is classified according to the target object. The metadata is each of resolution, aperture value, focal length, shutter speed, ISO (International Organization for Standardization) sensitivity, imaging date and time, and GPS (Global Positioning System) information,
The captured image data classification unit,
The multi-dimensional vector is generated for each of the captured image data using a plurality of parameters in the metadata, and each of the captured image data is classified according to the similarity between the multi-dimensional vectors. Captured image classification device. A captured image classification device for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects.
An identification information assigning unit that assigns identification information for identifying the target object to each of the captured image data obtained by capturing each of the target objects,
A captured image data classification unit that classifies each of the captured image data into an image data set in which each of the target objects is captured, corresponding to the identification information given to each of the captured image data. A picked-up image classifying device. Further comprising an imaging start detection unit that detects a time point when the imaging of the target object is started,
The identification information providing unit attaches the same identification information to each of the captured image data when the imaging start detection unit detects a time point when the imaging is started. Captured image classification device. The captured image classification apparatus according to any one of claims 1 to 6, further comprising an imaging end detection unit that detects a time point when the imaging of the target object is completed. A captured image classification method for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects,
The captured image data classification unit captures each of the captured image data and each of the target objects corresponding to the metadata added to the captured image data and related to the capturing conditions when capturing the captured image data. A method for classifying a captured image, characterized by including a process of classifying the image data set. A captured image classification method for classifying a multi-viewpoint image used for reconstruction of each three-dimensional shape model of a target object in association with each of the target objects,
An identification information providing unit, an identification information providing process of providing identification information for identifying the target object to each of the captured image data obtained by capturing each of the target objects,
A captured image in which the captured image data classification unit classifies each of the captured image data into an image data set of each of the target objects corresponding to the identification information given to each of the captured image data. A method for classifying a captured image, comprising: a data classification process.