JP6340675B1 - Object extraction device, object recognition system, and metadata creation system - Google Patents

Abstract

An object of the present invention is to extract and recognize an object from moving image data with high accuracy and improve the convenience of object information to improve the use and spread of moving images. An object extraction device performs a feature point extraction process on a processing frame 22 of a two-dimensional image of moving image data 21 to extract a plurality of feature points 23, and each feature point 23 on the two-dimensional image. A first feature amount is detected, a depth detection process is performed to detect a relative depth from the surrounding feature points 23, and a three-dimensional space estimation process is performed to perform a first feature of each feature point 23. Based on the amount and the depth, the actual three-dimensional space 26 of the processing frame 22 is estimated to detect the second feature amount of each feature point 23 on the actual three-dimensional space 26, and the second feature amount and color of each feature point 23 are detected. An object extraction process is performed based on the distribution to detect a feature point group 24 composed of two or more feature points 23 having a feature amount in the real three-dimensional space 26 and extract them as candidate objects 25. [Selection] Figure 2

Description

  The present invention relates to an object extraction device for extracting an object such as a person or an object displayed on an image such as a moving image, an object recognition system using the object extraction device, and a metadata creation system using the object recognition system.

  Conventionally, in a network such as the Internet, a computer such as a moving image server or a moving image database stores moving image data and distributes moving image data to the viewer terminal. In order to promote such moving image distribution, devices and systems for creating metadata related to moving image data and distributing them to viewers have been proposed.

  For example, the metadata distribution apparatus described in Patent Document 1 extracts and converts the metadata of the content of a net program broadcast on the local station from the metadata of the content of the key station using the extraction conversion table and the station-specific data. Since the extracted and converted metadata is distributed, the metadata of the content of the key station is distributed to the receiver as the metadata of the content of the local station. In the program, server type broadcasting is performed by using metadata of contents of the key station.

JP 2006-325134 A

  However, in an apparatus or system such as the metadata distribution apparatus described above, video information cannot be provided unless the broadcast station prepares video information metadata in advance. Therefore, moving image information cannot be provided for moving image data for which such metadata is not prepared.

  In addition, since various objects such as people and objects appear in the moving image data, it is desirable to describe information for identifying these objects, information on the appearance times of these objects, and the like in the metadata. . The worker who creates metadata can identify such objects and grasp the appearance time zone by viewing the video data and confirming the objects that appear. The burden on the person is great. Therefore, an apparatus or system that automatically recognizes an object from moving image data is desired.

  An apparatus or system that recognizes an object recognizes the object by, for example, extracting still image data from moving image data, extracting an object from still image data, and comparing the extracted object with learning data prepared in advance. However, such still image data is usually a two-dimensional planar image, whereas an actual object has a feature in a three-dimensional space having a depth, and is photographed from various angles. Therefore, it has been difficult to extract an accurate feature of an object from still image data of a two-dimensional planar image.

  In addition, when preparing learning data for each object photographed from various angles, a large amount of storage device is required to store a large amount of learning data, and a large number of comparisons are made. Since it is necessary to perform processing, the equipment cost and the number of processing steps increase. For the reasons described above, the accuracy of object recognition decreases, and further, there arises a problem that desired metadata cannot be generated. Further, since the metadata desired by the viewer cannot be distributed, the use and dissemination of moving image data may stagnate.

  When object recognition is performed from moving image data having 3D information of an object in advance, it is necessary to generate moving image data by shooting the object in advance with an imaging device that supports 3D, which increases equipment costs. End up.

  Therefore, in consideration of the above circumstances, the present invention extracts objects displayed in moving image data with high accuracy, recognizes them with high accuracy, and enhances the convenience of information on the recognized objects, thereby making it possible to use and disseminate moving images. The purpose is to improve.

  In order to solve the above-described problem, the first object extraction device of the present invention performs a feature point extraction process on a processing frame that is an object extraction target among a plurality of frames of a two-dimensional image constituting moving image data, and A plurality of feature points of the processing frame are extracted, a first feature amount on the two-dimensional image of each feature point is detected, a depth detection process is performed on the processing frame, and each feature point of the processing frame is surrounded Detecting a relative depth from the feature point, performing a three-dimensional space estimation process on the processing frame, and performing the processing based on at least the first feature amount and the depth of each of the plurality of feature points of the processing frame. Estimating a real three-dimensional space in the frame, detecting a second feature quantity in the real three-dimensional space of a plurality of feature points of the processing frame, and a plurality of feature points of the processing frame Object extraction processing is performed based on at least each of the second feature amount and color distribution, and a feature point group including a set of two or more feature points of the processing frame is detected, and the real three-dimensional space is detected. The feature point group having the feature amount is extracted as a candidate object of the processing frame.

  Further, the second object extraction device of the present invention is the same as the first object extraction device of the present invention described above, which is continuous in the time axis among the plurality of processing frames constituting the moving image data. When there are two or more common processing frames constituting a scene, when performing the depth detection processing, the three-dimensional space estimation processing, and the object extraction processing for each of the two or more common processing frames, A feature point group having a feature amount in a real three-dimensional space, wherein the feature point group detected in common in the two or more common processing frames is extracted as a candidate object of the same scene. And

  The third object extraction device of the present invention is the above-described second object extraction device of the present invention, wherein when the feature point extraction processing is performed on each of the two or more common processing frames, Using the difference between a plurality of feature points extracted from a common processing frame and a plurality of feature points extracted from the other common processing frame, the feature points of the one common processing frame are increased. It is characterized by.

  Further, the fourth object extraction device of the present invention is the above-described second or third object extraction device of the present invention, wherein the moving image data is extracted as the candidate object in the processing frame by performing high quality processing. The feature point of the feature point group to be added is increased.

  The fifth object extraction device of the present invention is the above-described first to fourth object extraction device of the present invention, wherein the candidate object of the same scene is a feature quantity in the real three-dimensional space. In addition, it is characterized by having a time displacement amount on the same scene of the feature amount in the real three-dimensional space.

  According to a sixth object extraction device of the present invention, in the fifth object extraction device of the present invention described above, the image quality enhancement processing is performed for each of a plurality of template images having various color distributions for each minute region. Learning data of various color distributions obtained by sampling the difference between the high-quality data and the low-quality data is stored in advance, and the processing frame is imaged using the learning data most suitable for each minute region of the processing frame. It is characterized by becoming.

  Furthermore, in order to solve the above problem, a first object recognition system of the present invention includes any one of the first to sixth object extraction devices of the present invention described above and a plurality of objects for recognizing the candidate object. A database for storing a reference object together with an original image of each reference object and attached information related to each reference object, wherein the feature point extraction process, the depth detection process, and the 3 for the original image of each reference object An object database storing each reference object extracted as a feature point group having a feature quantity in the actual three-dimensional space of the original image by the dimension space estimation process and the object extraction process, and the object extraction device The extracted candidate object is the object database. An object recognition device that performs an object recognition process corresponding to which of the plurality of reference objects stored in the object recognition device, wherein the object recognition device is configured such that the candidate object is one of the plurality of reference objects. When it is determined that it corresponds to a reference object, object information generated based on the attached information of the one reference object is added to the candidate object.

  Further, the second object recognition system of the present invention is the above-described first object recognition system of the present invention, wherein the object recognition processing includes the feature point group of the candidate object, the color distribution in the processing frame, and the reference. This is performed by comparing the feature point group of the object and the color distribution in the original image.

  Further, according to a third object recognition system of the present invention, in the first or second object recognition system of the present invention described above, the object database classifies the plurality of reference objects based on respective attached information. Thus, two or more reference objects having common attached information are classified and stored in a common category using the common attached information as classification information.

  Further, the fourth object recognition system of the present invention is determined to correspond to one reference object among the plurality of reference objects in the above-described first to third object recognition systems of the present invention. The candidate object is stored in the object database as a new reference object of a category into which the one reference object is classified.

  The fifth object recognition system of the present invention is the above-described candidate object that is determined not to correspond to any of the plurality of reference objects in the above-described fourth object recognition system of the present invention. It is stored in the object database as a reference object of a new category to be classified.

  Furthermore, in order to solve the above-described problem, a first metadata creation system of the present invention includes any one of the first to fifth object recognition systems of the present invention described above, and includes moving image information of predetermined moving image data. And the frame information of the plurality of processing frames constituting the predetermined moving image data and the object information of the candidate objects extracted and recognized from each of the plurality of processing frames, On the basis of this, metadata relating to the moving image data is created.

  According to the present invention, an object displayed in moving image data is extracted with high accuracy and recognized with high accuracy, and the convenience of information on the recognized object is enhanced, thereby improving the use and spread of moving images. It becomes possible.

It is a block diagram which shows the outline of the object recognition system and metadata production system which concern on one Embodiment of this invention. It is a schematic diagram which shows the example of the moving image data processed by the object recognition system which concerns on one Embodiment of this invention. It is a flowchart which shows the object recognition operation | movement and metadata production | generation operation | movement in the object recognition system and metadata production system which concern on one Embodiment of this invention.

  First, the overall configuration of an object recognition system 1 according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, an object recognition system 1 includes an object extraction device 2 that performs an object extraction process based on an image such as a moving image or a still image, and an object database (DB) 3 that stores objects used for the object recognition process. And an object recognition device 4 that performs object recognition processing.

  Objects have features of shapes, colors, and shades that can be identified on a two-dimensional plane, such as living things such as people and animals, stationary objects such as buildings and figurines, and display objects such as characters, symbols, and logo marks. As well as those having features of shape, color, and shading that can be identified in a three-dimensional space. Hereinafter, an object to be subjected to object extraction processing and object recognition processing, that is, an object extracted by the object extraction device 2 and recognized by the object recognition device 4 is referred to as a candidate object 25 (see FIG. 2). An object to be compared for recognition processing of the candidate object 25 and stored in the object DB 3 is referred to as a reference object 3a.

  In the present embodiment, an example will be described in which the object extracting device 2, the object DB 3, and the object recognizing device 4 are connected to be able to communicate with each other via a predetermined network 5 such as the Internet or a LAN (Local Area Network). The object extracting device 2, the object DB 3, and the object recognizing device 4 may be directly connected as long as they can transmit and receive data to each other, or any two or more of them may be integrally configured.

  In this embodiment, an example in which the object extracting device 2, the object DB 3, and the object recognizing device 4 are provided one by one will be described. However, a plurality of object extracting devices 2, a plurality of object DBs 3, and a plurality of object recognizing devices 4 are provided. May be provided. The plurality of objects DB3 collectively manage the reference objects 3a stored in each object DB3. When a keyword or category is specified, the reference object 3a corresponding to the keyword or category extends over the plurality of object DB3. Searched. The plurality of object DBs 3 store one reference object 3a based on one image in any one object DB 3 without being redundantly stored in two or more object DBs 3.

  First, the object extraction device 2 will be described. As described above, the object extraction device 2 is configured to extract the candidate object 25 displayed in an image such as a moving image or a still image. For example, the object extraction device 2 includes a control unit 10, a storage unit 11, and a communication unit 12. The object extraction device 2 includes an image input unit 13, a frame acquisition unit 14, a high image quality improvement unit 15, a frame adjustment unit 16, a feature point extraction unit 17, a depth detection unit 18, and a three-dimensional space estimation. A unit 19 and an object extraction unit 20 are provided. The image input unit 13, the frame acquisition unit 14, the image quality improvement unit 15, the frame adjustment unit 16, the feature point extraction unit 17, the depth detection unit 18, the three-dimensional space estimation unit 19, and the object extraction unit 20 are stored in the storage unit 11. And may be configured by a program that operates by being controlled by the control unit 10.

  The object extraction device 2 also uses extraction processing data 8a used for the object extraction processing (for example, a template image for high image quality processing, which will be described later, feature point distribution data for feature point distribution determination, and color displacement for color distribution determination. -It is connected via a network 5 to an extraction processing database (DB) 8 that stores depth data, object extraction data for object extraction, and the like. The object extraction device 2 may be directly connected or may be configured integrally as long as data can be transmitted / received to / from the extraction processing DB 8.

  The control unit 10 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and is configured to control the overall operation of the object extraction device 2. The storage unit 11 has a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and a recording medium such as a hard disk, and stores information, data, programs, and the like controlled by the control unit 10. Configured.

  The communication unit 12 is an interface for the object extraction device 2 to connect to the network 5, that is, connects the object extraction device 2 to the object DB 3 and the object recognition device 4 through the network 5.

  The image input unit 13 inputs, for example, image data such as moving image data 21 (see FIG. 2) and still image data to be subjected to object extraction processing. For example, the image input unit 13 communicates with an external moving image database (DB) 6 storing a plurality of moving image data 21 or other external computers via the communication unit 12 to perform object extraction processing. The moving image data 21 can be selected and input from the moving image DB 6. Alternatively, the image input unit 13 reads the moving image data 21 from the storage unit 11 or reads the moving image data 21 stored in a storage medium such as a DVD (Digital Versatile Disc) or Blu-ray Disc (registered trademark) as a reading device ( (Not shown), and may be input as moving image data 21 to be subjected to object extraction processing. The moving image data 21 includes moving image information such as a moving image title and contents set in advance in addition to video data and audio data.

  The image input unit 13 extracts image data information of the input image data. For example, in the case of moving image data 21, the image data information includes the moving image ID, the number of frames, the frame size and the format of the moving image data 21, the title of the moving image data 21, author information, the creation date, the category of the moving image, and the performer. Information, and moving image information such as thumbnails (URLs). Still image data includes still image information such as a still image title, data size, and format. When image data is acquired from a website, the image data information included in the description content of the website may be image data information.

  As shown in FIG. 2, the frame acquisition unit 14 acquires a plurality of still image frames constituting the moving image data 21 based on the frame rate when the moving image data 21 is a target of object extraction processing. Each of the plurality of still image frames becomes a processing frame 22 to be subjected to object extraction processing. When the image input unit 13 inputs still image data, the still image data directly becomes a processing frame 22 that is a target of object extraction processing. Note that the processing frame 22 corresponds to a two-dimensional planar image obtained by photographing a subject from one direction with a single imaging device.

  Further, the frame acquisition unit 14 extracts frame information of each acquired processing frame 22. The frame information includes, for example, a relation ID and a reproduction time (time stamp) in the moving image data 21 of the processing frame 22, a frame number (unique ID) of the processing frame 22, and the like.

  The image quality improving unit 15 performs an image quality improving process on the processing frame 22. Particularly in the present embodiment, the image quality improving unit 15 increases the image quality of the processing frame 22 so that the extraction amount of the feature points 23 of the processing frame 22 increases.

  For example, the high image quality improvement unit 15 includes a plurality of template images for high image quality processing having various color distributions (color displacements) for each minute region (for example, a × a pixel range, a is an odd number of 3 or more). Are stored in advance in the extraction process DB 8. Each template image is provided with high image quality data and low image quality data, and the low image quality data is indicated by the resolution of each template image. Further, the image quality enhancement unit 15 associates color displacement data for image quality enhancement obtained by sampling the difference (color displacement) for each minute area of the image quality data and low image quality data of each template image with each template image. , Stored in advance in the extraction process DB 8. Then, the image quality improving unit 15 performs a convolution operation for each minute region of the processing frame 22 using various color displacement data that matches the resolution of the processing frame 22, thereby changing the color displacement of each minute region of the processing frame 22. The processing frame 22 is improved in image quality by determining and combining (combining) the color displacement data having the highest probability (the most suitable) from the corresponding color displacement data. In this convolution, it is not always necessary to use all the color displacement data, and color displacement data that approximates the color data of each minute region of the processing frame 22 may be used.

  In addition, the image quality improvement unit 15 generates a template image having the original processing frame 22 (or its local region) as low image quality data and the processing frame 22 (or its local region) after image quality improvement as high image quality data, The extracted data is stored in the extraction process DB 8 as machine learning learning data for high image quality processing. Therefore, since the image quality improvement unit 15 uses the template image for image quality improvement processing accumulated in the extraction processing DB 8 by machine learning, the image quality improvement processing can be performed with higher accuracy each time processing is performed. it can.

  Furthermore, when the moving image data 21 is the object of the object extraction process, the image quality improving unit 15 continues the same scene in the time axis among the plurality of processing frames 22 constituting the moving image data 21. With respect to two or more common processing frames 22 that constitute, one common processing frame 22 is improved in image quality based on the other common processing frames 22. For example, the image quality improving unit 15 uses a difference between a plurality of feature points 23 extracted from one common processing frame 22 and a plurality of feature points 23 extracted from another common processing frame 22. The image quality of one common processing frame 22 is improved so that the feature points 23 of the one common processing frame 22 increase. Among the feature points 23 included in the other common processing frame 22, a feature point 23 not included in the one common processing frame 22 is added to the one common processing frame 22. 22 feature points 23 increase.

  The frame adjustment unit 16 performs various image processing on the processing frame 22 according to the property of the processing frame 22 and the property of the moving image data 21.

  For example, the frame adjustment unit 16 performs mosquito noise and block noise reduction processing of the processing frame 22. When the frame adjustment unit 16 detects mosquito noise from the processing frame 22, the frame adjustment unit 16 reduces the mosquito noise by performing smoothing using the peripheral information. When detecting the block noise from the processing frame 22, the frame adjustment unit 16 compares the block noise portion with the plurality of template images described above, and uses the learning data of the most suitable template image to improve the image quality. Reduce noise. If there is no template image that matches the block noise portion, the frame adjustment unit 16 reduces the block noise by performing unsharp mask processing, blurring processing, or the like on the block noise portion.

  In addition, when the processing frame 22 includes a high-contrast region, the frame adjustment unit 16 may perform HDR processing locally on the region because the region may lose many image details. In HDR processing, a plurality of local contrast data is created, and the most suitable contrast data is synthesized with a high-contrast region to generate an image having a high-quality tone balance.

  In addition, when the moving image data 21 is a target of object extraction processing, the frame adjustment unit 16 performs frame interpolation processing when the frame rate of the moving image data 21 is low. In the frame interpolation process, first, when the processing frame 22 itself is blurred due to a low frame rate, the blur is eliminated by sharpening or the like. Then, an interpolation frame is generated as an intermediate image of a predetermined time between two consecutive processing frames 22 and inserted between these two processing frames 22. For example, when two consecutive processing frames 22 are common processing frames 22 in the same scene and only the common candidate object 25 is moving, the depth of the feature point 23 between the two processing frames 22 Based on the movement vector, the common candidate object 25 is calculated by estimating the feature point 23 and its depth at a predetermined time between the two processing frames 22. Then, the candidate object 25 having the calculated feature point 23 and its depth is synthesized into a processing frame 22 similar to the two processing frames 22 to generate an interpolation frame for a predetermined time. Such frame interpolation processing may be performed after the feature point extraction processing by the feature point extraction unit 17 or the depth detection processing by the depth detection unit 18.

  In addition, when the frame rate of the moving image data 21 is low, the frame adjustment unit 16 performs afterimage reduction processing to reduce afterimages generated in the processing frame 22 due to the low frame rate.

  In addition, when the frame rate of the moving image data 21 is high, the frame adjustment unit 16 performs a thinning process, reduces the number of processing frames 22 in a predetermined period, and reduces the load and time required for subsequent image processing. . In the thinning-out process, the processing frame 22 having less influence is deleted, such as the processing frame 22 having a small motion vector (motion of the candidate object 25) of each feature point 23 in two or more continuous processing frames 22. It is preferable to leave the processing frame 22 having a large influence like the processing frame 22 before and after the scene change.

  In addition, when the moving image data 21 compressed into a predetermined format is input to the image input unit 13, the frame adjustment unit 16 evaluates the robustness of the compression algorithm of the format, and the moving image data 21 is converted into the format. The information lost for the processing frame 22 at the time of encoding is secured when the moving image data 21 is decoded, and the original processing frame 22 is reproduced.

  The feature point extraction unit 17 performs a feature point extraction process on the processing frame 22 to extract a plurality of feature points 23 of the processing frame 22 and detects a first feature amount of each feature point 23 on the two-dimensional image. . The feature point extraction unit 17 performs the feature point extraction processing on each of the two-dimensional images of the plurality of processing frames 22 constituting the moving image data 21 when the moving image data 21 is a target of the object extraction processing. For example, the first feature amount of each feature point 23 includes two-dimensional coordinates, luminance and color variables (RGB), a luminance gradient vector (luminance gradient with respect to the surrounding image or the entire image), and the like.

  For example, the feature point extraction unit 17 performs a sharpening process on the processing frame 22 as a pre-process of the feature point extraction process, thereby calculating a luminance displacement amount between pixels, and an acceleration converted from the displacement amount. An edge emphasis frame in which the edge is emphasized as the size increases is generated. Then, as the feature point extraction process, the feature point extraction unit 17 extracts a plurality of feature points 23 based on the edge emphasized in the edge enhancement frame, and calculates the first feature amount of each feature point 23.

  The depth detection unit 18 performs depth detection processing on the processing frame 22 from which the feature point 23 has been extracted by the feature point extraction unit 17, and the relative depth from the surrounding feature points 23 for each feature point 23 of the processing frame 22. Is detected.

  For example, the depth detection unit 18 first performs a convolution operation for each local region of the processing frame 22 using various feature point distribution data, so that the number of feature points 23 for each local region in the processing frame 22 (existence probability). ) Distribution. For example, the feature point distribution data does not need to specify an object, but is created to indicate the distribution of the feature points 23 of the object. Then, the depth detection unit 18 detects from the distribution of the feature points 23 in the processing frame 22 what is determined with a higher probability as a distribution corresponding to any feature point distribution detection data. Note that the distribution of the feature points 23 in the actual three-dimensional space (the actual three-dimensional space 26, see FIG. 2) in the image of the processing frame 22 can also be determined by performing this convolution operation in the two-dimensional direction.

  For example, the feature point distribution data is created so as to have the distribution of the feature points 23 of the reference object 3a stored in the object DB 3, and the feature point distribution data of objects of various categories and sizes is stored in the extraction processing DB 8 in advance. The The feature point distribution data may be created as machine learning learning data for feature point distribution determination when the object recognition device 4 stores the reference object 3a recognized with high accuracy in the object DB 3. In addition, the feature point distribution data may be created based on the feature point distribution determined from the processing frame 22 by the feature point distribution determination and determined with high accuracy. Therefore, since the depth detection unit 18 uses the feature point distribution data accumulated in the extraction processing DB 8 by machine learning, the depth detection unit 18 can perform feature point distribution determination with higher accuracy each time processing is performed.

  In addition, the depth detection unit 18 is a color displacement indicating a correspondence relationship between a color displacement of a minute region of various sizes (for example, an a × a pixel range, a is an integer of 3 or more) and a depth corresponding to the color displacement. Using the depth data, perform a convolution operation for each minute region of the processing frame 22 to determine the color distribution of the pixels in the processing frame 22. For example, the color displacement of the color displacement-depth data is the displacement of the color data (for example, RGB) of the surrounding pixels viewed from the center pixel in the minute area, and the depth is the relative of the surrounding pixels viewed from the center pixel in the minute area. Depth. Then, the depth detection unit 18 uses the color displacement-depth data of various objects having the same category and size with respect to the distribution of the feature points 23 of the processing frame 22 as a result of the above-described feature point distribution determination. Detect what is determined with higher probability as color displacement-depth data matching the color displacement of each minute region. Thereby, the depth detection unit 18 detects the relative depth from the surrounding feature points 23 for each feature point 23.

  For example, the color displacement-depth data is created so as to have a correspondence relationship between the color displacement and the depth corresponding to the color displacement for each minute area of the reference object 3a stored in the object DB 3, and has various categories and sizes. The color displacement-depth data of the object is stored in advance in the extraction process DB 8. The color displacement-depth data may be created as learning data for machine learning for color distribution determination when the reference object 3a recognized with high accuracy by the object recognition device 4 is stored in the object DB 3. In addition, the color displacement-depth data may be created by the color distribution determined from the processing frame 22 by the color distribution determination and determined with high accuracy. Therefore, since the depth detection unit 18 uses the color displacement-depth data accumulated in the extraction processing DB 8 by machine learning, it is possible to perform more accurate color distribution determination each time processing is performed.

  Further, the depth detection unit 18 calculates the direction vector of each feature point 23 of the processing frame 22 based on the result of the color distribution determination described above. For example, the direction vector of a predetermined feature point 23 has the coordinates of the feature point 23 and the relative depth of surrounding pixels from the feature point 23 (pixels in a minute region centered on the feature point 23). ing. In other words, the direction vector of the predetermined feature point 23 is a luminance gradient and a color displacement between the surrounding pixels in the actual three-dimensional space (real three-dimensional space 26, see FIG. 2) in the image of the processing frame 22. Direction of color gradient).

  Further, the depth detection unit 18 determines a curve passing between the feature points 23 in the processing frame 22 based on the direction vector of each feature point 23 according to the feature point distribution of the region where each feature point 23 exists. Create a new method. For example, the depth detection unit 18 generates a spline curve, a Bezier curve, or the like that passes through each feature point 23 and other feature points 23 (surrounding feature points) as control points. In addition, the depth detection unit 18 may be used when the distribution of the feature points 23 is inconvenient for generating a spline curve, a Bezier curve, etc. A curve between the feature points 23 is generated by making an approximate curve using a regression curve or the like. The curves generated as described above are the luminance gradient and color displacement (color gradient) between the feature points 23 in the actual three-dimensional space in the image of the processing frame 22 (the actual three-dimensional space 26, see FIG. 2). It becomes a curve along the direction of.

  Then, the depth detection unit 18 calculates a relative depth of the predetermined feature point 23 from the surrounding feature points 23 based on a curve generated for each surrounding feature point 23. Thereby, the depth detection unit 18 detects the relative positional relationship in the actual three-dimensional space 26 with the feature points 23 around each feature point 23. After calculating the depth (positional relationship) for each feature point 23 of the processing frame 22 in this way, the depth (positional relationship) for the predetermined feature point 23 is calculated for each surrounding feature point 23. You may adjust suitably based on (positional relationship).

  The three-dimensional space estimation unit 19 uses the first feature amount and the depth (relative depth from the surrounding feature points 23) of each of the plurality of feature points 23 of the process frame 22 to determine the actual feature in the image of the process frame 22. A three-dimensional space (real three-dimensional space 26) is estimated. For example, the three-dimensional space estimation unit 19 compares the first feature amount and the depth of each feature point 23 of the processing frame 22 with each other so that the depth of each feature point 23 can be matched. 26 is estimated and calculated. Further, the three-dimensional space estimation unit 19 detects the second feature amount on the real three-dimensional space 26 for the plurality of feature points 23 of the processing frame 22. For example, the second feature amount of each feature point 23 includes a three-dimensional coordinate on the real three-dimensional space 26.

  The object extraction unit 20 performs an object extraction process based on the second feature amount and the color distribution of each of the plurality of feature points 23 of the processing frame 22. Then, the object extraction unit 20 detects a feature point group 24 including a set of two or more feature points 23 according to the distribution state of the plurality of feature points 23 in the processing frame 22 by the object extraction process. For example, the object extraction unit 20 detects one or more sets of feature points 24 based on the distribution of the feature points 23 when the processing frame 22 is divided into the quadtree space. Each feature point group 24 has a feature amount (coordinates and the like) in the actual three-dimensional space 26, and the object extraction unit 20 uses the feature point group 24 detected in this way as a processing frame 22 as an extraction source. Is extracted as a candidate object 25.

  The object extraction unit 20 may detect a feature point group 24 corresponding to the object extraction data by performing a convolution operation for each local region of the processing frame 22 using various object extraction data. For example, the object extraction data is created so as to indicate the feature point group 24 of the reference object 3a stored in the object DB 3, and the object extraction data of objects of various categories and sizes is stored in the extraction processing DB 8 in advance. As the object extraction data, the reference object 3a recognized with high accuracy by the object recognition device 4 is also used as learning data for machine learning of the object extraction processing. In addition, the object extraction data may be created from the candidate object 25 extracted from the processing frame 22 by the object extraction process and determined with high accuracy. Therefore, since the object extraction unit 20 uses the reference object 3a accumulated in the object DB 3 by machine learning, it is possible to perform object extraction processing with higher accuracy each time processing is performed.

  Further, the object extraction unit 20 stores the extracted candidate object 25 in the storage unit 11 in association with the extraction source processing frame 22, and when the processing frame 22 is acquired from the moving image data 21, the extracted candidate object 25 is stored in the storage unit 11. The video data 21 is also associated. The candidate object 25 includes the first feature amount, the depth, and the second feature amount of each feature point 23 constituting the feature point group 24 as information regarding the corresponding feature point group 24. In addition, the object extraction unit 20 adds the source processing frame 22 to the candidate object 25.

  In addition, when the moving image data 21 is a target of the object extracting process, the object extracting unit 20 configures the same scene that is continuous in the time axis among the plurality of processing frames 22 constituting the moving image data 21. When object extraction processing is performed for each of two or more common processing frames 22, the feature point group 24 detected in common in the common processing frame 22 is set as a candidate object 25 common in the same scene. At this time, if the candidate object 25 is moving between the common processing frames 22 of the same scene, the candidate object 25 common to the same scene is the feature point group 24 (feature amount in the real three-dimensional space 26). Movement amount (time displacement amount on the same scene).

  Then, the object extraction device 2 outputs the candidate object 25 extracted as described above to the object recognition device 4 for object recognition processing together with the image data information and the frame information.

  Note that the object extraction device 2 performs approximation determination on the plurality of processing frames 22 after adjustment by the frame adjustment unit 16 when the moving image data 21 is a target of object extraction processing, and the processing frame 22 to be approximated is determined. You may exclude from the process target of the feature point extraction part 17, the depth detection part 18, the three-dimensional space estimation part 19, and the object extraction part 20. FIG. For example, for two processing frames 22 that are successively approximated before and after, the preceding processing frame 22 is a processing target and the subsequent processing frame 22 is excluded from the processing target. When the subsequent processing frame 22 is excluded from the processing target in the previous approximation determination, the processing frame 22 to be compared with the subsequent processing frame 22 in the current approximation determination is set as the processing target in the previous approximation determination. A processing frame 22 is obtained.

  Next, the object DB 3 will be described. As described above, the object DB 3 stores a plurality of reference objects 3a used for object recognition processing. When the object recognition device 4 performs an object recognition process and receives an instruction to output a reference object from the object recognition device 4, the object DB 3 sequentially outputs the stored reference objects 3 a to the object recognition device 4.

  Each reference object 3a is a feature point group of feature points having feature quantities in a real three-dimensional space from a two-dimensional planar image (hereinafter referred to as an original image), as with the candidate object 25 extracted by the object extracting device 2. And may include the first feature amount, the depth, and the second feature amount of each feature point constituting the feature point group. The original image is added to the reference object 3a, and additional information related to the reference object 3a is also added. The attached information includes, for example, specific information for specifying the reference object 3a, original image information attached to the original image, moving image information attached to the moving image from which the original image is acquired, and the website from which the original image and moving image are acquired. There is information included in the description.

  Furthermore, the object DB 3 stores a plurality of reference objects 3a classified into a plurality of categories based on the attached information. The categories may be divided into multiple categories such as large categories such as persons and objects, and small categories such as specific persons and specific objects. The object DB 3 classifies and stores two or more reference objects 3a having common auxiliary information by classifying the common auxiliary information into a common category using classification information. Note that when the object recognition device 4 performs an object recognition process, the object DB 3 specifies a category from the object recognition device 4 and receives a reference object output instruction, so that the reference object stored in the specified category is stored. 3a can also be output to the object recognition device 4.

  The reference object 3a stored in the object DB 3 can be created by the object extracting device 2 and the object recognizing device 4, and may be created by other means as long as it has the above configuration. For example, the object DB 3 can input the candidate object 25 after the object recognition process by the object recognition device 4 as learning data for machine learning of the object recognition process and store it as a reference object 3a. Therefore, since the object recognition device 4 uses the reference object 3a accumulated in the object DB 3 by machine learning, it is possible to perform object recognition processing with higher accuracy each time processing is performed.

  In this case, the candidate object 25 determined to correspond to the predetermined reference object 3a by the object recognition device 4 is added with object information to be described later based on the reference object 3a, and the category in which the predetermined reference object 3a is classified. Is stored in the object DB 3 as a new reference object 3a. On the other hand, the candidate object 25 determined not to correspond to any reference object 3a by the object recognition device 4 is stored in the object DB 3 as a reference object 3a of a new category into which the candidate object 25 is classified.

  Next, the object recognition device 4 will be described. The object recognition device 4 is configured to perform object recognition processing as described above, and the candidate object 25 extracted by the object extraction device 2 corresponds to any of the plurality of reference objects 3a stored in the object DB 3. Determine whether.

  For example, when the object recognition device 4 inputs the candidate object 25 from the object extraction device 2, the object recognition device 4 instructs the object DB 3 to output a reference object. Then, when the object recognition device 4 inputs the reference object 3a from the object DB 3, the object recognition device 4 determines whether the candidate object 25 corresponds to the reference object 3a. For example, the object recognizing device 4 uses the feature point group 24 (feature amount in the real three-dimensional space) of the candidate object 25 and the color distribution in the processing frame 22 from which the candidate object 25 is extracted, and the feature point group (real three-dimensional) of the reference object 3a. The feature amount in the space) and the color distribution in the original image are compared, and the similarity between the reference object 3a and the candidate object 25 is calculated. As described above, by adding the processing frame 22 to the candidate object 25 when extracting the candidate object 25, the color distribution of the candidate object 25 can also be used. In this comparison process, the candidate object 25 and the reference object 3a are compared in feature quantity in the actual three-dimensional space, so that it is not necessary to match one direction and size to the other.

  Then, the object recognition device 4 calculates the similarity between the plurality of reference objects 3a and the candidate object 25, and selects a reference object 3a having a higher similarity, for example, a reference object 3a having a predetermined similarity threshold or more as a candidate object. It is determined that it corresponds to 25. At this time, the object recognition device 4 acquires information related to the candidate object 25 from the attached information of the reference object 3a having a high similarity, generates object information, and adds the object information to the candidate object 25. In addition, for the creation of the object information, the attached information of only one reference object 3a may be used, or the attached information of two or more reference objects 3a having high similarity may be used. Furthermore, the classification information of the reference object 3a in the object DB 3 may be used for creating the object information.

  On the other hand, when the similarity between each reference object 3a and the candidate object 25 is less than a predetermined similarity threshold, the object recognition device 4 does not correspond to any reference object 3a. Is determined.

  Further, the object recognition device 4 outputs the candidate object 25 to the object DB 3 as learning data for machine learning of the object recognition processing and stores it as the reference object 3a regardless of any determination.

  The object recognition system 1 also includes an image collection device 7 in order to increase the number of reference objects 3a in the object DB 3. The image collection device 7 includes an image collection crawler that searches and collects a moving image or a still image having the reference object 3 a via the network 5. When the image collecting device 7 executes the image collecting crawler function, the moving image data 21 (see FIG. 2) stored in the external moving image DB 6 or other terminal connected to the image collecting device 7 via the network 5 Image data such as still image data is collected sequentially. The image collection device 7 may collect all the image data disclosed to the network 5, but searches and collects the image data based on the category (business type) or keyword selected by the operator. Also good.

  The image collection device 7 is connected to the object extraction device 2 and outputs the collected image data to the object extraction device 2. In the object extraction device 2, the candidate object 25 is extracted from the image data and output to the object recognition device 4 as described above. In the object recognition device 4, the object recognition processing of the candidate object 25 is performed using the reference object 3a of the object DB 3 as described above. Further, the candidate object 25 after the object recognition processing becomes learning data and becomes the reference object 3a. Is stored in the object DB3. Thus, by using the image collection device 7, the number of reference objects 3a stored in the object DB 3 increases, and the accuracy of object recognition processing by the object recognition device 4 can be increased.

  The image collecting device 7 may be provided independently of the object extracting device 2, the object DB 3, and the object recognizing device 4, or may be configured integrally with any one of them.

  The object recognition system 1 as described above is applied to a metadata creation system 30 that performs metadata creation processing of the moving image data 21 (see FIG. 2). The metadata creation system 30 includes a metadata creation device 31 that creates the metadata 32a and a metadata database (DB) 32 that stores the created metadata 32a. In this embodiment, an example in which one metadata creation device 31 and one metadata DB 32 are provided will be described. However, a plurality of metadata creation devices 31 and a plurality of metadata DBs 32 may be provided. The metadata DB 32 is not limited to that used by one metadata creation device 31, and may be provided so as to be usable by a plurality of metadata creation devices 31.

  The metadata creation device 31 is connected to the metadata DB 32 via the network 5 so as to be communicable with each other, and is also connected to the object extraction device 2 and the object recognition device 4 of the object recognition system 1 so as to be able to communicate with each other. Note that the metadata creation device 31 may be directly connected or may be integrally configured as long as data can be transmitted and received with the metadata DB 32.

  The metadata DB 32 stores a plurality of metadata 32a so that the metadata 32a corresponding to the moving image data 21 can be searched by using the title or moving image ID of the moving image data 21 as a search keyword. The metadata DB 32 may store the metadata 32a so that the metadata 32a such as the new moving image data 21 with the new creation date and time, the moving image data 21 with a high search frequency, and the recommended moving image data 21 are preferentially searched.

  The plurality of metadata DBs 32 collectively manage the metadata 32a stored in each metadata DB 32. When the title or the movie ID of the moving image data 21 is specified, the metadata 32a is spread over the plurality of metadata DBs 32. Searched. The plurality of metadata DBs 32 store one metadata 32 a based on one moving image data 21 in any one metadata DB 32 without storing the metadata 32 a in duplicate in two or more metadata DBs 32. The plurality of metadata DBs 32 may be provided for each category of the moving image data 21.

  The metadata creation device 31 includes the moving image information of the predetermined moving image data 21, each frame information of the plurality of processing frames 22 constituting the predetermined moving image data 21, and the object of the candidate object 25 extracted and recognized from each processing frame 22. When the information is input, the information 32 is totaled to create the metadata 32a of the predetermined moving image data 21. Further, the metadata creation device 31 stores the metadata 32 a created for the predetermined moving image data 21 in the metadata DB 32.

  For example, the metadata 32a describes moving image information such as the title of the moving image data 21, the names of performers, and the like. Furthermore, the reproduction time of each processing frame 22 in the reproduction order of the plurality of processing frames 22 constituting the moving image data 21, etc. Frame information is described. The metadata 32 a describes object information of the candidate object 25 extracted from each processing frame 22 along with the frame information of each processing frame 22. That is, in the metadata 32a, frame information and object information are shown on the timeline.

  For two or more common processing frames 22 in the same scene, scene information such as the time zone of the same scene is described in the metadata 32a, and the same scene is attached to the scene information of the same scene. Object information of the common candidate object 25 is described. Such scene information is also shown on the timeline in the metadata 32a. For each processing frame 22 other than the first and last processing frames 22 in the same scene, description of frame information and object information may be omitted.

  Next, the metadata creation operation of the predetermined moving image data 21 in the object recognition system 1 and the metadata creation system 30 having the above-described configuration will be described with reference to the flowchart of FIG.

  In the metadata creation system 30, when performing metadata creation processing for the predetermined moving image data 21, the predetermined moving image data 21 is input to the object extraction device 2 of the object recognition system 1 (step S 1), and the image input unit 13. Thus, image data information of the moving image data 21, that is, moving image information is extracted.

  In the object extraction device 2, the frame acquisition unit 14 acquires a plurality of processing frames 22 constituting the moving image data 21 (step S2), and extracts frame information of each processing frame 22. Further, each processing frame 22 is improved in image quality by the image quality improving unit 15 and adjusted by the frame adjusting unit 16 so that optimum feature point extraction processing and depth detection processing can be performed (step S3).

  Then, the feature point extraction unit 17 extracts a plurality of feature points 23 of each processing frame 22, and detects the first feature amount on the two-dimensional image of each feature point 23 (step S4), and further the depth. The detection unit 18 detects the depth from the feature points 23 around each feature point 23 (step S5). In addition, the three-dimensional space estimation unit 19 estimates the actual three-dimensional space 26 of each processing frame 22 based on the first feature amount and the depth of the plurality of feature points 23 of each processing frame 22. A second feature amount on the actual three-dimensional space 26 is detected (step S6).

  Next, a feature point group 24, that is, a candidate object 25 is extracted by the object extraction unit 20 based on the second feature amount and the color distribution of the plurality of feature points 23 of each processing frame 22 (step S7). A corresponding processing frame 22 is added to the object 25.

  Then, the object extraction device 2 extracts the candidate object 25 extracted from each processing frame 22 together with the moving image information of the predetermined moving image data 21 and each frame information of the plurality of processing frames 22 constituting the moving image data 21. To output.

  In the object recognition device 4, object recognition processing of the candidate object 25 input from the object extraction device 2 is performed as described above (step S8), and this candidate object 25 corresponds to the reference object 3a stored in the object DB 3. It is determined whether or not to do so.

  When it is determined that the candidate object 25 corresponds to one reference object 3a, object information is generated based on the attached information of the one reference object 3a and added to the candidate object 25 (step S9). On the other hand, if it is determined that the candidate object 25 does not correspond to any reference object 3a, the object generated based on the moving image information of the predetermined moving image data 21 and the frame information of the processing frame 22 corresponding to the candidate object Information is added to the candidate object 25.

  Then, the candidate object 25 after the object recognition process is stored in the object DB 3 as the reference object 3a for generating learning data (step S10).

  Further, in the object recognition device 4, metadata information of the moving image information of the predetermined moving image data 21, each frame information of the plurality of processing frames 22 constituting the moving image data 21, and object information of the candidate object 25 of each processing frame 22 31 is output.

  In the metadata creation device 31, the moving image information, each frame information, and each object information input from the object recognition device 4 are totaled, and metadata 32a of predetermined moving image data 21 is created based on the totaling result (step). S11). This metadata 32a is stored in the metadata DB 32 (step S12).

  The metadata creation system 30 as described above is applied to a metadata delivery system 40 that performs metadata delivery processing of predetermined moving image data 21. The metadata distribution system 40 includes a metadata distribution device 41 that distributes the metadata 32a.

  The metadata distribution device 41 is connected to the metadata DB 32 via the network 5 so as to be able to communicate with each other, and is also connected to the viewer terminal 42 so as to be able to communicate with each other. Note that the metadata distribution device 41 may be directly connected or may be integrally configured as long as data can be transmitted / received to / from the metadata DB 32. Further, the metadata distribution device 41 may be configured integrally with the metadata creation device 31.

  The metadata distribution device 41 is configured to acquire and provide the metadata 32 a of the moving image data 21 from the metadata DB 32 in response to access from the viewer terminal 42. Further, the metadata distribution device 41 is configured to give access authority to a predetermined viewer and provide the metadata 32a of the moving image data 21 in response to a request from the viewer terminal 42 of the viewer. Also good.

  The viewer terminal 42 is, for example, a mobile terminal such as a smartphone, a mobile phone, or a tablet that can be connected to the network 5 and can reproduce the moving image data 21 distributed via the network 5, a personal computer, a TV, or the like. A stationary terminal may be used. Alternatively, the viewer terminal 42 may be, for example, a playback device that can be connected to the network 5 and can read and play back the moving image data 21 stored in a storage medium such as a DVD.

  For example, the viewer terminal 42 is connected via the network 5 to the moving image DB 6 that distributes the moving image data 21 by the download method or the streaming method in response to the access from the viewer terminal 42, and the moving image data 21 distributed from the moving image DB 6. Play. The moving image DB 6 may be configured to grant access authority to a predetermined viewer and distribute the moving image data 21 in response to a request from the viewer terminal 42 of the viewer.

  In the present embodiment, as described above, the object extraction device 2 performs the feature point extraction process on the processing frame 22 to be extracted from the plurality of frames of the two-dimensional image constituting the moving image data 21 to obtain the processing frame. A plurality of feature points 23 of 22 are extracted, a first feature amount on the two-dimensional image of each feature point 23 is detected, a depth detection process is performed on the processing frame 22, and each feature point 23 of the processing frame 22 is detected. A relative depth from surrounding feature points 23 is detected, a three-dimensional space estimation process is performed on the processing frame 22, and processing is performed based on at least the first feature amount and the depth of each of the plurality of feature points 23 of the processing frame 22. A real three-dimensional space 26 in the frame 22 is estimated, and second feature quantities on the real three-dimensional space 26 of a plurality of feature points 23 in the processing frame 22 are detected. The object extraction processing is performed based on at least the second feature amount and the color distribution of each of the plurality of feature points 23 to detect a feature point group 24 including a set of two or more feature points 23 in the processing frame 22, A feature point group 24 having a feature amount on the three-dimensional space 26 is extracted as a candidate object 25 of the processing frame 22.

  With such a configuration, a feature point group 24 composed of a set of two or more feature points 23 is determined based on the second feature amount and color distribution of each feature point 23 in the processing frame 22 on the real three-dimensional space 26. Therefore, the object can be extracted with higher accuracy. In addition, the candidate object 25 having the feature amount in the real three-dimensional space 26 can be extracted from the processing frame 22 without using the moving image data generated by the imaging device that captures the three-dimensional image. Furthermore, since this candidate object 25 has a feature amount in the actual three-dimensional space 26, it can identify the feature of a person or an object without depending on the shooting angle, and therefore can recognize it with high accuracy. Can do. As a result, the convenience of the information of the recognized object can be improved, and the use and spread of the moving image can be improved.

  In addition, according to the present embodiment, the object extraction device 2 includes two or more common processes that are consecutive in the time axis and that constitute the same scene among the plurality of processing frames 22 that configure the moving image data 21. When there is a frame 22, when performing depth detection processing, three-dimensional space estimation processing, and object extraction processing for each of two or more common processing frames 22, a feature point group having feature quantities in the real three-dimensional space 26 24, the feature point group 24 detected in common in two or more common processing frames 22 is extracted as a candidate object 25 of the same scene.

  With such a configuration, the same object appearing in the same scene can be recognized with high accuracy regardless of the photographed angle by using the feature amount in the actual three-dimensional space 26.

  Further, according to the present embodiment, when the object extraction device 2 performs the feature point extraction process on each of the two or more common processing frames 22, a plurality of feature points extracted from the one common processing frame 22. The feature points 23 of one common processing frame 22 are increased by utilizing the difference between the feature points 23 extracted from the other common processing frame 22 and the plurality of feature points 23.

  With such a configuration, candidate objects 25 having more feature points 23 can be extracted, and more feature points 23 are used in the object recognition process, so that recognition accuracy of candidate objects 25 can be improved. .

  Further, according to the present embodiment, in the object extraction device 2, the candidate object 25 in the same scene is added to the feature scene in the real 3D space 26 in addition to the feature quantity in the real 3D space 26. There is also a time displacement amount.

  With such a configuration, it is possible to extract the feature amount of the motion on the real three-dimensional space 26 of the candidate object 25 in the same scene. Then, the reference object 3a in which various motion characteristics of the object are recorded is stored in the object DB 3, and the object recognition device 4 compares the motion of the candidate object 25 with the motion of the reference object 3a. It is also possible to determine what operation is being performed. Note that the feature amount of the motion in this case does not need to be specified up to the type of the candidate object 25, but the direction and amount of movement in the real three-dimensional space 26, such as the appearance and exit of the processing frame 22, the rotational motion, etc. It only needs to be identifiable.

  Alternatively, according to the present embodiment, the object extraction device 2 increases the feature points 23 of the feature point group 24 extracted as the candidate objects 25 in the processing frame 22 by performing the image quality enhancement processing on the moving image data 21.

  For example, in the image quality enhancement processing, learning data of various color distributions obtained by sampling the difference between the high image quality data and the low image quality data for each of a plurality of template images having various color distributions for each minute region is stored in advance. Then, the processing frame 22 is improved in image quality using learning data that is most suitable for each minute region of the processing frame.

  With such a configuration, candidate objects 25 having more feature points 23 can be extracted, and more feature points 23 are used in the object recognition process, so that the recognition accuracy of candidate objects 25 can be improved. it can.

  Further, in the present embodiment, as described above, the object recognition system 1 includes the object extraction device 2 and a plurality of reference objects 3a for recognizing the candidate object 25, the original image of each reference object 3a, and each reference object 3a. This is a database that is stored together with attached information related to the reference object 3a, and a feature point extraction process, a depth detection process, a three-dimensional space estimation process, and an object extraction process for the original image of each reference object 3a. An object DB 3 storing each reference object 3a extracted as a feature point group having a feature quantity in the dimensional space, and a plurality of candidate objects 25 extracted by the object extracting device 2 are stored in the object DB 3 Which of the reference objects 3a corresponds to It includes an object recognition apparatus 4 that performs object recognition processing, the. When the object recognition device 4 determines that the candidate object 25 corresponds to one reference object 3a among the plurality of reference objects 3a, the object recognition device 4 uses the object information generated based on the attached information of the one reference object 3a as a candidate. It is added to the object 25.

  With such a configuration, since the candidate object 25 extracted with high accuracy is compared with the reference object 3a extracted with high accuracy, the candidate object 25 can be recognized and processed with high accuracy. And since the refined object information is generated for the candidate object 25 based on the attached information of the reference object 3a, the object information for specifying the candidate object 25 more appropriately is added, and the convenience of the object information is improved. Can do.

  Further, according to the present embodiment, in the object recognition system 1, the object recognition processing includes the color distribution in the feature point group 24 and the processing frame 22 of the candidate object 25, the feature point group of the reference object 3 a, and the color distribution in the original image. And is done by comparing

  With such a configuration, a high-accuracy identification amount that identifies the candidate object 25 is compared with a high-accuracy identification amount that identifies the reference object 3a, thereby realizing a highly accurate recognition process for the candidate object 25. Yes.

  Further, according to the present embodiment, in the object recognition system 1, the object DB 3 classifies the plurality of reference objects 3a based on the respective attached information, and two or more reference objects 3a having common attached information. Are classified and stored in a common category using the common attached information as classification information.

  With such a configuration, the object DB 3 can easily search for the reference object 3a based on the attached information, and can also easily search based on the category.

  Further, according to the present embodiment, the object recognition system 1 selects a candidate object 25 determined to correspond to one reference object 3a among the plurality of reference objects 3a as a category in which the one reference object 3a is classified. Stored in the object DB 3 as a new reference object 3a.

  The object recognition system 1 stores the candidate object 25 determined not corresponding to any of the plurality of reference objects 3a in the object DB as a reference object 3a of a new category into which the candidate object 25 is classified.

  With such a configuration, the candidate object 25 as a result of the object recognition process can be used as learning data for the reference object 3a. Further, by performing object recognition processing of various moving image data, it is possible to increase learning data based on the candidate objects 25 of highly accurate recognition results. Therefore, the machine learning of the object recognition system 1 becomes more excellent, and the accuracy and efficiency of the object recognition process can be improved.

  Further, in the present embodiment, as described above, the metadata creation system includes the object recognition system 1 described above, and includes the moving image information of the predetermined moving image data 21 and the plurality of processing frames 22 constituting the predetermined moving image data 21. And the object information of the candidate object 25 extracted and recognized from each of the plurality of processing frames 22 are totaled, and the metadata 32a related to the moving image data 21 is created based on the totaled result.

  With this configuration, the metadata 32a is created using frame information in which the candidate object 25 is recognized with high accuracy and object information that more appropriately identifies the candidate object 25. Therefore, the processing frame 22 in which the candidate object 25 appears is appropriately described in the metadata 32a, and the explanation of the candidate object 25 is appropriately described, so that the content of the moving image data 21 is appropriately reflected. It will be. As a result, the utility value of the metadata 32a can be improved, and further, the use and spread of the moving image data 21 corresponding to the metadata 32a can be improved.

  In the present embodiment, the configuration in which the object extraction device 2 is applied to the object recognition system 1 has been described. However, the configuration is not limited to this configuration. For example, in another embodiment, the object extraction device 2 extracts a candidate object 25 of a subject from an image obtained by photographing the subject, and stereoscopically displays a stereoscopic image based on the feature amount of the candidate object 25 in a three-dimensional space. The present invention can also be applied to a stereoscopic display system that displays on a screen.

1 Object Recognition System 2 Object Extractor 3 Object Database (DB)
3a Reference object 4 Object recognition device 5 Network 6 Movie database (DB)
7 Image collection device 8 Extraction processing database (DB)
DESCRIPTION OF SYMBOLS 10 Control part 11 Memory | storage part 12 Communication part 13 Movie input part 14 Frame acquisition part 15 Image quality improvement part 16 Frame adjustment part 17 Feature point extraction part 18 Depth detection part 19 3D space estimation part 20 Object extraction part 21 Movie data 22 Process Frame 23 Feature point 24 Feature point group 25 Candidate object 26 Real 3D space 30 Metadata creation system 31 Metadata creation device 32 Metadata database (DB)
32a Metadata 40 Metadata distribution system 41 Metadata distribution device 42 Viewer terminal

Claims (12)

Among the plurality of frames of the two-dimensional image constituting the moving image data, the feature point extraction process is performed on the processing frame to be extracted, and a plurality of feature points of the processing frame are extracted. Detecting a first feature on the image;
Performing a depth detection process on the processing frame to detect a relative depth from surrounding feature points for each feature point of the processing frame;
Performing a three-dimensional space estimation process on the processing frame, estimating a real three-dimensional space in the processing frame based on at least the first feature amount and the depth of each of a plurality of feature points of the processing frame; Detecting a second feature amount in the real three-dimensional space of a plurality of feature points of the frame;
Performing object extraction processing based on at least the second feature amount and color distribution of each of the plurality of feature points of the processing frame to detect a feature point group consisting of a set of two or more feature points of the processing frame; An object extraction apparatus, wherein the feature point group having the feature amount in the real three-dimensional space is extracted as a candidate object of the processing frame. In the case where there are two or more common processing frames that constitute the same scene in the time axis among the plurality of processing frames constituting the moving image data,
When performing the depth detection process, the three-dimensional space estimation process, and the object extraction process for each of the two or more common processing frames, a feature point group having a feature amount in the real three-dimensional space, 2. The object extraction apparatus according to claim 1, wherein a feature point group detected in common in the two or more common processing frames is extracted as a candidate object of the same scene.   When the feature point extraction processing is performed on each of the two or more common processing frames, a plurality of feature points extracted from one common processing frame and a plurality of feature points extracted from another common processing frame The object extraction apparatus according to claim 2, wherein the feature points of the one common processing frame are increased using a difference from the feature points of the object.   The candidate object of the same scene has a time displacement amount on the same scene of the feature amount on the real three-dimensional space in addition to the feature amount on the real three-dimensional space. 3. The object extraction device according to 3.   The object according to any one of claims 1 to 4, wherein a feature point of a feature point group extracted as the candidate object in the processing frame is increased by performing high-quality processing on the moving image data. Extraction device.   In the image quality enhancement process, learning data of various color distributions obtained by sampling the difference between the high image quality data and the low image quality data for each of a plurality of template images having various color distributions for each minute region is stored in advance. The object extraction device according to claim 5, wherein the processing frame is improved in image quality using the learning data most suitable for each minute region of the processing frame. The object extraction device according to any one of claims 1 to 6,
A database for storing a plurality of reference objects for recognizing the candidate objects together with an original image of each reference object and attached information related to each reference object, wherein the feature points with respect to the original image of each reference object Each reference object extracted as a feature point group having a feature amount in the actual three-dimensional space of the original image by the extraction process, the depth detection process, the three-dimensional space estimation process, and the object extraction process is stored. An object database
An object recognition device that performs object recognition processing on which the candidate object extracted by the object extraction device corresponds to which of the plurality of reference objects stored in the object database;
When the object recognition device determines that the candidate object corresponds to one reference object of the plurality of reference objects, the object recognition device generates object information generated based on the attached information of the one reference object. An object recognition system characterized by being added to the object.   The object recognition process is performed by comparing a feature point group of the candidate object and a color distribution in the processing frame with a feature point group of the reference object and a color distribution in an original image. 8. The object recognition system according to 7.   The object database classifies the plurality of reference objects on the basis of the respective accessory information. For two or more reference objects having common accessory information, the common reference information using the common accessory information is classified. 9. The object recognition system according to claim 7, wherein the object recognition system is classified and stored.   The candidate object determined to correspond to one reference object among the plurality of reference objects is stored in the object database as a new reference object of a category into which the one reference object is classified. The object recognition system of any one of Claims 7-9.   11. The candidate object that is determined not to correspond to any of the plurality of reference objects is stored in the object database as a reference object of a new category into which the candidate object is classified. Object recognition system. The object recognition system according to any one of claims 7 to 11, comprising:
Moving image information of the predetermined moving image data, frame information of the plurality of processing frames constituting the predetermined moving image data, and the object information of the candidate object extracted and recognized from each of the plurality of processing frames. A metadata creation system comprising: summing up and creating metadata relating to the moving image data based on the summation result.

Images