KR102384419B1 - Method, system and non-transitory computer-readable recording medium for estimating information about objects - Google Patents

Abstract

According to an aspect of the present invention, there is provided a method for estimating information about an object, when a photographed image is obtained, an object of interest is detected from the photographed image, and the object of interest is stored in a comparison object pool for the detected object of interest. deriving a candidate object from among included comparison target objects, and based on identification information on the object of interest determined with reference to the derived identification information on the candidate object and state information obtained with respect to the detected object of interest A method is provided that includes estimating a price of a detected object of interest.

Description

A method for estimating information about an object, a system and a non-transitory computer-readable recording medium

The present invention relates to a method, a system, and a non-transitory computer-readable recording medium for estimating information about an object.

In recent years, as goods transactions between individuals (eg, used goods transactions) become more active, the demand for services that support such transactions in a simple and reliable way is increasing.

As an example of the related art, the technology disclosed in Korean Patent Publication No. 10-1678970 may be given as an example, and according to this, receiving a photographed image including a partial image and an entire image of an article; analyzing the photographed image; extracting feature point information of the captured image based on the analysis result of the captured image; determining the authenticity and appearance grade of the article by comparing the feature point information with reference information about the article; and calculating an evaluation amount of the article based on the authenticity and appearance grade of the article, and after analyzing the photographed image, deriving a product model name for the article; And based on the product model name, further comprising the step of searching for product information of the product in an external server or storage unit, wherein the product information includes the product price, manufacturer, seller, It includes at least one of vendor, release year, production year, color, brand, model, grade, weight, material, shape, manufacturer, and inventory amount information for each vendor, and the step of receiving the photographed image includes: The method further comprising: receiving a photographed image of at least one of an appraisal, a guarantee card, and a warranty card, wherein the extracting of the feature point information comprises at least one of a bibliography of the article, a warranty, an appraisal, a guarantee card, and a warranty card It characterized in that it further comprises the step of extracting feature point information about at least one of a serial number, a reference number, a code number, a barcode, an image, a shape, and a character included in the photographed image.

However, according to the techniques introduced so far, including the prior art as described above, in order to accurately identify an object included in a photographed image, a warranty card in which information about the object is described in detail is required, or, in particular, in the case of used luxury goods, There was a limit that an expert's appraisal was needed to accurately calculate the value of the object.

Accordingly, the present inventor(s) proposes a technique for estimating information of an object detected from a photographed image simply and accurately, and evaluating the value of the object based on this.

Registered Patent Publication No. 10-1678970 (2016. 11. 24)

An object of the present invention is to solve all of the problems of the prior art described above.

In addition, in the present invention, when a captured image is obtained, an object of interest is detected from the captured image, and a candidate object is derived from among comparison objects included in a comparison object pool with respect to the detected object of interest, and , estimating the price of the detected object of interest based on the identification information on the object of interest determined with reference to the derived identification information on the candidate object and the state information obtained with respect to the detected object of interest for other purposes.

In addition, another object of the present invention is to conveniently and accurately estimate information on an object detected in a photographed image and evaluate the value of the object based on this to support convenient transaction of goods between users.

A representative configuration of the present invention for achieving the above object is as follows.

According to an aspect of the present invention, when a captured image is obtained, an object of interest is detected from the captured image, and a candidate object is derived from among comparison objects included in a comparison object pool related to the detected object of interest. estimating the price of the detected object of interest based on identification information on the object of interest determined with reference to the derived identification information on the candidate object and state information obtained with respect to the detected object of interest A method comprising

According to another aspect of the present invention, when a captured image is obtained, an object of interest manager detects an object of interest from the captured image and obtains state information about the detected object of interest, and a comparison target object related to the detected object of interest Based on the obtained state information and the identification information on the object of interest determined by referring to the candidate object management unit for deriving a candidate object from among the comparison objects included in the pool, and identification information on the derived candidate object There is provided a system including a price management unit estimating the price of the detected object of interest.

In addition to this, another method for implementing the present invention, another system, and a non-transitory computer-readable recording medium for recording a computer program for executing the method are further provided.

According to the present invention, when a photographed image is obtained, an object of interest is detected from the photographed image, and a candidate object is derived from among comparison objects included in a comparison object pool with respect to the detected object of interest, To estimate the price of the detected object of interest based on the identification information on the object of interest determined with reference to the derived identification information on the candidate object and the state information obtained with respect to the detected object of interest do.

In addition, according to the present invention, it is possible to conveniently and accurately estimate information on an object detected in a photographed image, and evaluate the value of the object based on this, thereby supporting convenient transaction of goods between users.

1 is a diagram illustrating a schematic configuration of an entire system for estimating information about an object according to an embodiment of the present invention.
2 is a diagram illustrating in detail an internal configuration of an object information estimation system according to an embodiment of the present invention.
3 to 5, 9 and 10 are diagrams exemplarily illustrating a process of detecting an object of interest according to an embodiment of the present invention.
6 to 8 are diagrams exemplarily illustrating a process for calculating a degree of similarity between an object of interest and a comparison object according to an embodiment of the present invention.
11 to 12 are diagrams exemplarily illustrating a process of obtaining partial identification information from a user with respect to an object of interest according to an embodiment of the present invention.
13 is a diagram exemplarily illustrating a process in which a detection region and state information are obtained from a user with respect to an object of interest according to an embodiment of the present invention.
14 to 17 are diagrams exemplarily illustrating a process of deriving a candidate object according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the present invention. In addition, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention should be taken as encompassing the scope of the claims and all equivalents thereto. In the drawings, like reference numerals refer to the same or similar elements throughout the various aspects.

Hereinafter, various preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

In this specification, an embodiment of the object information estimation service is mainly described focusing on price estimation of a used bag, but the service is not limitedly applicable only to a specific type of item, and all types of transactions available such as watches, ornaments, etc. It should be understood that it can be applied to the articles of

Whole system configuration

1 is a diagram showing a schematic configuration of an entire system for estimating information about an object according to an embodiment of the present invention.

As shown in FIG. 1 , the entire system according to an embodiment of the present invention may include a communication network 100 , an object information estimation system 200 , and a device 300 .

First, the communication network 100 according to an embodiment of the present invention may be configured regardless of communication aspects such as wired communication or wireless communication, and includes a local area network (LAN), a metropolitan area network (MAN) ), a wide area network (WAN), etc. may be configured as various communication networks. Preferably, the communication network 100 as used herein may be a well-known Internet or World Wide Web (WWW). However, the communication network 100 is not necessarily limited thereto, and may include a known wired/wireless data communication network, a known telephone network, or a known wired/wireless television communication network in at least a part thereof.

For example, the communication network 100 is a wireless data communication network, such as Wi-Fi communication, Wi-Fi Direct communication, Long Term Evolution (LTE) communication, 5G communication, Bluetooth communication (Low Energy Bluetooth (BLE)) (including Bluetooth Low Energy) communication), infrared communication, ultrasonic communication, and the like, may be implemented in at least a part thereof. As another example, the communication network 100 may be an optical communication network that implements at least a part of a conventional communication method such as LiFi (Light Fidelity).

Next, when a captured image is obtained, the object information estimation system 200 according to an embodiment of the present invention detects an object of interest from the captured image, and a comparison target object pool for the detected object of interest ), based on identification information on the object of interest determined by deriving a candidate object from among the comparison target objects included in Thus, the function of estimating the price of the object of interest detected above may be performed.

The configuration and function of the object information estimation system 200 according to the present invention will be described in detail through the following detailed description.

Next, the device 300 according to an embodiment of the present invention is a digital device including a function to communicate after accessing the object information estimation system 200, a smart phone, a tablet, a smart watch, a smart band, Any digital device equipped with memory means such as smart glasses, desktop computer, notebook computer, workstation, PDA, web pad, mobile phone, etc. and equipped with a microprocessor and equipped with computing power is adopted as the device 300 according to the present invention. can be

In particular, the device 300 may include an application (not shown) that supports a user to receive a service, such as object information estimation, from the object information estimation system 200 . Such an application may be downloaded from the object information estimation system 200 or an external application distribution server (not shown). On the other hand, the nature of this application is the object of interest management unit 210, the candidate object management unit 220, the price management unit 230, the communication unit 240 and the control unit 250 of the object information estimation system 200 to be described later and overall can be similar to Here, at least a part of the application may be replaced with a hardware device or a firmware device capable of performing substantially the same or equivalent function as the application, if necessary.

Composition of object information estimation system

Hereinafter, the internal configuration of the object information estimation system 200 that performs an important function for the implementation of the present invention and the function of each component will be described.

2 is a diagram illustrating in detail the internal configuration of the object information estimation system 200 according to an embodiment of the present invention.

As shown in FIG. 2 , the object information estimation system 200 according to an embodiment of the present invention includes an object of interest management unit 210 , a candidate object management unit 220 , a price management unit 230 , a communication unit 240 and The controller 250 may be included.

According to an embodiment of the present invention, the object of interest management unit 210 , the candidate object management unit 220 , the price management unit 230 , the communication unit 240 , and the control unit 250 include at least some of them in an external system (not shown). It may be a program module that communicates with Such a program module may be included in the object information estimation system 200 in the form of an operating system, an application program module, or other program modules, and may be physically stored in various known storage devices. Also, such a program module may be stored in a remote storage device capable of communicating with the object information estimation system 200 . Meanwhile, such a program module includes, but is not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks or execute specific abstract data types according to the present invention.

On the other hand, although described above with respect to the object information estimation system 200, these descriptions are exemplary, and at least some of the components or functions of the object information estimation system 200 may be used as the device 300 or the server (not shown) as needed. It will be apparent to those skilled in the art that it may be implemented within an external system (not shown) or implemented within an external system (not shown).

First, the object-of-interest manager 210 according to an embodiment of the present invention may perform a function of detecting an object of interest in a photographed image when a photographed image is obtained.

Specifically, the object of interest management unit 210 according to an embodiment of the present invention, when a captured image is obtained, R-CNN (Region-based Convolutional Neural Networks), YOLO (You Only Look Once), SSD (Single Shot multibox) Detector) may be used to detect at least one object of interest in the captured image using an object recognition model. However, the object recognition model according to an embodiment of the present invention is not limited to those listed above, and may be variously changed within a range that can achieve the object of the present invention.

Continuing, the detection of the object of interest by the object of interest management unit 210 according to an embodiment of the present invention includes the category of the object of interest and the location of the object of interest on the photographed image (or the location of the object of interest on the photographed image. region) and calculating the reliability of the recognized information. Here, according to an embodiment of the present invention, recognizing the category of the corresponding object of interest includes identification information about the corresponding object of interest (ie, information necessary for estimating the price of the corresponding object of interest, such as a specific model name, of the corresponding interest It can be distinguished from determining information that can be specified by distinguishing an object from other objects.

More specifically, according to an embodiment of the present invention, the category of the object of interest may mean a higher level concept including identification information on the object of interest. For example, according to an embodiment of the present invention, when the object of interest is a bag, bags, handbags, tote bags, etc. may correspond to the category of the object of interest, and specific model information such as 'Louis Vuitton Neo Alma BB', etc. This may correspond to identification information about the object of interest.

In addition, the detection result of the object-of-interest by the object-of-interest manager 210 according to an embodiment of the present invention may include a main category and a sub-category regarding the object of interest detected in the captured image.

Specifically, according to an embodiment of the present invention, a plurality of categories may be recognized with respect to one object of interest, such as a main category and a sub-category. For example, when a handbag is detected as an object of interest in the captured image, a main category related to the object of interest may be a bag, and a sub-category may be a handbag. As another example, when a cat-shaped bag or a bag with a cat pattern on the surface is detected as the object of interest in the photographed image, the main category for the object of interest may be a bag, and the sub-category may be an animal (or cat). there is.

However, the category according to an embodiment of the present invention is not limited to that described above, and may be variously changed within the scope that can achieve the object of the present invention. For example, according to an embodiment of the present invention, brand information such as Chanel and Louis Vuitton may be included in the above category.

In addition, in the detection of at least one object of interest by the object of interest manager 210 according to an embodiment of the present invention, at least one boundary region (eg, a bounding box surrounding the object of interest) of the object of interest is selected. A setting process may be included, and through this, the location of at least one object of interest (or a region occupied by the object of interest) on the captured image may be recognized. However, according to an embodiment of the present invention, the process of setting the boundary region above is not necessarily distinct from the process of recognizing the category of the object of interest and calculating the reliability of the recognized information, and is substantially one process. It should be understood that it may proceed as

Meanwhile, according to an embodiment of the present invention, the object of interest may mean a target whose price can be estimated by the price management unit 230 according to an embodiment of the present invention.

For example, among at least one object detected in a photographed image by the object of interest management unit 210 according to an embodiment of the present invention, the price estimation function by the price management unit 230 according to an embodiment of the present invention is At least one supported object may be determined as an object of interest. In this case, the object-of-interest management unit 210 according to an embodiment of the present invention determines that the category of the detected object is substantially the same as the category of the specific object included in the comparison object pool. The detected object of may be determined as the object of interest. Meanwhile, details of the comparison target object pool will be described later.

And, the object of interest management unit 210 according to an embodiment of the present invention, when partial identification information is obtained from the user with respect to the object for which the price is to be estimated, based on at least one of the obtained information and the detection result to determine the object of interest.

Specifically, the object-of-interest management unit 210 according to an embodiment of the present invention may obtain partial identification information from the user regarding the object of interest. According to an embodiment of the present invention, partial identification information obtained from the user with respect to the object of interest may mean partial information for identifying the object of interest. For example, according to an embodiment of the present invention, when the object of interest is a bag, partial identification information obtained from the user with respect to the object of interest includes a category (eg, handbag), brand, material, material, pattern , color, size (including actual size), presence or absence of a cover, presence or absence of a bottom surface, presence or absence of a handle, the number of handles, guarantee card number, product code number, product name, engraving number, year number, etc. may be included. And, according to an embodiment of the present invention, the partial identification information obtained from the user with respect to the object of interest may be obtained in advance before the user captures the object, or may be obtained immediately after the user captures the object. Also, it may be obtained after the object of interest is detected in the captured image.

3 is a diagram exemplarily illustrating a process of detecting an object of interest according to an embodiment of the present invention.

For example, referring to FIG. 3B , when the handbag 321 and the bag 322 are detected from the photographed image by the object-of-interest management unit 210 according to an embodiment of the present invention, the object of interest If information that there is a handle is obtained from the user as partial identification information about the , the object of interest may be determined as the handbag 321 .

Continuing, according to an embodiment of the present invention, partial identification information obtained from the user with respect to the object of interest may be obtained step by step. For example, when the object of interest is a Louis Vuitton bag, the object of interest manager 210 according to an embodiment of the present invention provides the user with upper images (eg, Damier, monogram, etc.) related to the pattern of the Louis Vuitton bag. ) is provided, and when a specific image (eg, Damier) is selected by the user above among the upper images, the lower images of the upper image (eg, Damier Eben, Damier Azur, etc.) It is possible to obtain partial identification information from the user with respect to the object of interest by providing the .

11 to 12 are diagrams exemplarily illustrating a process of obtaining partial identification information from a user with respect to an object of interest according to an embodiment of the present invention.

Referring to FIGS. 11A and 11B , when the object of interest is a bag, the object of interest manager 210 according to an embodiment of the present invention receives the brand and material (or pattern) of the bag from the user. information can be obtained.

Referring to FIG. 12 , when the object of interest is a bag, the object of interest manager 210 according to an embodiment of the present invention may obtain information about the size of the bag from the user. For example, the object of interest management unit 210 according to an embodiment of the present invention may obtain information about the size of the corresponding bag by allowing the user to select any one of the size categories determined according to the brand of the corresponding bag. There is (Fig. 12 (a)).

Also, the object of interest manager 210 according to an embodiment of the present invention may obtain information about the size of the corresponding bag by allowing the user to directly input the size of the corresponding bag (FIG. 12(b)). In this case, the object-of-interest management unit 210 according to an embodiment of the present invention may provide the user with a function that supports measuring the size of the corresponding bag. For example, a ruler may be displayed on the display of the user device 300 or the size of the corresponding bag estimated based on a distance between the camera and the corresponding bag may be provided to the user.

However, the type, acquisition time, and acquisition method of partial identification information acquired from the user with respect to the object of interest according to an embodiment of the present invention are not limited to those described above, and are within the scope capable of achieving the object of the present invention. can be changed in various ways.

3 to 5, 9 and 10 are diagrams exemplarily illustrating a process of detecting an object of interest according to an embodiment of the present invention.

For example, referring to FIG. 3A , the object-of-interest manager 210 according to an embodiment of the present invention may set a boundary area 311 for an object detected in a photographed image, and the boundary A category of an object included in the area 311 may be recognized as a handbag. In addition, the object of interest management unit 210 according to an embodiment of the present invention determines that the above-recognized information, that is, the handbag, is substantially the same as the category of a specific object included in the comparison object pool, the above detected An object may be determined as an object of interest. Here, in determining the object of interest as described above, the object of interest manager 210 according to an embodiment of the present invention determines the reliability of the recognized information and the size (or area, pixels of the boundary area 311 ). size, etc.), and should be understood similarly even if not separately mentioned in other embodiments described herein.

For another example, referring to FIG. 3B , the object-of-interest management unit 210 according to an embodiment of the present invention includes a plurality of boundary regions 321 and 322 for one object detected in a photographed image. can be set. And, the object-of-interest management unit 210 according to an embodiment of the present invention may recognize a category of an object included in the first boundary area 321 among the plurality of boundary areas 321 and 322 as a handbag, A category of an object included in the second boundary area 322 may be recognized as a bag. And, the object of interest management unit 210 according to an embodiment of the present invention, when it is determined that the recognized information, that is, the handbag and the bag, is substantially the same as the category of a specific object included in the comparison object pool, the above The detected object may be determined as the object of interest. Here, the object-of-interest manager 210 according to an embodiment of the present invention may determine, as the category of the object of interest, a category with high reliability among the recognized information, that is, a handbag and a bag. In addition, the object-of-interest management unit 210 according to an embodiment of the present invention determines the main category as a bag in consideration of the above recognized information, that is, the reliability and/or inclusion relationship of the handbag and the bag, and selects the sub-category. You can also decide with a handbag.

For another example, referring to FIG. 9A , the object-of-interest manager 210 according to an embodiment of the present invention provides a plurality of boundary regions ( 911 and 912) can be set. And, the object-of-interest management unit 210 according to an embodiment of the present invention may recognize the category of the object included in the first boundary area 911 among the plurality of boundary areas 911 and 912 as a bag, The category of the object included in the second boundary area 912 may be recognized as an animal (or cat). In addition, the object of interest management unit 210 according to an embodiment of the present invention determines that the above recognized information, that is, a bag and an animal (or cat), is substantially the same as the category of a specific object included in the comparison object pool. If it is determined, the detected object above may be determined as an object of interest. Here, the object of interest management unit 210 according to an embodiment of the present invention includes the above recognized information, that is, the reliability of the bag and the animal (or cat) and/or the inclusion between the plurality of boundary regions 911 and 912 . Considering the relationship, the main category may be determined as a bag, and the sub-category may be determined as an animal (or a cat).

Continuing, for example, referring to FIG. 9B , the object-of-interest manager 210 according to an embodiment of the present invention captures a small area 912 among the plurality of boundary areas 911 and 912 as a captured image. The sub-category of the object of interest is selected according to the result of cropping at 910 and detecting 921 again on the cropped image 920 (eg, whether the reliability is above a predetermined level). It can be determined as an animal (or cat). Alternatively, the object-of-interest management unit 210 according to an embodiment of the present invention sets one boundary area 911 for a first object detected in the photographed image 910 , and By dividing the included image at equal intervals or performing segmentation, a second object corresponding to a part of the above object may be detected ( 921 ). Also, the object-of-interest manager 210 according to an embodiment of the present invention may determine the sub-category of the object of interest to be an animal (or a cat) as a result of detecting the second object.

For another example, referring to FIG. 10 , the object-of-interest manager 210 according to an embodiment of the present invention may set a boundary area 1021 with respect to an object detected in a photographed image 1020 , and Although the photographed object is a cat-shaped bag, it may be assumed that the category of the object included in the boundary area 1021 is recognized as an animal (or cat). In this case, according to an embodiment of the present invention, the object detected above because there is no object having the category of animal (or cat) in the object pool to which the recognized information, ie, animal (or cat), is compared. may not be determined as an object of interest. At this time, if there is information that the category of the object of interest is the bag 1010 as partial identification information obtained from the user with respect to the object of interest, the object of interest management unit 210 according to an embodiment of the present invention, the recognized information That is, in consideration of the reliability of the animal (or cat) and/or partial identification information 1010 obtained from the user with respect to the object of interest, the main category may be determined as a bag, and the sub-category may be determined as the animal (or cat). there is.

As another example, referring to FIGS. 4A and 4B , the object-of-interest management unit 210 according to an embodiment of the present invention may include boundary regions 411 to 413 and 421 to 424 may be set, and it may be determined whether a plurality of objects included in the boundary regions 411 to 413 and 421 to 424 respectively correspond to the object of interest. And, if the plurality of objects all correspond to the object of interest, the candidate object manager 220 according to an embodiment of the present invention may derive a candidate object for each of the plurality of objects. Meanwhile, details on the function of the candidate object management unit 220 will be described later.

Meanwhile, the photographed image according to an embodiment of the present invention may be an image photographed based on a photographing guideline.

Specifically, the object-of-interest management unit 210 according to an embodiment of the present invention may induce a user who intends to use the object information estimation system 200 to photograph the object of interest according to the instruction by providing a photographing instruction. there is.

For example, referring to FIG. 5A , the object-of-interest manager 210 according to an embodiment of the present invention sets an area where the object of interest is to be located in the form of a guideline 512 to the user device ( 300), it is possible to induce the above user to correctly photograph the object of interest. Further, the object-of-interest manager 210 according to an embodiment of the present invention may increase the reliability of the detection result by detecting the object of interest in the region inside the guide line.

For another example, the object of interest management unit 210 according to an embodiment of the present invention displays on the display of the user device 300 which object is currently recognized among objects displayed through the camera (eg, For example, by displaying the color of the recognized object differently from the original color), it is possible to induce the user to correctly photograph the object of interest. And, the object-of-interest manager 210 according to an embodiment of the present invention may increase the reliability of the detection result by detecting the object of interest on the recognized object.

As another example, the object-of-interest manager 210 according to an embodiment of the present invention may provide information on a method of photographing an object of interest to the user. For example, the object-of-interest management unit 210 according to an embodiment of the present invention may include a surface (front, rear, bottom, inner surface, etc.) of the object of interest to be photographed, the order in which each surface is photographed, and photographing. It is possible to provide the user with information about an appropriate area that the object of interest should occupy on the image, a lighting level required for photographing, and the like.

Meanwhile, the object-of-interest management unit 210 according to an embodiment of the present invention performs a function of pre-processing a captured image before the derivation of the candidate object is performed by the object-of-interest management unit 210 according to an embodiment of the present invention. can be done

Specifically, the object of interest manager 210 according to an embodiment of the present invention may pre-process the captured image before detecting the object of interest in the captured image. According to an embodiment of the present invention, such pre-processing includes adjustment of the size, resolution, capacity, etc. of the photographed image, cropping, histogram equalization, normalization, color correction, and lighting. Illumination correction, foreground extraction (background removal), segmentation, padding, and the like may be included. However, the pretreatment according to an embodiment of the present invention is not limited to those listed above, and may be variously changed within the range that can achieve the object of the present invention.

For example, referring to FIGS. 5A and 5B , it may be assumed that an object of interest is photographed while being included in the guideline 512 ( 511 ). In this case, the object-of-interest management unit 210 according to an embodiment of the present invention crops the captured image according to the above guideline 512 , and detects the object of interest from the cropped captured image 521 . (522) You can. Continuing, for example, referring to FIGS. 5 ( b ) and 5 ( c ), the captured image is cropped according to the guidelines, the cropped image 521 includes the padding 533 , and the An object of interest may be detected 532 from the captured image 531 including padding. Here, it should be understood that the padding may be variously combined and/or changed to be included in the photographed image without being limited by color, pattern, position, and size.

On the other hand, the object of interest management unit 210 according to an embodiment of the present invention, after detecting an object in the captured image (which may be a pre-processed image), pre-process the captured image (pre-processing the pre-processed image again) can.) can be done. For example, the object-of-interest management unit 210 according to an embodiment of the present invention pre-processes the captured image when the reliability of the object detected in the captured image is less than or equal to a predetermined level, and returns the object to the pre-processed image. can be detected. Then, the object-of-interest manager 210 according to an embodiment of the present invention may determine the object as the object of interest in consideration of whether the reliability of the object detected as a result of the re-detection is equal to or greater than a predetermined level.

Also, the object-of-interest management unit 210 according to an embodiment of the present invention detects the object of interest in the photographed image that has been subjected to at least one of the pre-processing as described above and the interest in the photographed image that has not been pre-processed as described above. An object of interest may be determined by referring to at least one of the results of detecting the object.

For example, referring to FIG. 3B , the handbag 321 and the bag 322 are objects of interest in the captured image that has not been pre-processed by the object-of-interest manager 210 according to an embodiment of the present invention. It can be assumed that it is detected as In this case, referring to FIGS. 5B and 5C , the object-of-interest management unit 210 according to an embodiment of the present invention provides handbags 522 and 532 in the pre-processed captured images 521 and 531, respectively. ) is detected as the object of interest, it may be determined that the reliability of the handbag is higher than the reliability of the bag, and accordingly, the handbag may be determined as the object of interest.

Meanwhile, the object-of-interest manager 210 according to an embodiment of the present invention may perform the above pre-processing based on metadata included in the captured image.

Specifically, according to an embodiment of the present invention, such metadata includes the location of the photographed image (for example, whether indoors or outdoors, longitude, latitude, etc.), the recording time, and the device ( 300 ), focal length, aperture value, exposure time, ISO speed, whether the flash is activated, the actual size of the object of interest measured using the device 300 , and the like may be included. In addition, the object-of-interest manager 210 according to an embodiment of the present invention may perform pre-processing such as color correction and lighting correction using such metadata.

For example, assuming that information that a photographed image was photographed at noon on a specific date at a specific place in Korea and outdoors is acquired as metadata, the object of interest management unit 210 according to an embodiment of the present invention, Pre-processing of estimating the intensity and direction of sunlight based on the obtained information, and correcting the captured image based on the estimated information may be performed.

However, metadata according to an embodiment of the present invention is not limited to those listed above, and may be variously changed within a range that can achieve the object of the present invention.

Meanwhile, the object-of-interest manager 210 according to an embodiment of the present invention may detect the object of interest from the captured image with reference to a detection area obtained from the user with respect to the object of interest.

Specifically, the object-of-interest management unit 210 according to an embodiment of the present invention, when a captured image is obtained, allows the user to input a detection area indicating where the object of interest is located in the captured image. A detection area can be obtained. In addition, the object of interest manager 210 according to an embodiment of the present invention may increase the reliability of the detection result by detecting the object of interest in the above detection area.

In addition, the detection area obtained from the user as described above is a kind of labeling in learning of an AI-based object-of-interest detection model (not shown) that may be included in the object-of-interest management unit 210 according to an embodiment of the present invention. It can be used as learning data. That is, according to an embodiment of the present invention, an object of interest may be detected from a captured image based on an object of interest detection model that is learned using a detection region obtained from a user with respect to the object of interest as learning data.

On the other hand, the object-of-interest manager 210 according to an embodiment of the present invention is based on an artificial intelligence-based state information estimation model (not shown) that can be included in the object-of-interest management unit 210 according to an embodiment of the present invention. Based on the state information about the object of interest detected in the captured image may be obtained.

Specifically, according to an embodiment of the present invention, the state information on the object of interest may include the location, size, and/or the extent of scratches, stains, and aged parts present in the object of interest. However, the state information about the object of interest according to an embodiment of the present invention is not limited to those listed above, and may be variously changed within a range capable of achieving the object of the present invention.

Continuing, the object-of-interest management unit 210 according to an embodiment of the present invention obtains the above state information by allowing the user to input the state information on the object of interest as above from the captured image when the captured image is obtained. can In addition, the price estimator 230 according to an embodiment of the present invention may estimate the price of the object of interest based on state information obtained from the user regarding the object of interest as described above. Meanwhile, details of the function of the price estimator 230 according to an embodiment of the present invention will be described later.

In addition, the state information obtained from the user as described above may be used as a kind of labeled learning data in learning the state information estimation model that may be included in the object-of-interest management unit 210 according to an embodiment of the present invention. That is, the object-of-interest management unit 210 according to an embodiment of the present invention, based on a state information estimation model that is learned by using state information obtained from a user about the object of interest as learning data, the object of interest detected from the captured image. status information can be obtained.

13 is a diagram exemplarily illustrating a process of obtaining a detection region and state information from a user with respect to an object of interest according to an embodiment of the present invention.

For example, referring to FIG. 13A , the object-of-interest manager 210 according to an embodiment of the present invention provides a detection area ( By allowing the user 1300 to input 1312 , the detection area 1312 may be obtained. According to an embodiment of the present invention, such a detection area may be input by a method in which the user 1300 drags and displays 1312 the area in which the object of interest is located on the screen of the device 300, but this method is limited. it is not going to be

For another example, referring to FIG. 13B , the object-of-interest management unit 210 according to an embodiment of the present invention allows the user to input state information about the object of interest in the captured image, thereby providing the state information ( 1321, 1322 and 1323) can be obtained. According to an embodiment of the present invention, such state information is displayed 1321 by the user 1300 by dragging the area where the scratch, etc. existing in the object of interest is located on the screen of the device 300, or A shape may be input by a method of directly drawing and displaying 1322 by hand, but is not limited thereto. In addition, according to an embodiment of the present invention, the user 1300 may select the information 1323 on the severity of the scratch or the like.

As another example, referring to FIG. 13C , the object-of-interest manager 210 according to an embodiment of the present invention allows the user to input state information about the object of interest in the captured image, thereby providing the state information. (1331 and 1332) can be obtained. According to an embodiment of the present invention, such state information may be input by a method in which the user 1300 selects and displays (1331 and 1332) an area where a spot, etc. existing on an object of interest is located on the screen of the device 300. However, it is not limited to this method. In addition, according to an embodiment of the present invention, the user 1300 can provide information on the size of the stain, such as the stain, to the user 1300, such as images of various types of coins. Means for sizing 1341 and 1342; that is, the size of the blob 1331 is similar to the size of the 500 won coin 1341, and the size of the blob 1332 is similar to the size of the 50 won coin 1342. ) can also be provided.

Meanwhile, the object-of-interest management unit 210 according to an embodiment of the present invention, when an object of interest is detected in the captured image, is included in the comparison target object pool for the object of interest based on the validity of the object of interest. It is possible to perform a function of determining whether to derive a candidate object from among the comparison target objects.

Specifically, the object-of-interest management unit 210 according to an embodiment of the present invention determines the level or presence of validity with respect to the object of interest detected in the captured image, and whether to derive the above candidate object based on the determination can be decided And, when it is determined that the above candidate object is not derived, the object of interest management unit 210 according to an embodiment of the present invention may request the user to provide a correct photographed image.

More specifically, the object-of-interest management unit 210 according to an embodiment of the present invention may include the number of objects of interest detected in the photographed image, the size of the boundary region of the object of interest detected in the photographed image, and the object of interest detected in the photographed image. Shooting with reference to at least one of a relationship between a boundary region of , and a boundary region of another object (including an object of interest and/or an object of interest) detected in the captured image and whether an uninterested object is detected in the captured image It is possible to determine the level or presence of validity with respect to the object of interest detected in the image. Here, according to an embodiment of the present invention, the uninterested object is an object when there is no object having the category of the uninterested object in the comparison object pool, and the price management unit 230 according to an embodiment of the present invention. It may mean an object whose price estimation function is not supported by .

For example, referring to FIG. 4A , it may be assumed that a handbag is detected 412 as an object of interest in a photographed image, and a notebook and a keyboard are detected 411 and 413 as uninterested objects. In this case, the object-of-interest management unit 210 according to an embodiment of the present invention is configured such that the boundary area 412 of the handbag as the object of interest is included in the boundary area 411 of the notebook as the object of interest, or the object of interest is With reference to the detected information 411 and 413 , it may be determined that the validity of the object of interest is below a predetermined level or not, and it may be determined that the candidate object is not derived.

For another example, referring to FIG. 4(b), when a handbag is detected 421 as an object of interest in the photographed image, and a bottle, a box, and a keyboard are detected (422, 423, and 424, respectively) as uninterested objects can be assumed. In this case, the object-of-interest management unit 210 according to an embodiment of the present invention refers to information that the boundary region 421 of the handbag, which is the object of interest, is not included in the boundary regions 422 to 424 of non-interested objects. It may be determined that the validity of the object of interest is greater than or equal to a predetermined level, and it may be determined that the candidate object is derived.

For another example, the object-of-interest management unit 210 according to an embodiment of the present invention may determine that the number of objects of interest detected in the captured image is greater than or equal to a predetermined number, or the size of the boundary region of the object of interest detected in the captured image is at a predetermined level. or less, or when the overlapping area of the boundary area of any two objects of interest detected in the captured image is greater than or equal to a predetermined size, it is determined by referring to such information that the validity of the object of interest is less than or equal to a predetermined level or not, and the candidate object is selected It can be decided not to derive.

Next, when an object of interest is detected in the captured image by the object-of-interest manager 210 according to an embodiment of the present invention, the candidate object manager 220 according to an embodiment of the present invention provides information on the detected object of interest. It is possible to perform a function of deriving a candidate object from among the comparison target objects included in the comparison target object pool (Pool).

Specifically, according to an embodiment of the present invention, the comparison target object may mean an object that can be compared with the object of interest in order to determine identification information about the object of interest. And, the comparison target object pool according to an embodiment of the present invention may mean a database related to the comparison target object. According to an embodiment of the present invention, in such a comparison object pool, a category, brand, material, material, pattern, color, size (including actual size), Information about the shape, product code number, product name, official price (the price of a new product), and the price of a used product may be included. In addition, at least a portion of information included in the comparison object pool may be collected in real time through web crawling or the like.

For example, when the object of interest according to an embodiment of the present invention is a bag, a category (eg, handbag) for each of the various bags along with images of various bags in the comparison object pool related to the bag, Information regarding brand, material, material, pattern, color, size (including actual size), presence or absence of a cover, presence or absence of a bottom surface, presence or absence of a handle, number of handles, and the like may be included.

However, the information included in the comparison object pool according to an embodiment of the present invention is not limited to the above-listed ones, and may be variously changed within the scope capable of achieving the object of the present invention.

Continuing, the candidate object manager 220 according to an embodiment of the present invention calculates a degree of similarity between the comparison target included in the comparison target pool for the target of interest and the target of interest detected in the captured image, and the calculation A candidate object may be derived based on the similarity. For example, the candidate object manager 220 according to an embodiment of the present invention may derive a predetermined number of comparison target objects in the order of the above similarity as the candidate objects. According to an embodiment of the present invention, the similarity may be calculated using a distance value between feature vectors, a K-Nearest Neighbors (KNN) algorithm, or the like, but is not limited thereto.

Meanwhile, before calculating the above similarity, the candidate object manager 220 according to an embodiment of the present invention may perform pre-processing on a region including the above object of interest on the above photographed image. According to an embodiment of the present invention, since this pre-processing may be performed in a similar manner to the pre-processing of the photographed image described in detail above, the detailed description will be omitted.

Continuing, according to an embodiment of the present invention, the candidate object is at least one comparison target object having a high probability of corresponding to the object of interest detected in the photographed image among the comparison target objects included in the comparison target object pool related to the object of interest. can mean According to an embodiment of the present invention, when only one such candidate object is derived (for example, when a difference in probability between a candidate object having the highest probability corresponding to the object of interest and a candidate object having the next highest probability is greater than or equal to a predetermined level) etc.), but a case in which a plurality of candidate objects are derived is also possible. Then, the candidate object management unit 220 according to an embodiment of the present invention, when the derived candidate object is derived as described above, an image of the candidate object and identification information of the candidate object (eg, model name, brand, etc.) can be provided to users.

Continuing, when an object of interest is detected in the captured image by the object-of-interest manager 210 according to an embodiment of the present invention, the candidate object manager 220 according to an embodiment of the present invention provides information on the detected object of interest. Information about at least one of a characteristic, a shape, a pattern, and a color may be derived, respectively. And, the candidate object management unit 220 according to an embodiment of the present invention, each of the information about at least one of the derived features, shapes, patterns, and colors, and each of the information about the object of interest is included in the object pool for comparison. By comparing corresponding information of the comparison target object with each other, a degree of similarity between the comparison target object and the detected object of interest may be calculated.

Here, according to an embodiment of the present invention, information about a feature derived with respect to an object of interest may mean at least one of a keypoint and a feature map. And, according to an embodiment of the present invention, the corresponding information of the comparison target object, that is, information about at least one of the characteristics, shape, pattern, and color of the comparison target object, is an embodiment of the present invention It may be derived in advance by the candidate object management unit 220 according to , and stored in the comparison target object pool.

Continuing, the candidate object management unit 220 according to an embodiment of the present invention, based on information about at least one of features, shapes, patterns, and colors derived for the object of interest as described above, respectively, the derived features The degree of similarity based on the information on , the degree of similarity based on the information on the shape derived above, the degree of similarity based on the information on the pattern derived above, and the degree of similarity based on the information on the color derived above can be calculated, respectively. In addition, the candidate object manager 220 according to an embodiment of the present invention assigns an appropriate weight to each similarity calculated as above, and synthesizes them to determine the similarity between the comparison target object and the object of interest detected in the photographed image. A predetermined number of comparison target objects may be derived as candidate objects in the highest order.

Continuing, the candidate object management unit 220 according to an embodiment of the present invention may apply different algorithms in deriving information about at least one of a characteristic, a shape, a pattern, and a color for the object of interest as described above. there is.

For example, the candidate object manager 220 according to an embodiment of the present invention applies an Oriented FAST and Rotated BRIEF (ORB) algorithm to deriving feature-related information, and Contour to deriving shape-related information. ) detection algorithm, and LBP (Local Binary Pattern) algorithm can be applied to derivation of pattern information. However, algorithms applied in deriving information on at least one of features, shapes, patterns and colors according to an embodiment of the present invention are not limited to those listed above, and the scope of achieving the object of the present invention It can be changed in various ways. For example, the candidate object management unit 220 according to an embodiment of the present invention, as well as the image processing techniques as described above, deep learning-based algorithms such as CNN (Convolutional Neural Networks) or Auto Encoder (Auto Encoder) more may be used.

In addition, when an object of interest is detected in the captured image, the candidate object manager 220 according to an embodiment of the present invention includes at least a portion of the detected object of interest and a comparison object pool related to the detected object of interest. By comparing at least some of the comparison target objects to be compared with each other, a degree of similarity between the comparison target object and the detected object of interest may be calculated. And, according to an embodiment of the present invention, at least a portion of the comparison target object and at least a portion of the detected object of interest may be associated with each other.

Specifically, at least a portion of the comparison target object and at least a portion of the detected object of interest are determined by the candidate object manager 220 according to an embodiment of the present invention, including various embodiments described below. method can be determined. In addition, at least a portion of the comparison target object and at least a part of the detected target object of interest may be determined by referring to a region of interest preset with respect to the comparison target object. Here, according to an embodiment of the present invention, the region of interest preset with respect to the comparison target object is a region that is a characteristic of the comparison target object (eg, a region with a special pattern or decoration, based on a deep learning model) Including a predetermined area that can be derived from can do.

6 to 8 are diagrams exemplarily illustrating a process for calculating a degree of similarity between an object of interest and a comparison object according to an embodiment of the present invention. Here, FIGS. 6 (a), 7 (a), and 8 (a) each show a photographed image of an object of interest, and FIGS. 6 (b) to (d) and FIG. 7 (b) to (d) and (b) to (d) of FIG. 8 each represent a photographed image of a comparison target object. And, referring to FIGS. 6 (b) to (d) and 7 (b) to (d), according to an embodiment of the present invention, preset ROIs 621 to 623 and 631 for a comparison target object to 633, 641 to 642, 721 to 723, 731 to 733, and 741 to 742) can be confirmed.

For example, referring to FIGS. 6A to 6D , the candidate object manager 220 according to an embodiment of the present invention provides an object of interest in FIG. 6A and an object of interest in FIG. 6B . In the case of calculating the degree of similarity between the comparison target objects of can Similarly, when the candidate object manager 220 according to an embodiment of the present invention calculates the similarity between the object of interest of FIG. 6(a) and the comparison object of FIG. 6(c), the comparison target object The similarity may be calculated by comparing the ROIs 631 to 633 of . Since the degree of similarity between the object of interest of FIG. 6A and the comparison target object of FIG. 6D can also be calculated in a similar manner to the above, a detailed description thereof will be omitted.

For another example, referring to FIGS. 7A to 7D , the candidate object manager 220 according to an embodiment of the present invention provides an object of interest in FIG. 7A and FIG. 7B ), the similarity may be calculated in consideration of the number of features 711 of the object of interest included in each of the ROIs 721 to 723 of the comparison object. Similarly, when the candidate object manager 220 according to an embodiment of the present invention calculates the similarity between the object of interest of FIG. 7A and the comparison object of FIG. 7C , the comparison target object The similarity may be calculated in consideration of the number of features 711 of the object of interest included in each of the ROIs 731 to 733 of . Since the degree of similarity between the object of interest of FIG. 7A and the comparison target object of FIG. 7D can also be calculated in a similar manner to the above, a detailed description thereof will be omitted.

As another example, referring to FIGS. 8A to 8D , the candidate object manager 220 according to an embodiment of the present invention provides an object of interest in FIG. 8A and an object of interest in FIG. 8( In the case of calculating the degree of similarity between objects to be compared in b), cluster regions 812 and 813 are generated by clustering features 811 of the object of interest, and corresponding to the generated cluster regions 812 and 813, respectively. The similarity may be calculated by comparing the regions 822 and 823 of the comparison object with each other. The degree of similarity between the object of interest of FIG. 8(a) and the comparison target of FIG. 8(c) and the similarity between the object of interest of FIG. 8(a) and the comparison target of FIG. 8(d) is also similar to the above Since it can be calculated by this method, a detailed description thereof will be omitted.

As another example, the candidate object management unit 220 according to an embodiment of the present invention removes the background area from the captured image of the object of interest and sets the remaining foreground area regarding the object of interest and the foreground area regarding the comparison target object. By comparing with each other, a degree of similarity between the object of interest and the object to be compared may be calculated. Here, the foreground area of the comparison target object may be preset as described above. In addition, the foreground area for the object of interest and the object to be compared may be derived by applying algorithms such as Grabcut, Selective Search, Mask R-CNN, and Semantic Segmentation. However, the algorithms applied to the foreground region derivation according to an embodiment of the present invention are not limited to those listed above, and may be variously changed within a range capable of achieving the object of the present invention.

In the above, the process for calculating the degree of similarity between the object of interest and the object to be compared has been described in detail with reference to FIGS. 6 to 8 , but in the present invention, the process of calculating the degree of similarity is necessarily performed in the embodiment of FIGS. 6 to 8 . It is not limited to, and it is pointed out that any number of changes or combinations can be made within the scope that can achieve the object of the present invention. In addition, in the contents described above with reference to FIGS. 6 to 8, the similarity is calculated based on information on at least one of shape, pattern, and color, respectively, in addition to the case of calculating the degree of similarity based on information about the characteristic. It should be understood that both cases can be applied.

On the other hand, when an object of interest is detected in the captured image, the candidate object management unit 220 according to an embodiment of the present invention provides identification information on the comparison object included in the comparison object pool for the detected object of interest, the above may perform a function of deriving a candidate object based on at least one of group information about the comparison target object, partial identification information obtained from a user with respect to the detected object of interest, and a detection result of the object of interest .

Specifically, according to an embodiment of the present invention, the group information on the comparison target object includes identification information about at least one other comparison target object that matches at least one of the identification information on the comparison target object. may include That is, according to an embodiment of the present invention, in the group information about the comparison target object, the comparison target object is not exactly the same object as the comparison target object, but another comparison target that shares at least some characteristics with the comparison target object Identification information about the object may be included.

For example, according to an embodiment of the present invention, assuming that the object of interest is a handbag, group information about a specific comparison target object includes a size, a size, At least one of a canvas, a material, a color, and a pattern may include identification information on a comparison target object different from each other. Here, according to an embodiment of the present invention, comparison target objects having similar design to the specific comparison target object are determined with reference to the identification information of the specific comparison target object (eg, of the specific comparison target object above) Another comparison target object having a model name partially identical to the model name may be determined as a comparison target object having a similar design), and may be classified by a person or an external system and stored in a comparison target object pool.

14 to 17 are diagrams exemplarily illustrating a process of deriving a candidate object according to an embodiment of the present invention.

For example, referring to (a) and (b) of FIG. 14 , as described in detail above, the candidate object manager 220 according to an embodiment of the present invention is included in the comparison target object pool for the object of interest. A degree of similarity between the comparison target object to be compared and the object of interest detected in the captured image 1400 may be calculated, and candidate objects 1420 to 1460 may be derived based on the calculated similarity degree. Here, according to an embodiment of the present invention, a number on the left side of the image of each candidate object 1420 to 1460 may indicate a ranking based on similarity. And, if the shapes of the figures surrounding the number are the same, it may indicate that they belong to the same group. That is, according to an embodiment of the present invention, the group information on the comparison object 1420 may include identification information on the comparison object 1460 , and vice versa. Then, as partial identification information obtained from the user with respect to the object of interest, it may be assumed that the pattern of the object of interest is a monogram and identification information that the size is BB is obtained ( 1410 ).

Continuing, for example, referring to FIG. 15 , the candidate object manager 220 according to an embodiment of the present invention provides the candidate object 1420 derived based on the degree of similarity as above, based on group information of the comparison target object. to 1460) can be adjusted. That is, the candidate object manager 220 according to an embodiment of the present invention, with respect to the candidate objects 1420 to 1460 derived based on the degree of similarity as described above, the candidate objects belonging to the same group are determined based on the degree of similarity in the group. The ranks of the candidate objects 1420 to 1460 derived based on the similarity may be adjusted to follow the ranks of the candidate objects located in the highest rank. For example, the ranking of the candidate object 1460 derived based on the similarity may be adjusted from 21st to 1st according to the ranking of the highest ranked candidate object 1420 among candidate objects corresponding to the same group ( 1510 ). Similarly, the ranking of the candidate object 1450 derived based on the similarity may be adjusted from 7th to 2nd according to the ranking of the highest-ranked candidate object 1430 among the candidate objects corresponding to the same group ( 1520 ).

Continuing, for example, referring to FIG. 16A , the candidate object manager 220 according to an embodiment of the present invention determines the rank of the candidate object adjusted based on the group information of the comparison target object as described above. As a reference, a candidate object may be derived based on partial identification information 1410 obtained from a user with respect to the object of interest. For example, from among the candidate objects 1510 to 1530 adjusted based on group information of the comparison object, objects having the pattern of the monogram 1410 may be derived as candidate objects ( 1610 to 1630 ).

Furthermore, referring to FIG. 16B , the candidate object manager 220 according to an embodiment of the present invention selects the candidate object based on the group information on the candidate objects 1610 to 1630 derived as described above. If there is an object having a size of BB 1410 in the group to which it belongs, the object may be derived as a candidate object ( 1640 ). Alternatively, if the partial identification information obtained from the user corresponds to the actual size of the object of interest, the candidate object manager 220 according to an embodiment of the present invention selects the object having the smallest size difference from the actual size as the candidate object. It can also be derived as

For another example, the candidate object manager 220 according to an embodiment of the present invention calculates a degree of similarity between the object of interest and the object of interest above, based on information about the characteristics, shape, and pattern of the object of interest, A candidate object may be derived based on the calculated similarity. In addition, the candidate object manager 220 according to an embodiment of the present invention may derive, as a candidate object, an object having the most similar color to the object of interest among other comparison objects belonging to the same group as the derived candidate object. Here, the degree of color similarity may be determined by calculating, by the candidate object manager 220 according to an embodiment of the present invention, the degree of similarity between the comparison target object and the object of interest above, based on information about the color of the object of interest. .

For another example, the candidate object management unit 220 according to an embodiment of the present invention calculates a degree of similarity between the comparison target object and the object of interest above, based on information about the characteristic, pattern, and color of the object of interest, A candidate object may be derived based on the calculated similarity. And, the candidate object management unit 220 according to an embodiment of the present invention, from among the derived candidate objects, a candidate object having a similar shape (eg, a vertical length longer than a horizontal length) to the object of interest is ranked higher. You can adjust the rank of objects. Here, the degree of similarity in shape may be determined when the candidate object manager 220 according to an embodiment of the present invention calculates the degree of similarity between the object of interest and the object of interest above, based on information about the shape of the object of interest. .

As another example, referring to FIGS. 9 and 17 , as described above, as a result of the detection of the object of interest 911 by the object of interest manager 210 according to an embodiment of the present invention, the object of interest It may be assumed that the main category is recognized as a bag and the sub-category is recognized as an animal (or cat). In this case, the candidate object manager 220 according to an embodiment of the present invention may adjust the rank 1710 of the candidate object derived based on the degree of similarity based on the above detection result ( 1720 ).

As another example, referring to FIGS. 10 and 17 , as described above, as a result of the detection of the object of interest 1021 by the object of interest manager 210 according to an embodiment of the present invention, the object of interest is It may be assumed that the main category is recognized as a bag and the sub-category is recognized as an animal (or cat). In this case, the candidate object manager 220 according to an embodiment of the present invention may adjust the rank 1710 of the candidate object derived based on the degree of similarity based on the above detection result ( 1720 ).

On the other hand, the process of deriving the candidate object by the candidate object manager 220 according to an embodiment of the present invention is not limited to the method described above, and may be variously changed within the scope that can achieve the object of the present invention. can

Meanwhile, the candidate object manager 220 according to an embodiment of the present invention may perform a function of filtering the comparison target object pool based on partial identification information obtained from the user with respect to the object of interest detected in the photographed image. there is.

Specifically, the candidate object management unit 220 according to an embodiment of the present invention may include partial identification information obtained from a user with respect to the object of interest from among the comparison objects included in the comparison object pool related to the object of interest; A comparison target object having associated identification information may be filtered. And, by allowing candidate objects to be derived only from the filtered comparison target objects, the amount of computation of the candidate object management unit 210 according to an embodiment of the present invention can be reduced and the accuracy of the derived candidate objects can be increased.

Meanwhile, the relevance evaluation unit (not shown) according to an embodiment of the present invention evaluates the degree of relevance between the object of interest detected in the captured image and partial identification information obtained from the user with respect to the detected object of interest. function can be performed.

Specifically, when an object of interest is detected from a photographed image by the object of interest manager 210 according to an embodiment of the present invention, the relevance evaluation unit according to an embodiment of the present invention is configured to The degree of correlation between partial identification information obtained from the user with respect to the detected object of interest may be evaluated. Furthermore, the relevance evaluation unit according to an embodiment of the present invention may provide feedback to the above user based on the evaluated relevance level.

For example, it may be assumed that the category of the object of interest detected in the captured image by the object of interest manager 210 according to an embodiment of the present invention is a Louis Vuitton bag. In this case, if the partial identification information obtained from the user with respect to the detected object of interest is Chanel, the relevance evaluation unit according to an embodiment of the present invention includes the partial identification information obtained from the user and the object of interest. It may be evaluated that the degree of association between the identification information is less than or equal to a predetermined level. In addition, the user may again provide a photographed image of the object of interest to the above user or request as feedback to select or input partial identification information (eg, brand) obtained from the above user again.

On the other hand, the function for the relevance evaluation unit according to an embodiment of the present invention to evaluate the degree of relevance described above includes the object of interest management unit 210 , the candidate object management unit 220 and the price management unit ( It should be understood that various functions performed by 230) may be performed in parallel and/or independently.

Next, the price management unit 230 according to an embodiment of the present invention provides an object of interest determined with reference to identification information on a candidate object derived by the candidate object management unit 220 according to an embodiment of the present invention. It is possible to perform a function of estimating the price of the object of interest based on the identification information about the object of interest and the state information obtained about the object of interest.

Specifically, the price management unit 230 according to an embodiment of the present invention identifies the object of interest with reference to the identification information on the candidate object derived by the candidate object management unit 220 according to the embodiment of the present invention. information can be determined. According to an embodiment of the present invention, the identification information on the object of interest is information necessary for estimating the price of the object of interest as described above. It may mean that there is information

For example, the price management unit 230 according to an embodiment of the present invention may determine identification information on an object of the highest rank among candidate objects as identification information on the object of interest. For another example, the price management unit 230 according to an embodiment of the present invention may determine identification information on an object selected by a user from among candidate objects as identification information on an object of interest.

Continuing, the state information obtained with respect to the object of interest detected by the object-of-interest manager 210 according to an embodiment of the present invention may include state information obtained from the user regarding the object of interest. That is, according to an embodiment of the present invention, as described above with reference to FIGS. 13 (b) and (c), the state information obtained with respect to the object of interest may be obtained from the user, and thus obtained from the user It may be obtained based on an artificial intelligence-based state information estimation model that is learned by using the state information as learning data.

More specifically, the object-of-interest management unit 210 according to an embodiment of the present invention may include, based on the above state information estimation model, the location, size, and/or the size of scratches, stains, aged parts, etc. existing in the object of interest. By estimating the degree, it is possible to obtain state information about the object of interest above. In addition, the price management unit 230 according to an embodiment of the present invention may evaluate the quality of the object of interest based on the state information obtained about the object of interest and the identification information about the object of interest as described above. According to an embodiment of the present invention, the quality of the object of interest evaluated as described above may mean a quantitative numerical value, and may mean a grade classified as high, medium, and low, but is not limited thereto.

For example, if the object of interest is a bag, the price management unit 230 according to an embodiment of the present invention may include a location of a scratch (or a stain, an aged part, etc.) (eg, a scratch on the front of the bag). If there are scratches, they may be rated as a lower quality than if they were on the bottom), size, and color of the leather (white leather is more prone to scratches than black leather, so if there are scratches on white leather, it will be black). It is possible to evaluate the quality of the object of interest above based on evaluation factors such as a color leather with scratches) and a type of leather. In this case, the interest price management unit 230 according to an embodiment of the present invention may evaluate the quality of the object of interest with reference to a predetermined weight assigned to each evaluation element.

Continuing, the price management unit 230 according to an embodiment of the present invention, based on the identification information on the object of interest, from the comparison object pool, price information (eg, Official prices, second-hand prices, etc.) can be obtained. According to an embodiment of the present invention, the above price information may be collected in real time by a method such as web crawling. And, the price management unit 230 according to an embodiment of the present invention, when the quality of the object of interest is evaluated based on the state information obtained with respect to the object of interest as described above, the evaluated quality and the price information With reference to it, the price of the object of interest above can be estimated.

Meanwhile, the price management unit 230 according to an embodiment of the present invention may estimate the price of the object of interest further based on the popularity of the object of interest detected in the captured image.

Specifically, according to an embodiment of the present invention, the popularity of the object of interest may include the popularity of the object of interest itself and the popularity of a brand to which the object of interest belongs. And, the price management unit 230 according to an embodiment of the present invention, the search frequency of the search engine (eg, increase or decrease of the search frequency within a specific period), the search frequency of hashtags in SNS, online community posts, comments It is possible to determine the popularity of the above object of interest by referring to information such as.

Meanwhile, the price management unit 230 according to an embodiment of the present invention may estimate the price of the object of interest detected in the captured image as described above, and provide the estimated price to the user.

For example, when a plurality of candidate objects are derived by the candidate object management unit 220 according to an embodiment of the present invention, the candidate object management unit 220 according to an embodiment of the present invention receives the derived plurality of objects. An image and identification information of the candidate object may be provided to the user. And, when the user selects an object corresponding to the object of interest (ie, the same type of object as the object of interest) from among the plurality of candidate objects above, the price management unit 230 according to an embodiment of the present invention, the selected object determines the identification information of the object of interest as identification information about the object of interest, and provides the user with the price of the object of interest estimated based on the identification information about the object of interest and state information obtained with respect to the object of interest can do.

Meanwhile, the price management unit 230 according to an embodiment of the present invention may determine a display unit of a price to be provided to the user based on metadata included in the captured image. For example, the price management unit 230 according to an embodiment of the present invention, based on the country information (ie, metadata) in which the photographed image is photographed, a currency (eg, dollar, won, etc.) used in the country ) may be determined as the display unit of the price to be provided to the above user.

Next, the communication unit 240 according to an embodiment of the present invention may perform a function of enabling data transmission/reception to and from the object of interest management unit 210 , the candidate object management unit 220 , and the price management unit 230 . there is.

Finally, the control unit 250 according to an embodiment of the present invention performs a function of controlling the flow of data between the object of interest management unit 210 , the candidate object management unit 220 , the price management unit 230 , and the communication unit 240 . can do. That is, the control unit 250 according to the present invention controls the data flow to/from the outside of the object information estimation system 200 or the data flow between each component of the object information estimation system 200 , thereby controlling the object of interest management unit 210 . ), the candidate object management unit 220 , the price management unit 230 , and the communication unit 240 may be controlled to perform their own functions, respectively.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and used by those skilled in the computer software field. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be converted into one or more software modules to perform processing in accordance with the present invention, and vice versa.

In the above, the present invention has been described with reference to specific matters such as specific components and limited embodiments and drawings, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, Those of ordinary skill in the art to which the invention pertains can make various modifications and changes from these descriptions.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the spirit of the present invention is not limited to the scope of the scope of the present invention. will be said to belong to

100: communication network
200: object information estimation system
210: object of interest management unit
220: candidate object management unit
230: price control department
240: communication department
250: control unit
300: device

Claims (18)

A method implemented in an object information estimation system for estimating information about an object, wherein the system includes an object of interest management unit, a candidate object management unit, and a price management unit;
When the captured image is obtained, the object of interest manager detects an object of interest from the captured image, and the candidate object manager includes a candidate among comparison objects included in a comparison object pool for the detected object of interest. deriving an object, and
estimating, by the price management unit, the price of the detected object of interest based on identification information on the object of interest determined with reference to the derived identification information on the candidate object and status information obtained with respect to the detected object of interest comprising steps;
The deriving step includes calculating, by the candidate object manager, a degree of similarity between the comparison target object and the detected object of interest,
The candidate object is derived based on the calculated similarity,
In the calculating step, the candidate object manager calculates the degree of similarity by comparing at least a portion of the detected object of interest and a preset area of interest with respect to the comparison target object,
At least a portion of the detected object of interest corresponds to a region of interest preset with respect to the comparison target object,
In the derivation step, the object-of-interest management unit determines whether to derive the candidate object based on validity with respect to the detected object of interest,
In the derivation step, the object-of-interest manager is configured to: the number of the objects of interest detected in the captured image, the size of the boundary region of the object of interest detected in the photographed image, and the boundary region of the object of interest detected in the photographed image and determining validity of the detected object of interest by referring to at least one of a relationship between a boundary region of another object detected in the photographed image and whether an uninterested object is detected in the photographed image,
State information about the object of interest by the object-of-interest management unit receiving input from a user regarding at least one of a location of a damaged portion of the object of interest, a shape of the damaged portion, a size of the damaged portion, and a degree of damage to the damaged portion to obtain,
The location of the damaged part and the shape of the damaged part are input by the object-of-interest management unit, so that at least one of the location of the damaged part and the shape of the damaged part is displayed on the photographed image by the user,
The size of the damaged part and the degree of damage of the damaged part are input by the object-of-interest management unit allowing the user to select a graphic object capable of estimating at least one of the size of the damaged part and the damage degree of the damaged part. subject
Way. delete According to claim 1,
The system further comprises a relevance evaluation unit,
The deriving step includes, by the association evaluation unit, evaluating a degree of association between partial identification information obtained from a user with respect to the detected object of interest and the detected object of interest
Way. 4. The method of claim 3,
Further comprising the step of providing feedback to the user based on the evaluated degree of relevance
Way. According to claim 1,
The comparison object pool is filtered based on partial identification information obtained from a user with respect to the detected object of interest.
Way. According to claim 1,
In the calculating step, information on at least one of a feature, a shape, a pattern, and a color is derived with respect to the detected object of interest, and the degree of similarity is based on information about at least one of the derived feature, shape, pattern, and color, respectively. calculated based on
Way. delete delete According to claim 1,
The obtained photographed image is an image photographed based on a photographing guideline.
Way. According to claim 1,
In the derivation step, before the derivation is performed, the obtained captured image is pre-processed
Way. 11. The method of claim 10,
The pre-processing is performed based on metadata included in the captured image.
Way. According to claim 1,
The detection result includes a main category and a sub-category related to the detected object of interest.
Way. According to claim 1,
In the derivation step, detecting the object of interest from the captured image with reference to a detection area obtained from a user with respect to the object of interest
Way. 14. The method of claim 13,
In the derivation step, the object of interest is detected based on the object of interest detection model,
The object of interest detection model is learned by using the obtained detection region as learning data.
Way. According to claim 1,
The state information about the object of interest obtained from the user is used as learning data of a state information estimation model for estimating state information about the object of interest.
Way. According to claim 1,
In the estimation step, estimating the price of the object of interest further based on the popularity of the detected object of interest
Way. A non-transitory computer-readable recording medium storing a computer program for executing the method according to claim 1 . A system for estimating information about an object, comprising:
an object of interest manager configured to detect an object of interest from the captured image when a captured image is obtained, and obtain state information about the detected object of interest;
a candidate object manager for deriving a candidate object from among comparison target objects included in a comparison target object pool for the detected object of interest; and
a price management unit estimating the price of the detected object of interest based on the obtained state information and identification information on the object of interest determined with reference to the derived identification information on the candidate object;
The candidate object manager calculates a degree of similarity between the comparison target object and the detected object of interest,
The candidate object is derived based on the calculated similarity,
The candidate object manager calculates the degree of similarity by comparing at least a portion of the detected object of interest with a preset area of interest for the comparison target object,
At least a portion of the detected object of interest corresponds to a region of interest preset with respect to the comparison target object,
The object-of-interest management unit determines whether to derive the candidate object based on validity with respect to the detected object of interest;
The object-of-interest management unit is configured to include: the number of objects of interest detected in the captured image, a size of a boundary area of the object of interest detected in the photographed image, a boundary area of the object of interest detected in the photographed image, and the captured image. determining validity with respect to the detected object of interest by referring to at least one of a relationship between boundary regions of other detected objects and whether an uninterested object is detected in the photographed image,
State information about the object of interest by the object-of-interest management unit receiving input from a user regarding at least one of a location of a damaged portion of the object of interest, a shape of the damaged portion, a size of the damaged portion, and a degree of damage to the damaged portion to obtain,
The location of the damaged part and the shape of the damaged part are input by the object-of-interest management unit, so that at least one of the location of the damaged part and the shape of the damaged part is displayed on the photographed image by the user,
The size of the damaged part and the degree of damage of the damaged part are input by the object-of-interest management unit allowing the user to select a graphic object capable of estimating at least one of the size of the damaged part and the damage degree of the damaged part. subject
system.

Images