JPWO2020044510A1 - Computer systems, object detection methods and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a computer system, an object detection method and a program in which it is easy to accurately estimate the weight of an object. A computer system acquires a captured image, extracts a feature amount from the image, detects an object, and estimates the weight of the detected object from the size shown in the image of the object. Further, the computer system estimates the size of the object from the size shown in the image of the object, and estimates the weight of the object by referring to the detected weight density of the object based on the estimated size. do. The computer system also learns the correlation between the estimated actual weight of the object and the detected image of the object (at least one of the object's density, name, size, or distance to the object). , Estimate the weight of the object. [Selection diagram] Fig. 1

Description

The present invention relates to a computer system for estimating the weight of an object, an object detection method and a program.

In recent years, the weight of an object has been estimated by performing image analysis on an image of the object.

As a technique for estimating the weight of such an object, there is disclosed a technique for estimating the weight of an object carried into an elevator by photographing the door of the elevator (see Patent Document 1). In this case, as a method of estimating the weight of the object, an image of the object carried into the car is photographed by a photographing device or the like, and the contour of the object is estimated by image analysis of this image. Further, the image is divided into predetermined blocks, and it is determined whether or not the weight is equal to or greater than the predetermined weight based on the number of blocks, the variation in the block position, and the estimated contour of the object.

Japanese Unexamined Patent Publication No. 2016-169072

However, in the configuration of Patent Document 1, each of the plurality of images is divided into a plurality of blocks, the similarity of the motion vector in each block in each image is calculated, and the contour of the object is estimated. .. In addition, it merely determines whether or not the weight of the object is equal to or greater than a predetermined value based on the number of blocks and the variation in the block position, and does not estimate the weight of the object itself, but is set in advance. It was just a judgment of whether or not it was overweight. Therefore, it was difficult to accurately estimate the weight of the object.

An object of the present invention is to provide a computer system, an object detection method and a program in which it is easy to accurately estimate the weight of an object.

The present invention provides the following solutions.

The present invention comprises an acquisition means for acquiring a captured image and
A detection means that extracts a feature amount from the image and detects an object,
An inferring means for estimating the detected weight of the object from the size shown in the image of the object, and
Provide a computer system characterized by the provision of.

According to the present invention, the computer system acquires a captured image, extracts a feature amount from the image, detects an object, and estimates the weight of the detected object from the size shown in the image of the object.

The present invention is in the category of computer systems, but other categories such as methods and programs also exhibit similar actions and effects according to the categories.

According to the present invention, it is possible to provide a computer system, an object detection method and a program in which it is easy to accurately estimate the weight of an object.

FIG. 1 is a diagram showing an outline of the object detection system 1. FIG. 2 is an overall configuration diagram of the object detection system 1. FIG. 3 is a flowchart showing the first object detection process executed by the computer 10. FIG. 4 is a flowchart showing a second object detection process executed by the computer 10. FIG. 5 is a flowchart showing a third object detection process executed by the computer 10. FIG. 6 is a flowchart showing a learning process executed by the computer 10. FIG. 7 is a diagram showing an example of an object table. FIG. 8 is a diagram showing an example of an image. FIG. 9 is a diagram showing an example of an image. FIG. 10 is a diagram showing an example of an image. FIG. 11 is a diagram showing an example of a notification screen. FIG. 12 is a diagram showing an example of a state in which a predetermined region is superimposed on the image. FIG. 13 is a diagram showing an example of a state in which a predetermined region is superimposed on the image. FIG. 14 is a diagram showing an example of a notification screen. FIG. 15 is a diagram showing an example of an image. FIG. 16 is a diagram showing an example of a complemented image that complements the image. FIG. 17 is a diagram showing an example of a notification screen.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. It should be noted that this is only an example, and the technical scope of the present invention is not limited to these.

[Overview of Object Detection System 1]
An outline of a preferred embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram for explaining an outline of an object detection system 1 which is a preferred embodiment of the present invention. The object detection system 1 is a computer system composed of a computer 10 and estimating the weight of an object.

In addition to the computer 10, the object detection system 1 includes other devices such as a photographing device for photographing an object, a terminal device for displaying an estimated weight, and a user terminal for receiving a predetermined input from a user. You may.

The computer 10 is connected to other devices (not shown) so as to be capable of data communication via a public line network or the like, and executes necessary data transmission / reception.

The computer 10 acquires an image of an object (for example, a heavy machine such as an excavator car, an agricultural product such as a vegetable, or a person) taken by an imaging device (not shown) as image data. This image data includes position information of a shooting point. The position information of the shooting point is obtained by the shooting device acquiring its own current position from GPS (Global Positioning System) or the like and using the acquired current position as the position information of the shooting point.

The computer 10 analyzes the image included in the image data and extracts the feature amounts (for example, statistical numerical values such as average, variance, and histogram of pixel values, and the shape and contour of the object). The computer 10 detects an object in this image based on the extracted features.

The computer 10 estimates the weight of the detected object. For example, the computer 10 estimates the distance from the shooting point to the object based on the position information of the shooting point, and based on this distance and the size (area) of the object in the image, the size of the object ( Volume) is estimated. The computer 10 estimates the weight of this object based on the size of this object and the weight density of the detected object.

Further, the computer 10 estimates the weight of the object by learning the correlation between the actual weight of the detected object and the acquired image of the object. For example, when the computer 10 newly acquires the image data, the computer 10 estimates the weight of the object reflected in the newly acquired image data in consideration of the learning result. At this time, the computer 10 uses at least one of the density, name, size, and distance to the object as learning as a correlation.

The outline of the processing executed by the object detection system 1 will be described.

First, the computer 10 acquires image data (step S01). The computer 10 acquires an image taken by a photographing device (not shown) and position information (position information of a photographing point) of the photographing device as image data. The photographing device acquires its own position information from GPS or the like, and the computer 10 acquires this position information as the position information of the photographing point.

The computer 10 executes image analysis based on the image data and extracts the feature amount of the image in the image data (step S02). The computer 10 extracts statistical numerical values such as average, variance, and histogram of pixel values, shape of an object, contour, and the like as feature quantities of an image.

The computer 10 detects an object reflected in this image based on the extracted features (step S03). The computer 10 refers to an object table in which the identifiers (name, model number, product number, etc.) of various objects are associated with the feature amounts of each object, and identifies the identifiers of the objects corresponding to the feature amounts extracted this time. , Detects an object.

The computer 10 estimates the weight of the detected object from the size shown in the image of the object (step S04). The computer 10 estimates the weight of the object, for example, based on the size shown in the image of the detected object (ie, the area of the object in the image).

Further, the computer 10 estimates the distance from the shooting point of this image to the object by a method such as three-point survey, and estimates the weight based on this distance and the size of the object shown in the image. .. At this time, the computer 10 refers to a weight table in which an object identifier (name, model number, product number, etc.), a size (volume) of each object, and a weight density of each object are associated with each other, and this time detection is performed. Estimate the weight of the object. The computer 10 identifies the size and weight density associated with the identifier of the object detected this time by referring to the weight table, and based on the specified size and weight density and the estimated size. Estimate the weight of the object.

Further, the computer 10 learns the correlation between the actual weight of the detected object and the image acquired this time (at least one correlation between the density, name, size of the object, or the distance to the object). From the next time onward, the computer 10 estimates the weight of the object in consideration of the learning result when estimating the weight of the object reflected in the acquired image.

The above is the outline of the object detection system 1.

[System configuration of object detection system 1]
The system configuration of the object detection system 1 which is a preferred embodiment of the present invention will be described with reference to FIG. FIG. 2 is a diagram showing a system configuration of an object detection system 1 which is a preferred embodiment of the present invention. In FIG. 2, the object detection system 1 is a computer system composed of a computer 10 and estimating the weight of an object. The computer 10 is connected to other devices such as the above-mentioned photographing device, terminal device, and user terminal (not shown) so as to be capable of data communication via a public line network or the like.

The computer 10 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as a control unit, and a device for enabling communication with other terminals and devices as a communication unit. For example, it is provided with a Wi-Fi (Wi-Fi) compatible device or the like conforming to IEEE802.11. Further, the computer 10 includes a data storage unit such as a hard disk, a semiconductor memory, a storage medium, and a memory card as a storage unit. Further, the computer 10 includes various devices and the like that execute various processes as a processing unit.

In the computer 10, when the control unit reads a predetermined program, the control unit cooperates with the communication unit to acquire the image data acquisition module 20, the notification module 21, the object designation data acquisition module 22, the area designation data acquisition module 23, and the object data acquisition. The module 24 and the actual weight data acquisition module 25 are realized. Further, in the computer 10, the control unit reads a predetermined program to realize the storage module 30 in cooperation with the storage unit. Further, in the computer 10, when the control unit reads a predetermined program, the feature quantity extraction module 40, the object detection module 41, the distance estimation module 42, the size estimation module 43, and the size estimation module cooperate with the processing unit. 44, the weight estimation module 45, the object complement module 46, and the learning module 47 are realized.

[First object detection process]
The first object detection process executed by the object detection system 1 will be described with reference to FIG. FIG. 3 is a diagram showing a flowchart of the first object detection process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.

The image data acquisition module 20 acquires an image captured by an imaging device (not shown) and position information of the imaging device as image data (step S10). In step S10, the photographing device transmits the image captured by itself and its own position information acquired from GPS or the like to the computer 10 as image data. The own position information acquired by this photographing device becomes the position information of the photographing point. By receiving this image data, the image data acquisition module 20 acquires an image captured by the photographing apparatus and position information of a photographing point of this image.

Based on the acquired image data, the feature amount extraction module 40 performs image analysis on the image included in the image data and extracts the feature amount of this image (step S11). In step S11, the feature amount extraction module 40 extracts statistical numerical values such as average, variance, and histogram of pixel values, shape of an object, contour, and the like as feature amounts.

The object detection module 41 determines whether or not an object is reflected in this image based on the extracted feature amount (step S12). In step S12, the object detection module 41 creates an object table in which identifiers (name, model number, product number, etc.) of various objects stored in advance by the storage module 30 and feature quantities (s) of the objects are associated with each other. By referring to it, it is determined whether or not the object corresponding to the extracted feature amount is reflected in this image. For example, the storage module 30 stores the identifier of the heavy machine (heavy machine name, heavy machine name classified by sales manufacturer, model number, product number, etc.) and the feature amount of the heavy machine in association with each other as an object table. Further, the storage module 30 associates the crop identifier (crop name, crop name classified by producer, identification number capable of identifying each crop, etc.) with the characteristic amount of the crop as an object table. Remember. Further, the storage module 30 stores a person's identifier (age, gender, race, height, weight, etc.) as an object table in association with the feature amount of the person.

The object detection module 41 compares the feature amount extracted this time with the object table, and determines whether or not the object corresponding to this feature amount is stored as the object table, so that the object is reflected in the image. Judge whether or not it is.

In step S12, when the object detection module 41 determines that the object is not reflected in the image (step S12 NO), the computer 10 ends this process.

On the other hand, in step S12, when the object detection module 41 determines that an object is reflected in the image (YES in step S12), the object detection module 41 detects the object reflected in this image (step S13). ). In step S13, the object detection module 41 compares the extracted feature amount with the object table, and detects the identifier of the object corresponding to the extracted feature amount as the object reflected in this image.

The distance estimation module 42 estimates the distance between the imaging device and the object based on the position information of the imaging point included in the image data (step S14). In step S14, the distance estimation module 42 estimates the distance from the shooting position to the object by, for example, triangulation. At this time, the distance estimation module 42 determines in advance the points at both ends of the baseline passing through the imaging point, and estimates the distance between the imaging point and the object.

The method by which the distance estimation module 42 estimates the distance between the photographing device and the object is not limited to the above-mentioned example, and can be appropriately changed.

The size estimation module 43 estimates the size (area) of the detected object in the image (step S15). In step S15, the size estimation module 43 estimates the region where the object exists as the size of the object, for example, based on the contour and shape of the detected object. The size estimation module 43 estimates, for example, the total number of pixels in this area as the size.

The method by which the size estimation module 43 estimates the size of the object is not limited to the above-mentioned example, and can be appropriately changed.

The size estimation module 44 estimates the size (volume) of this object based on the estimated distance from the imaging point to the object and the estimated size of the object (step S16). In step S16, the size estimation module 44 estimates the size of the object based on the ratio of the size of the object to the image and the distance.

The method by which the size estimation module 44 estimates the size of the object is not limited to the above-mentioned example, and can be appropriately changed.

The weight estimation module 45 refers to a weight table in which an identifier (name, model number, product number, etc.) of an object stored in advance by the storage module 30 is associated with the size of the object and the weight density of the object. Estimate the weight (step S17). In step S17, the weight estimation module 45 refers to the weight table based on the identifier of the object detected this time and the size of the object, and identifies the weight density corresponding to the detected object. The weight estimation module 45 estimates the weight of an object based on the identified weight density and the estimated size of the object.

The weight table stored in the storage module 30 will be described with reference to FIG. 7. FIG. 7 is a diagram schematically showing a weight table stored in the storage module 30. In FIG. 7, the storage module 30 registers the name of the object, which is the identifier of the object, the size (volume) of the object, and the weight density of the object in the weight table, and registers the registered weight table in the weight table. ,Remember. For example, the storage module 30 registers the excavator car (small), which is the name of the object, V1 which is the size of the object, and D1 which is the weight density of the object in association with each other as a weight table. Similarly, the storage module 30 registers the excavator car (large), V2, and D2 in association with each other as a weight table. Similarly, the storage module 30 registers the cabbage, V3, and D3 in association with each other as a weight table. Similarly, the storage module 30 registers a person (male, 20s), V4, and D4 as a weight table in association with each other. These data are acquired via input from the user terminal, an external computer, or the like, and the storage module 30 registers these data in the weight table and stores the weight table.

The identifier of the object registered as the weight table is not limited to the name but may be other. Further, when the object is a person, the size and the weight density may be associated with one or a combination of a plurality of ages, genders, races, heights, weights and the like as identifiers.

Further, the weight estimation module 45 may estimate the weight of the object by a method other than the method of referring to the weight table when estimating the weight of the object. For example, the weight estimation module 45 may estimate the weight of an object based on the function of the identifier of the object and the size and weight density.

The notification module 21 notifies the user terminal or the like of the estimated weight (step S18). In step S18, the notification module 21 generates a weight notification in which the inferred object identifier and weight are superimposed on the acquired image, and notifies the generated weight notification to the user terminal or the like. The user terminal or the like acquires this notification and displays it on its own display unit or the like to notify the user of the weight of this object. In this way, the computer 10 notifies the user of the weight of the object by displaying the estimated weight of the object on the user terminal or the like.

The above is the first object detection process.

Regarding the first object detection process executed by the computer 10 described above, a method of estimating the weights of heavy machinery, crops, and humans will be described.

Based on FIGS. 8, 9 and 10, a method of estimating the weight of a heavy machine, a crop, and a person as an object by the computer 10 will be described. 8, 9 and 10 are diagrams showing an example of image data acquired by the image data acquisition module 20. In addition to the image, each image data includes position information of each shooting point. FIG. 8 is an image of an excavator car as a heavy machine. FIG. 9 is an image of cabbage as an agricultural product. FIG. 10 is an image of a person.

The image data acquisition module 20 acquires the image data shown in FIGS. 8, 9 and 10 by the process of step S10 described above.

The feature amount extraction module 40 extracts the feature amount from each image data by the process of step S11 described above.

The object detection module 41 detects an object reflected in this image by the processing of steps S12 and S13 described above. The object detection module 41 detects the excavator car (small) 100 as an object based on the feature amount extracted from the image shown in FIG. Further, the object detection module 41 detects the cabbage 110 as an object based on the feature amount extracted from the image shown in FIG. The object detection module 41 detects a person (male, 20s) 120 as an object based on the feature amount extracted from the image shown in FIG.

When it is detected that a plurality of objects exist in a short distance such as a person or an agricultural product, each object is detected individually.

The distance estimation module 42 estimates the distance between the photographing device and the object by the process of step S14 described above.

The size estimation module 43 estimates the size (area) of the object in the image by the process of step S15 described above. That is, the size estimation module 43 estimates the size in the image of the shovel car (small) 100, estimates the size in the image of the cabbage 110, and estimates the size in the image of the person (male, 20s) 120.

The size estimation module 44 estimates the size (volume) of the object by the process of step S16 described above. That is, the size estimation module 44 estimates the size of the excavator car (small) 100, estimates the size of the cabbage 110, and estimates the size of a person (male, 20s) 120.

The weight estimation module 45 estimates the weight of the object by the process of step S17 described above. Since the identifier of the object is "excavator car (small)", the weight estimation module 45 refers to the weight table and identifies the weight density D1 associated with this "excavator car (small)". The weight estimation module 45 estimates the weight W1 of the excavator car (small) 100 based on the specified weight density D1 and the estimated size. Similarly, since the identifier of the object is "cabbage", the weight estimation module 45 refers to the weight table and identifies the weight density D3 associated with this "cabbage". The weight estimation module 45 estimates the weight W3 of the cabbage 110 based on the specified weight density D3 and the estimated size. Similarly, since the identifier of the object is "person (male, 20s)", the weight estimation module 45 refers to the weight table and the weight density associated with this "person (male, 20s)". Identify D4. The weight estimation module 45 estimates the weight W4 of this person (male, 20s) 120 based on the specified weight density D4 and the estimated size.

The notification module 21 notifies the estimated weight by the process of step S18 described above. A notification screen for the notification module 21 to notify the user terminal or the like will be described with reference to FIG. FIG. 11 is a diagram showing an example of a notification screen in which the notification module 21 notifies a user terminal or the like of the weight of an object. Of the three objects described above, the excavator car (small) will be described as an example. The notification module 21 similarly notifies cabbage and people.

As a notification screen, the notification module 21 uses a screen in which an identifier of an object (here, a shovel car (small), which is a name as an identifier) and an estimated weight W1 are superimposed on an acquired image as a notification screen. Display it on the user terminal, etc. As this notification screen, the notification module 21 performs processing such as surrounding, highlighting, and changing the color of the specified object to clarify which object has been specified. Finally, the notification module 21 superimposes the surrounding line surrounding the excavator car (small) 100, the identifier and the weight of the object on the acquired image as the notification screen on the user terminal or the like as the notification screen. Display it.

Similarly, for the other cabbage 110 and the person 120, the notification module 21 displays the surrounding line, the identifier, and the weight surrounding the cabbage 100 superimposed on the acquired image on the user terminal or the like as a notification screen. The surrounding line, the identifier, and the weight surrounding the person 120 are superimposed on the acquired image and displayed on the user terminal or the like as a notification screen.

The above is an explanation using an actual object in the first object detection process as an example.

[Second object detection process]
The second object detection process executed by the object detection system 1 will be described with reference to FIG. FIG. 4 is a diagram showing a flowchart of a second object detection process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.

The detailed description of the same processing as the first object detection processing described above will be omitted.

First, the object designation data acquisition module 22 determines whether or not the object designation data for designating the object to be detected has been acquired (step S20). In step S20, the object designation data acquisition module 22 is data for designating an object existing over a predetermined range such as soil, earth and sand (things generated by a disaster, excavated objects under construction, etc.), water, and one-sided agricultural products. Is acquired as object designation data. At this time, the user terminal or the like accepts an input for designating an object to be detected, and transmits the object for which the input is received to the computer 10 as object designation data. The object designation data acquisition module 22 acquires the object designation data by receiving the object designation data transmitted by the user terminal.

The source of the object designation data acquired by the object designation data acquisition module 22 can be changed as appropriate. Further, this process does not necessarily have to be executed. In this case, the computer 10 may execute the processes after step S21 described later.

When it is determined in step S20 that the object designation data acquisition module 22 has not acquired the object designation data (step S20 NO), this process ends. In this case, the computer 10 may execute the above-mentioned first object detection process.

On the other hand, in step S20, when the object designation data acquisition module 22 determines that the object designation data has been acquired (YES in step S20), the image data acquisition module 20 acquires the image data (step S21). The process of step S21 is the same as the process of step S10 described above. At this time, the photographing device transmits image data to a user terminal (not shown) in addition to the computer 10. The user terminal receives this image data and uses the image included in the image data in the processing described later.

The area designation data acquisition module 23 acquires area designation data for designating a predetermined area for the acquired image (step S22). In step S22, when the terminal device receives the image data, the terminal device and the computer 10 receive the same image data. The user terminal displays an image based on this image data on its own display unit, accepts input such as a tap operation from the user, and accepts input from the user to specify a predetermined area for this image. The user terminal specifies the coordinates in the image of this predetermined area. For example, when this area is a rectangle, the terminal device specifies the coordinates of each vertex and specifies the area surrounded by the rectangle as a predetermined area. Further, when this area is circular, the user terminal specifies the coordinates of the center, specifies the radius from the center to the circumference, and specifies the area surrounded by the circle as a predetermined area. The user terminal transmits the specified predetermined area as area designation data to the computer 10. By receiving this area designation data, the area designation data acquisition module 23 acquires the area designation data for designating a predetermined area for the image.

The feature amount extraction module 40 performs image analysis of this image based on the acquired image data, and extracts the feature amount of this image (step S23). The process of step S23 is the same as the process of step S11 described above.

The object detection module 41 detects an object reflected in a predetermined area based on the extracted feature amount (step S24). In step S24, the object detection module 41 identifies an area corresponding to this predetermined area based on the acquired area designation data. For example, when the predetermined area is a rectangle, the object detection module 41 specifies the coordinates of each vertex of the rectangle based on the area designation data, and sets the rectangular area connecting the vertices in the designated predetermined area. Identify as being. Further, when the predetermined area is circular, the object detection module 41 specifies the center coordinates and radius of the circular shape based on the area designation data, and the area surrounded by the circle is the designated predetermined area. Identify as a thing. The object detection module 41 detects an object reflected in the area surrounded by the predetermined area. The method of detecting the object by the object detection module 41 is the same as the process of steps S12 and S13 described above.

The distance estimation module 42 estimates the distance between the imaging device and the object based on the position information of the imaging point included in the image data (step S25). The process of step S25 is the same as the process of step S14 described above.

The size estimation module 43 estimates the size (area) of the image of the object reflected in the detected predetermined area (step S26). The process of step S26 is the same as the process of step S15 described above.

The size estimation module 44 estimates the size (volume) of this object based on the estimated distance and the estimated size (step S27). The process of step S27 is the same as the process of step S16 described above.

The weight estimation module 45 refers to a weight table in which an identifier (name, model number, product number, etc.) of an object stored in advance by the storage module 30 is associated with the size of the object and the weight density of the object. Estimate the weight (step S28). The process of step S28 is the same as the process of step S17 described above.

The notification module 21 notifies the user terminal of the estimated weight (step S29). The process of step S29 is the same as the process of step S18 described above.

The above is the second object detection process.

Regarding the second object detection process executed by the computer 10 described above, a method of estimating the weights of the heavy machinery and the crops will be described. The weight can be estimated for humans in the same way.

A method in which the computer 10 estimates the weights of heavy machinery and agricultural products as objects will be described with reference to FIGS. 12 and 13. 12 and 13 are diagrams showing an example of a state in which a predetermined area designated based on the area designation data acquired by the area designation data acquisition module 23 is superimposed on the image acquired by the image data acquisition module 20. Is. In addition to the image, each image data includes position information of each shooting point. FIG. 12 shows an image containing earth and sand and a designated predetermined area superimposed on the image. FIG. 13 shows an image of an agricultural product and a designated predetermined area superimposed on the image.

The object designation data acquisition module 22 acquires the object designation data by the process of step S20 described above. Here, the object designation data acquisition module 22 acquires "earth and sand" as object designation data in FIG. 12, and acquires "cabbage" as object designation data in FIG. 13.

The image data acquisition module 20 acquires image data by the process of step S21 described above.

The area designation data acquisition module 23 acquires the area designation data by the process of step S22 described above.

The feature amount extraction module 40 extracts the feature amount of the image by the process of step S23 described above.

The object detection module 41 detects an object reflected in a designated predetermined area based on the above-mentioned feature amount by the process of the above-mentioned step S24. In FIG. 12, the object detection module 41 has designated “earth and sand” as an object, and in addition, since the designated predetermined area 200 exists, the object detection module 41 detects the earth and sand existing in the predetermined area 200. Similarly, in FIG. 13, in the object detection module 41, “cabbage” is designated as an object, and in addition, since the designated predetermined area 210 exists, the cabbage existing in the predetermined area 210 exists. Is detected.

The distance estimation module 42 estimates the distance between the photographing device and the object by the process of step S25 described above.

The size estimation module 43 estimates the size (area) of the object reflected in the predetermined area in the image by the process of step S26 described above. The size estimation module 43 estimates the size of the earth and sand existing in the predetermined area 200 and the size of the cabbage existing in the predetermined area 210.

At this time, as shown in FIG. 12, when another object is reflected in the predetermined area 200, the size in the image of the object is estimated assuming that the target object exists in the portion corresponding to the other object. do. Specifically, the size estimation module 43 adds the size of the heavy equipment existing in the predetermined area 200 to the size of the earth and sand, and estimates the size of the earth and sand existing in the predetermined area 200. This is effective when a large object such as a heavy machine hides the target object.

Further, as shown in FIG. 13, when another object is reflected in the predetermined area 210, the size in the image of the object is estimated assuming that the target object does not exist in the portion corresponding to the other object. .. Specifically, the size estimation module 43 excludes the size of the cabbage in which one ball is not completely contained in the predetermined area 210 among the cabbage existing in the predetermined area 210, and one ball is completely in this predetermined area 210. The size of the cabbage existing in the predetermined area 210 is estimated based only on the size of the cabbage contained in the area 210. This is effective when only a part of an object such as an agricultural product exists in a predetermined area.

The size estimation module 44 estimates the size (volume) of this object by the process of step S27 described above. That is, the size estimation module 44 estimates the size of the earth and sand in the predetermined area 200, and estimates the size of the cabbage in the predetermined area 210.

The weight estimation module 45 estimates the weight of the object by the process of step S28 described above. Since the identifier of the object is "earth and sand", the weight estimation module 45 refers to the weight table and identifies the weight density D5 associated with this "earth and sand". The weight estimation module 45 estimates the weight W5 of "earth and sand" existing in the predetermined region 200 based on the specified weight density D5 and the estimated size. Similarly, since the identifier of the object is "cabbage", the weight estimation module 45 refers to the weight table and identifies the weight density D3 associated with this "cabbage". The weight estimation module 45 estimates the weight W6 of the "cabbage" existing in the predetermined region 210 based on the specified weight density D3 and the estimated size.

The notification module 21 notifies the estimated weight by the process of step S29 described above. A notification screen for the notification module 21 to notify the user terminal or the like will be described with reference to FIG. FIG. 14 is a diagram showing an example of a notification screen in which the notification module 21 notifies the user terminal or the like of the weight of an object reflected in a predetermined area. Of the two objects described above, earth and sand will be described as an example. The notification module 21 similarly notifies the cabbage.

As a notification screen, the notification module 21 uses a screen in which an identifier of an object (here, earth and sand as an identifier), an estimated weight W of 5 kg, and a predetermined area 200 are superimposed on the acquired image as a notification screen, such as a user terminal or the like. To display.

Similarly, for the cabbage, the notification module 21 displays the predetermined area 200, the identifier and the weight of the object superimposed on the acquired screen on the user terminal as a notification screen.

Although the description is omitted, even when the object is a human being, the same processing as the above-mentioned earth and sand and cabbage may be executed. As this notification screen, the notification module 21 performs processing such as highlighting a predetermined area and changing the color to clarify in which place in the image the object reflected is specified. Finally, the notification module 21 causes the user terminal to display a notification screen in which a predetermined area, an identifier of the object, and the weight are superimposed on the acquired screen.

The above is an explanation using an actual object in the second object detection process as an example.

[Third object detection process]
A third object detection process executed by the object detection system 1 will be described with reference to FIG. FIG. 5 is a diagram showing a flowchart of a third object detection process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.

The detailed description of the above-described first object detection process or the same process as the second object detection process will be omitted.

First, the image data acquisition module 20 acquires an image taken by a photographing device (not shown) and position information of the photographing device as image data (step S30). The process of step S30 is the same as the process of step S10 described above.

Based on the acquired image data, the feature amount extraction module 40 performs image analysis on the image included in the image data and extracts the feature amount of this image (step S31). The process of step S31 is the same as the process of step S11 described above.

The object detection module 41 determines whether or not an object is reflected in this image based on the extracted feature amount (step S32). The process of step S32 is the same as the process of step S12 described above.

In step S32, when the object detection module 41 determines that the object is not reflected in the image (step S32 NO), the computer 10 ends this process.

On the other hand, in step S32, when the object detection module 41 determines that an object is reflected in the image (YES in step S32), the object detection module 41 detects the object reflected in this image (step S33). ). The process of step S33 is the same as the process of step S13 described above.

The object detection module 41 determines whether or not the entire object has been detected (step S34). In step S34, the object detection module 41 detects that a part of the detected object is at the edge of the image (for example, when the image is rectangular, a part of the object touches each side or a part of the object is on each side. It is judged whether or not it is in the position). Further, the object detection module 41 determines whether or not a part of the contour or shape of the detected object is interrupted in the middle.

The object detection module 41 may determine whether or not the entire object has been detected by a method other than the above-mentioned method.

If it is determined in step S34 that the object detection module 41 has detected the entire object (YES in step S34), the computer 10 ends this process. In this case, the computer 10 may execute the above-mentioned first object detection process.

On the other hand, in step S34, when the object detection module 41 determines that the entire object could not be detected (step S34 NO), the object data acquisition module 24 has an identifier of this object stored in advance by the external computer or the storage module 30. Acquire the object data which is the data related to the image, size, size, etc. of the object corresponding to (step S35). In step S35, the object data acquisition module 24 acquires the object data corresponding to the identifier of the object by referring to various tables and the like stored in the external computer and the storage module 30 for the identifier of the detected object.

The object complement module 46 complements the missing portion of the detected object based on the acquired object data (step S36). In step S36, the object complement module 46 compares the image in the acquired object data with the image of the object detected this time, and estimates the ratio. The object complement module 46 corrects by reducing or enlarging the image in the acquired object data based on the estimated ratio. The object complement module 46 compares the corrected image with the detected object. The object complement module 46 identifies the missing portion of the detected object in the corrected image. By connecting the specified portion to the image of the detected object, the object complement module 46 pseudo-corrects the entire object as an image and complements the missing portion.

The distance estimation module 42 estimates the distance between the imaging device and the object based on the position information of the imaging point included in the image data (step S37). The process of step S37 is the same as the process of step S14 described above.

The size estimation module 43 estimates the size (area) of the object shown in the complemented image based on the image of the complemented object and the acquired object data (step S38). The process of step S38 is the same as the process of step S15 described above.

The size estimation module 44 estimates the size (volume) of this object based on the estimated distance from the imaging point to the object and the estimated size of the object (step S39). The process of step S39 is the same as the process of step S16 described above.

The weight estimation module 45 estimates the weight of this object by referring to the weight table (step S40). The process of step S40 is the same as the process of step S17 described above.

The notification module 21 notifies the user terminal of the estimated weight (step S41). The process of step S41 is the same as the process of step S18 described above.

The above is the third object detection process.

A method of estimating the weight of a heavy machine will be described for the third object detection process executed by the computer 10 described above. The weight of agricultural products and humans can be estimated in the same way.

A method of estimating the weight of a heavy machine as an object by the computer 10 will be described with reference to FIG. FIG. 15 is a diagram showing an example of image data acquired by the image data acquisition module 20. The image data includes position information of the shooting point in addition to the image. FIG. 15 is an image of an excavator car as a heavy machine.

The image data acquisition module 20 acquires the image data shown in FIG. 15 by the process of step S30 described above.

The feature amount extraction module 40 extracts the feature amount from the image data by the process of step S31 described above.

The object detection module 41 detects an object reflected in this image by the processing of steps S32 and S33 described above. The object detection module 41 detects the excavator car (small) 300 as an object based on the feature amount extracted from the image shown in FIG.

The object detection module 41 determines that a part of the detected object is missing by the process of step S34 described above, and the object data acquisition module 24 obtains object data related to the object detected by the process of step S35 described above. get.

The object complement module 46 complements the missing portion of the detected object by the process of step S36 described above.

FIG. 16 is a diagram showing an excavator car (small) 310 in which the object complement module 46 complements a portion lacking in the excavator car (small) 300. In FIG. 16, the object complement module 46 complements the excavator car (small) 310 in which the correction portion 320 is corrected based on the object data.

The distance estimation module 42 estimates the distance between the photographing device and the object by the process of step S37 described above.

The size estimation module 43 estimates the size (area) of the complemented object by the process of step S38 described above. That is, the size estimation module 43 estimates the size of the complemented excavator car (small) 310 in the image.

The size estimation module 44 estimates the size (volume) of the object by the process of step S39 described above. That is, the size estimation module 44 estimates the size of the excavator car (small) 310 after this complementation.

The weight estimation module 45 estimates the weight of the object by the process of step S40 described above. Since the identifier of the object is "excavator car (small)", the weight estimation module 45 refers to the weight table and identifies the weight density D1 associated with this "excavator car (small)". The weight estimation module 45 estimates the weight W7 of the excavator car (small) 100 based on the specified weight density D1 and the estimated size.

The notification module 21 notifies the estimated weight by the process of step S41 described above. A notification screen for the notification module 21 to notify the user terminal or the like will be described with reference to FIG. FIG. 17 is a diagram showing an example of a notification screen in which the notification module 21 notifies a user terminal or the like of the weight of an object.

The notification module 21 is a screen in which an identifier of an object (here, a shovel car (small), which is a name as an identifier) and an estimated weight are superimposed on an acquired image (image before correction) as a notification screen. Is displayed on the user terminal or the like as a notification screen. As this notification screen, the notification module 21 performs processing such as surrounding, highlighting, and changing the color of the specified object to clarify which object has been specified. Finally, the notification module 21 superimposes the surrounding line surrounding the excavator car (small) 300, the identifier and the weight of the object on the acquired image as the notification screen, and displays the notification screen on the user terminal or the like. Display it.

The notification module 21 displays the same notification on the user terminal as a notification screen for other crops and people.

The above is an explanation using an actual object in the object detection process of the second plan as an example.

[Learning process]
The learning process executed by the object detection system 1 will be described with reference to FIG. FIG. 6 is a diagram showing a flowchart of a learning process executed by the computer 10. The process executed by each of the above-mentioned modules will be described together with this process.

First, the actual weight data acquisition module 25 acquires actual weight data indicating the actual weight, which is the actual weight of the object whose weight is estimated by the first, second, and third object detection processes described above (step S50). .. In step S50, the terminal device inputs the result of actually measuring the weight of the object or acquires the measured result, and transmits the identifier, the image, and the data related to the actual weight of the object to the computer 10 as the actual weight data. do. The actual weight data acquisition module 25 acquires the estimated actual weight of the object by receiving the actual weight data.

The learning module 47 learns the correlation between the actual weight (actual weight) of the acquired object and the image of the detected object (step S51). In step S51, the learning module 47 learns at least one correlation between the actual weight and the image, such as the density, name, size, or distance to the object.

The storage module 30 stores the learning result (step S52).

The weight estimation module 45 estimates the weight of the object in consideration of the learning result in the processing of steps S17, S28, and S40 described above. That is, when estimating the weight of an object, the weight estimation module 45 estimates the weight of this object by referring to the weight table and making corrections based on the correlation based on the learning result.

The above is the learning process.

The above-mentioned means and functions are realized by a computer (including a CPU, an information processing device, and various terminals) reading and executing a predetermined program. The program is provided, for example, in the form of being provided from a computer via a network (Software as a Service). Further, the program is provided in a form stored in a computer-readable storage medium such as a flexible disk, a CD (CD-ROM or the like), or a DVD (DVD-ROM, DVD-RAM or the like). In this case, the computer reads the program from the storage medium, transfers the program to the internal storage device or the external storage device, stores the program, and executes the program. Further, the program may be stored in advance in a storage device (storage medium) such as a magnetic disk, an optical disk, or a magneto-optical disk, and the program may be provided from the storage device to a computer via a communication line.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments described above. In addition, the effects described in the embodiments of the present invention merely list the most preferable effects arising from the present invention, and the effects according to the present invention are limited to those described in the embodiments of the present invention. is not it.

1 object detection system, 10 computers

The present invention comprises an acquisition means for acquiring a captured image and
A detection means that extracts a feature amount from the image and determines whether or not the entire object has been detected.
When the entire object is detected, the distance between the shooting point and the detected object is estimated based on the position information of the shooting point, and based on the distance and the size of the detected object shown in the image. An estimation means for estimating the detected size of the object and estimating the detected weight of the object from the size shown in the image of the object based on the size of the object and the detected weight density of the object.
An object data acquisition means for acquiring object data when it is determined that the entire object could not be detected,
An object complementing means that complements the missing part of the detected object based on the acquired object data,
Provide a computer system characterized by the provision of.

According to the present invention, the computer system obtains the captured image, and extracts a feature quantity from the image, to determine Taka not detect the entire object, when detecting entire object, the image pickup point The distance from the shooting point to the detected object is estimated based on the position information, and the size of the detected object is estimated based on the distance and the size shown in the image of the detected object. Based on the weight density of the object, the detected weight of the object is estimated from the size shown in the image of the object, and when it is determined that the entire object cannot be detected, the object data is acquired and acquired. Based on the object data, the missing part in the detected object is complemented .

Claims (6)

An acquisition method for acquiring captured images, and
A detection means that extracts a feature amount from the image and detects an object,
An inferring means for estimating the detected weight of the object from the size shown in the image of the object, and
A computer system characterized by being equipped with. The guessing means estimates the size of the object from the size shown in the image of the object, and estimates the weight of the object by referring to the detected weight density of the object based on the estimated size. ,
The computer system according to claim 1. The guessing means estimates the weight of the object by learning the correlation between the estimated actual weight of the object and the detected image of the object.
The computer system according to claim 1. The guessing means learns at least one correlation between the estimated actual weight of the object and the density of the detected object, the name of the detected object, the size of the detected object or the distance to the detected object. By doing so, the weight of the object is estimated.
The computer system according to claim 3. It is an object detection method executed by a computer system.
Steps to get the captured image and
A step of extracting a feature amount from the image and detecting an object,
A step of estimating the detected weight of the object from the size shown in the image of the object, and
An object detection method characterized by comprising. For computer systems
Steps to get the captured image,
A step of extracting a feature amount from the image and detecting an object,
A step of estimating the detected weight of the object from the size shown in the image of the object,
A computer-readable program for running.

Images