JP2023140450A - Information processing device, information processing system, and information processing method - Google Patents

Abstract

To estimate a carrying method of an object owned by a person.SOLUTION: An information processing device includes: an image acquisition unit that obtains an image; an object detection unit that detects a region including a person and a region including an object other than the person, from the image; a related object specification unit that specifies an object having a predetermined relation to the person as a related object; and carrying method estimation means that estimates a carrying method of the related object, based on a region including the related object.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing system, and an information processing method.

2. Description of the Related Art A method is known for acquiring information about objects related to a person in an image.

For example, Patent Document 1 describes an image in which a person area representing at least a part of the person's body is detected from a captured image, and objects related to the person are detected based on an extraction area set according to the person area. A recognition system is disclosed.

International Publication No. 2021/192206

According to Patent Document 1, although it is possible to detect an object carried by a person, it is not possible to estimate how the object is being carried.

In view of the above-mentioned problems, an object of the present disclosure is to provide an information processing device, an information processing system, and an information processing method that are capable of estimating how a person carries an object.

An information processing device of the present disclosure includes an image acquisition unit that acquires an image, an object detection unit that detects an area including a person and an area including an object other than the person from the image, and an object that has a predetermined relationship with the person. The apparatus includes a related object specifying section that specifies a related object as a related object, and a carrying method estimating section that estimates a carrying method of the related object based on a region including the related object.

The information processing system of the present disclosure includes an image acquisition unit that acquires an image, an object detection unit that detects from the image an area including a person and an area including an object other than the person, and an object that has a predetermined relationship with the person. The apparatus includes a related object specifying means for specifying a related object as a related object, and a carrying method estimating means for estimating a carrying method for the related object based on a region including the related object.

The information processing method of the present disclosure acquires an image, detects an object that detects a region including a person and a region including an object other than the person from the image, and selects an object that has a predetermined relationship with the person as a related object. and estimating a carrying method for the related object based on a region including the related object.

According to the present disclosure, it is possible to provide an information processing device and an information processing method that are capable of estimating how a person carries an object.

1 is a diagram showing the configuration of an information processing apparatus according to a first embodiment; FIG. 3 is a flowchart illustrating a process performed by the information processing apparatus according to the first embodiment. FIG. 3 is a diagram showing the configuration of an information processing device according to a second embodiment. FIG. 3 is a diagram for explaining how to estimate a carrying method. FIG. 3 is a diagram for explaining a method for determining the length of a region including a related object. FIG. 3 is a diagram for explaining the shape of a region including related objects. FIG. 3 is a diagram for explaining the shape of a region including related objects. 7 is a flowchart illustrating a process performed by the information processing apparatus of Embodiment 2. FIG. FIG. 7 is a diagram for explaining a usage example of the information processing device according to the second embodiment. FIG. 7 is a diagram for explaining a usage example of the information processing device according to the second embodiment. FIG. 7 is a diagram for explaining a usage example of the information processing device according to the second embodiment. FIG. 7 is a diagram for explaining a usage example of the information processing device according to the second embodiment. FIG. 3 is a diagram showing the configuration of an information processing device according to a third embodiment. FIG. 3 is a diagram for explaining a method of correcting information regarding a region including a related object. FIG. 3 is a diagram for explaining a method of correcting information regarding a region including a related object. FIG. 3 is a diagram for explaining a method of correcting information regarding a region including a related object. FIG. 3 is a diagram for explaining a method of correcting information regarding a region including a related object. 12 is a flowchart illustrating processing performed by the information processing apparatus of Embodiment 3. FIG. 7 is a diagram showing the configuration of an information processing device according to a fourth embodiment. 12 is a flowchart illustrating processing performed by the information processing apparatus of Embodiment 4. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fourth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fourth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fourth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fourth embodiment. FIG. 7 is a diagram showing the configuration of an information processing device according to a fifth embodiment. 12 is a flowchart illustrating processing performed by the information processing apparatus of Embodiment 5. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fifth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fifth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fifth embodiment. FIG. 7 is a diagram for explaining an example of use of the information processing apparatus according to the fifth embodiment. 1 is a diagram illustrating the configuration of a computer that constitutes an information processing device of the present disclosure.

<Embodiment 1>
Embodiment 1 of the present disclosure will be described using FIGS. 1 and 2.
FIG. 1 is a block diagram showing the configuration of an information processing apparatus 10 according to the first embodiment. As shown in FIG. 1, the information processing device 10 includes an image acquisition section 11, an object detection section 12, a related object identification section 13, and a carrying method estimation section 14.

The image acquisition unit 11 acquires an image. The object detection unit 12 detects an area including a person and an area including an object other than the person from the image. Note that the object other than the person is, for example, an object that can be carried by the person. Objects that can be carried by a person may include various objects classified as furniture, fruit, musical instruments, tools, appliances, and the like. The related object identifying unit 13 identifies objects that have a predetermined relationship with a person as related objects. Note that the predetermined relationship includes that the person is carrying an object. A person carrying an object means, for example, that a person is holding an object in their hand. Furthermore, whether there is a predetermined relationship may be determined based on the distance between the person and the object, for example. The carrying method estimation unit 14 estimates the carrying method of the related object based on the area including the related object. For example, the carrying method may include carrying an object perpendicular to the ground, carrying the object horizontally to the ground, or carrying the object horizontally to the ground. This may include carrying the device at an angle.

Next, with reference to FIG. 2, a process for estimating the method of carrying an object owned by a person will be described. FIG. 2 is a flowchart illustrating processing performed by the information processing apparatus 10 of the first embodiment. First, the image acquisition unit 11 acquires an image (S101). In the subsequent process, the object detection unit 12 detects a region including a person and a region including an object other than the person from the image (S102). Next, the related object identifying unit 13 identifies an object that has a predetermined relationship with the person as a related object (S103). Then, the carrying method estimating unit 14 estimates the carrying method of the related object based on the area including the related object (S104).

In this manner, the information processing apparatus 10 of the first embodiment identifies objects that have a predetermined relationship with the person as related objects based on the person and objects other than the person in the image. Furthermore, it is estimated how the related object is being held based on the area containing the related object. This makes it possible to estimate how the person is holding the object.

<Embodiment 2>
Embodiment 2 of the present disclosure will be described using FIGS. 3 to 8. FIG. 3 is a block diagram showing the configuration of the information processing device 20 according to the second embodiment. As shown in FIG. 3, the information processing device 20 includes an image acquisition section 21, an object detection section 22, a related object identification section 23, and a carrying method estimation section 24.

The image acquisition unit 21 serves as an image acquisition means and acquires images from various devices. The various devices include, for example, an imaging device, a mobile terminal, and a storage device. The imaging device is not limited to a surveillance camera that monitors a specific area, but may also be a wearable camera that can be worn by a person, a drone, a vehicle-mounted camera, or the like. The mobile terminal may be a portable terminal such as a smartphone. The storage device may be a database, cloud storage, or the like. Moreover, such various devices may be configured integrally with the information processing device 20.

The object detection unit 22 serves as an object detection means and detects an area including a person and an area including an object other than the person from the image. The object detection unit 22 has learned feature amounts of a person, and detects a person from an image. It also learns the features of objects other than people, and detects objects other than people. The object other than the person is an object that can be carried by the person. Objects that can be carried by a person may include, for example, objects classified into classes such as furniture, fruit, musical instruments, tools, and appliances. Furthermore, the object information is not limited to objects classified into these classes, but may include objects classified into various classes in known data sets such as ImageNet and OpenImage.

To detect the above-mentioned objects, for example, R-CNN (Region Based Convolutional Neural Networks), YOLO (You Look Only Once), SSD (Single Shot multibox Detector), etc. method can be used. Note that the object detection unit 22 may realize the process of detecting an area including a person and the process of detecting an area including an object other than a person using different methods. Further, the object detection section 22 may be configured to be divided into a first detection section that detects an area that includes a person, and a second detection section that detects an area that includes an object other than a person. Furthermore, the first detection section and the second detection section may be incorporated into different devices.

The related object identifying unit 23 serves as an object identifying means, and identifies objects that have a predetermined relationship with a person as related objects based on the area containing the person and the area containing objects other than the person detected by the object detection unit 22. do. The predetermined relationship includes that the person is carrying an object. A person carrying an object means, for example, that a person is holding an object in their hand. The related object specifying unit 23 can also specify a plurality of related objects when a plurality of related objects exist.

Examples of a method for specifying related objects include a method for specifying, as a related object, an object in which the distance between the person and the object is shorter than a reference distance. As the reference distance, a value set in advance may be used, or a value corrected as appropriate may be used. When correcting the reference distance, the correction may be performed depending on the detected person or object. For example, the smaller the size of the detected person or object, the shorter the reference distance is corrected. In addition to this, the reference distance may be corrected to be longer as the position of the detected person or object is farther from the center of the image.

The distance between a person and an object may be calculated using the distance of a straight line connecting a point set on an area containing a person and a point set on an area containing an object other than the person. Furthermore, points may be set on a region including the person, taking into account the positions of the body parts that make up the person. For example, the position of a hand is specified as a body part, and a point on a person is set at the position of the hand. Note that when determining the body parts that make up a person, you may use known skeletal estimation technology, or you can use a learning device that has learned the features of each body part to find the position of the desired body part. It may be specified.

The carrying method estimating unit 24 serves as a carrying method estimating means, and estimates the carrying method of the related object based on the area including the related object. The carrying method estimating unit 24 estimates, for example, that a person carries an object perpendicular to the ground. It is also estimated that the person is carrying the object horizontally to the ground. Furthermore, it is estimated that the person is carrying the object at an angle to the ground. Furthermore, based on the definition described later, it is estimated whether the related object is held vertically, horizontally, or diagonally as a more specific carrying method. Note that "vertical holding" is an example of an estimation result indicating that the person is carrying the object perpendicular to the ground. Further, horizontal carrying is an example of an estimation result indicating that a person carries an object horizontally with respect to the ground. Further, "holding at an angle" is an example of an estimation result indicating that a person is carrying an object at an angle with respect to the ground.

Here, a method of estimating the carrying method will be explained using FIG. 4. FIG. 4 is a diagram for explaining how to estimate the carrying method. FIG. 4 shows an image 100 of a person and an umbrella. The carrying method estimating unit 24 acquires the first length 120 and the second length 130 from the region 110 including the related object specified by the related object specifying unit 23. In the example shown in FIG. 4, the related object specifying unit 23 specifies the umbrella as the related object, and acquires the first length 120 and the second length 130 from the area 110 including the related object. . Here, the first length 120 is the length along the first direction in the region 110 including the related object. Further, the second length 130 is the length along the second direction in the region 110 including the related object. Here, the description will be made assuming that the first direction is the vertical direction of the image, and the second direction is the horizontal direction of the image. Note that the first direction and the second direction may be determined based on an area including a person. For example, a direction along the height direction of the person is defined as a first direction, and a direction perpendicular to the first direction is defined as a second direction.

Then, the first length 120 and the second length 130 are compared. As a result of the comparison, if the first length 120 is longer than the second length 130 (first length 120>second length 130), it is determined that the related object is held vertically. On the other hand, if the second length 130 is longer than the first length 120 (first length 120<second length 130), it is determined that the related object is held sideways. . Further, if the first length 120 and the second length 130 are equal (first length 120=second length 130), it is determined that the related object is held obliquely.

The first length 120 and the second length 130 in the region 110 including the related object may be determined by the method shown in FIG. 5, for example. FIG. 5 is a diagram for explaining a method for determining the length of the region 110 including the related object. The first length 120 may be obtained by determining the distance between a point 121 and a point 122 set on the area 110 including the related object, as shown in FIG. More specifically, the distance between the points 121 and 122 in the vertical direction of the image 100 may be found. Note that the combination of points 121 and 122 may be automatically determined by calculation. For example, a combination that provides the longest distance between points 121 and 122 is searched for.

Similarly, the second length 130 may be obtained by determining the distance between a point 131 and a point 132 set on the area 110 including the related object. More specifically, the distance between the points 131 and 132 in the direction perpendicular to the vertical direction of the image 100 may be found.

The carrying method estimating unit 24 may further determine that the difference between the first length 120 and the second length 130 is longer than a predetermined length when determining whether to carry vertically or horizontally. . Thereby, when the difference between the first length 120 and the second length 130 is small, it is possible to suppress variations in the estimation results of the carrying method. Further, when determining whether the device is held diagonally, it may be determined that the difference between the first length 120 and the second length 130 is shorter than a predetermined length. Thereby, even if there is some difference between the first length 120 and the second length 130, it can be determined that the related object is being held obliquely.

Further, the area 110 containing the related object is not limited to the rectangular area surrounding the related object as shown in FIGS. 4 and 5, but may have various shapes as shown in FIGS. 6A and 6B. 6A and 6B are diagrams for explaining the shape of the region 110 including related objects. For example, the shape of the area may be a circle surrounding the related object, like the area 111 including the related object shown in FIG. 6A. Alternatively, the shape may be along the related object, as in the area including the related object shown in FIG. 6B.

The carrying method estimating unit 24 is not limited to estimating the carrying method based on the magnitude relationship between the first length 120 and the second length 130, and may estimate the carrying method using various methods. . For example, the method of carrying the related object may be estimated based on the ratio of the first length 120 and the second length 130. In such a case, if the ratio of the second length 130 to the first length 120 is less than 1, it may be assumed that the related object is held vertically. Furthermore, if the ratio of the second length 130 to the first length 120 is greater than 1, it may be assumed that the related object is being held horizontally. Furthermore, if the ratio of the second length 130 to the first length 120 is equal to 1, it may be assumed that the related object is held obliquely. As the ratio of the second length 130 to the first length 120, the aspect ratio of the region 110 including the related object may be used.

Next, with reference to FIG. 7, a process for estimating the method of carrying an object owned by a person will be described. FIG. 7 is a flowchart illustrating an information processing method performed by the information processing apparatus 20 of the second embodiment.

The image acquisition unit 21 acquires images 100 from various devices (S201). Subsequently, the object detection unit 22 detects an area including a person and an area including an object other than the person from the image 100 acquired by the image acquisition unit 21 (S202). Then, the related object identifying unit 23 identifies objects other than the person who has a predetermined relationship with the person detected by the object detecting unit 22 as related objects (S203).

The carrying method estimating unit 24 acquires the first length 120 and the second length 130 from the region 110 including the related object specified by the related object specifying unit 23 (S204). Thereafter, it is determined whether the first length 120 and the second length 130 are equal (S205). If it is determined in S205 that they are equal, it is output that the person is carrying the object at an angle to the ground. It outputs that the related object is being held diagonally (S206) and ends the flow.

On the other hand, if it is determined in S205 that the first length 120 and the second length 130 are not equal, it is further determined whether the first length 120 is longer than the second length 130 ( S207). Then, in the determination in S207, if it is determined that the first length 120 is longer than the second length 130, it outputs that the related object is held vertically (S208) and ends the flow. . On the other hand, if it is determined in S207 that the first length 120 is not longer than the second length 130, it is outputted that the related object is held sideways (S209), and the flow end.

Note that in S206, the method of carrying the related object is to output that the related object is held diagonally, but the output mode is not limited to this. Instead of holding the object diagonally, various outputs may be provided to indicate that the person is holding the object at an angle to the ground. Similarly, the output obtained by the process of S208 is not limited to holding the object vertically, but may be various outputs indicating that the person is holding the object perpendicular to the ground. Furthermore, the output obtained by the process of S209 is not limited to horizontal carrying, but may be various outputs indicating that the person is carrying the object horizontally with respect to the ground.

Next, an example of use of the information processing device 20 according to the second embodiment will be described using FIGS. 8A to 8D. 8A to 8D are diagrams for explaining usage examples of the information processing device 20 of the second embodiment. FIG. 8A shows a situation where the surveillance camera 210 is photographing a person A carrying an umbrella B. In such a situation, the information processing device 20 cooperates with the surveillance camera 210 to acquire the image 100.

The information processing device 20 detects a region containing a person and a region containing objects other than the person from the image 100. As a result, as shown in FIG. 8B, an area 101 including a person and an area 102 including an object other than the person are detected. Then, as shown in FIG. 8C, the information processing device 20 connects a point 151 set on the area 101 containing the detected person and a point 152 set on the area 102 containing objects other than the person. Determine whether the straight line distance is less than the reference distance. This specifies whether the region 102 containing objects other than people is the region 110 containing related objects.

Then, as shown in FIG. 8D, the information processing device 20 acquires the first length 120 and the second length 130 from the region 110 including the related object. The information processing device 20 then compares the first length 120 and the second length 130. As a result of the comparison, the first length 120 is determined to be no longer than the second length 130 in this example. As a result, it is estimated that the related object is being held horizontally.

In this way, according to the configuration of the second embodiment, the related object identifying unit 23 can specify an object that has a predetermined relationship with a person as a related object. Furthermore, the carrying method estimating unit 24 can estimate how the related object is being carried based on the area 110 including the related object. This makes it possible to estimate how the person is carrying the object.

Note that the second embodiment of the present disclosure is not limited to the above-described aspect, and may be configured as follows, for example.
The information processing device 20 may be configured to record a processing history for each person detected by the object detection unit 22. Further, the processing history may include that the processing for estimating the carrying method of the related object has not been performed. Furthermore, the estimation results for each image may be linked to the same person. According to such a configuration, it is possible to prevent the same process from being repeated for a person who has already undergone the process of estimating the carrying method of the related object. Additionally, if it is not possible to estimate how a person is holding a related object from a specific image, the system considers the carrying method estimated from other images to estimate how the person is holding the related object. be able to.

Further, the object detection unit 22 may be configured to suppress detection of objects other than elongated objects when detecting the region 102 including objects other than people. Alternatively, the configuration may be such that objects other than long objects are not detected. According to such a configuration, it is possible to suppress estimation of the carrying method from being performed for related objects that are not suitable for estimation of the carrying method.

To explain in more detail, when a square object is specified as a related object, the first length 120 and the second length 130 determined from the region 110 including the related object are equal. Therefore, the carrying method estimating unit 24 estimates that the square object is always held obliquely, regardless of the actual carrying method. On the other hand, if the object detection unit 22 is configured to suppress or not detect objects other than elongated objects, detection of square objects can be suppressed. Therefore, it is possible to prevent the carrying method from being estimated for objects that are not suitable for carrying method estimation.

Further, the carrying method estimating unit 24 may output the estimated carrying method if the estimation results of the carrying method are the same for a predetermined number of times or more. Furthermore, if the time period during which it is estimated that the same carrying method is used continues for a predetermined period of time or more, the estimated carrying method may be output. According to such a configuration, the estimation accuracy of the carrying method can be improved.

<Embodiment 3>
Embodiment 3 of the present disclosure will be described using FIGS. 9 to 11. The information processing apparatus of this embodiment is different from the information processing apparatus of the first or second embodiment described above in that it further includes an area adjustment section.

FIG. 9 is a block diagram showing the configuration of the information processing device 30 according to the third embodiment. The information processing device 30 includes at least an image acquisition section 31, an object detection section 32, a related object identification section 33, a carrying method estimation section 34, and an area adjustment section 35.

The area adjustment unit 35 corrects the information regarding the area including the related object based on any of the information regarding the image, the information about the person, and the information about objects other than the person. The information regarding the image may include information about the device that generated the image. The information on the device that generated the image may include, for example, the position of the imaging device, the imaging direction, the angle of depression, the magnification of the lens, and the like. The person information may include the position of the person on the image, the direction of movement of the person, and the like. The information on objects other than people may include the position of objects other than people on the image, the moving direction of objects other than people, and the like. Note that the information on a person and the information on an object other than a person may be obtained by taking into consideration images acquired in the past. For example, the information includes the position of a person or object on an image at a specific time, the line of movement of the person or object, and the like.

Here, a method for correcting information regarding a region including a related object will be described using FIGS. 10A to 10D. FIGS. 10A to 10D are diagrams for explaining a method of correcting information regarding a region including a related object. FIG. 10A shows an image 100 depicting a person walking toward the lower left of the page and carrying an umbrella.

The area adjustment unit 35 corrects information regarding the area including the related object based on information about the person. Here, the moving direction of the person is used as the information about the person. First, the area adjustment unit 35 determines the moving direction of the person appearing in the image 100. The moving direction of the person can be determined from, for example, the person's line of sight or the person's line of movement. Alternatively, it may be obtained by analyzing optical flow or the like. Then, the angle formed by the direction of movement of the person and the reference direction is determined. The reference direction may be, for example, the vertical direction or the horizontal direction in the image 100. Here, the reference direction will be explained as the horizontal direction in the image 100. As a result, the angle Θ between the moving direction V of the person and the reference direction is determined as shown in FIG. 10B. In this example, the angle formed by the direction of movement of the person and the reference direction is determined, but the angle formed by the direction of movement of the object and the reference direction may be determined instead of the direction of movement of the person.

Then, the area adjustment unit 35 corrects information regarding the area including the related object identified from the image 100. For example, as shown in FIG. 10C, the first length 120 and second length 130 in the region 110 including the related object are multiplied by a predetermined function using an angle as an argument. As a result of such processing, as shown in FIG. 10D, the first length 125 after correction and the second length 135 after correction are obtained by correcting the first length 120 and the second length 130. can be found.

The area adjustment unit 35 is not limited to the method of performing correction based on the moving direction of the person described above, but may also perform correction using the following method. For example, when correction is performed based on information regarding the device that generated the image 100, a function may be defined using arguments such as the position of the imaging device, the imaging direction, the angle of depression, and the magnification of the lens. Such a function may be for converting into the size or position of a region containing the related object detected when the related object is imaged from a specific viewpoint or distance, for example.

Further, the region adjustment unit 35 may correct information regarding the region including various related objects, without being limited to the first length 120 and the second length 130 in the region 110 including the related object. For example, the ratio of the second length to the first length is corrected.

Next, with reference to FIG. 11, a process of estimating a method of carrying a related object based on information regarding a region including the corrected related object will be described. FIG. 11 is a flowchart illustrating processing performed by the information processing device 30 of the third embodiment. Note that some explanations of processes that overlap with those of the above-described embodiments are omitted.

The information processing device 30 identifies the region 110 containing the related object from the image 100 (S301 to S303), as in the above-described embodiment. Furthermore, the information processing device 30 obtains the first length 120 and the second length 130 from the region 110 including the related object (S304).

Next, the area adjustment unit 35 acquires information regarding the image (S305). Then, the area adjustment unit 35 corrects the information regarding the area including the related object based on the information regarding the acquired image (S306). The information corrected in this process is, for example, the first length 120 and the second length 130 in the region 110 including the related object acquired in S304. Thereafter, the information processing device 30 estimates how to carry the related object based on the corrected first length 125 and second length 135 (S307 to S311).

Note that the processing order is not limited to the processing order described in the flowchart illustrated in FIG. 11, and the area adjustment unit 35 may execute the process of obtaining information regarding the image (S305) after obtaining the image in S301. Similarly, part of the processing in S306 may be executed prior to S302 to S304. Part of the processing in S306 is processing for calculating a correction term based on information regarding the image. Further, the area adjustment unit 35 may execute part of the processing in S305 and S306 in parallel with S302 to S304.

In this manner, according to the configuration of the third embodiment, the area adjustment unit 35 can correct information regarding the area including the related object based on information regarding the image 100. With this, it is possible to estimate the method of carrying a related object, taking into consideration the position and orientation of a person or an object other than a person in the image.

<Embodiment 4>
Embodiment 4 of the present disclosure will be described using FIGS. 12 to 15. The information processing apparatus of this embodiment is different from any of the information processing apparatuses described in Embodiments 1 to 3 described above in that it further includes a detailed specifying section.

FIG. 12 is a block diagram showing the configuration of the information processing device 40 according to the fourth embodiment. The information processing device 40 includes at least an image acquisition section 41 , an object detection section 42 , a related object identification section 43 , a carrying method estimation section 44 , and a detailed identification section 46 .

The details specifying unit 46 specifies details of the related object based on the method of carrying the related object. Further, when the method of carrying the related object is a predetermined carrying method, the details specifying unit 46 specifies the details of the related object. The details of the related object may include, for example, not only the specific name of the related object, but also its shape, color, size, and the like. Additionally, the process of identifying details of the relevant object may be achieved by using a detector that is tuned to detect the specific object. For example, if it is estimated that the relevant object is being carried at an angle to the ground, the first detector may be used. The first detector is a detector for detecting an object, such as a white cane, that is assumed to be carried at an angle to the ground. Similarly, a second detector may be used if it is estimated that the relevant object is being carried perpendicular to the ground. The second detector is, for example, a detector for detecting an object such as an umbrella that is assumed to be carried perpendicular to the ground. Furthermore, if it is estimated that the relevant object is being carried horizontally with respect to the ground, a third detector may be used. The third detector is a detector for detecting an object such as a bag that is assumed to be carried horizontally with respect to the ground.

The first detector may include multiple detectors. Similarly, the second detector and the third detector may have a plurality of detectors. Furthermore, the first detector may be capable of detecting the same type of object that the second detector or the third detector is capable of detecting. Similarly, the second detector may be capable of detecting the same type of object that the first detector or the third detector is capable of detecting. Further, the third detector may be capable of detecting the same type of object as the first detector or the second detector.

Furthermore, when the first detector is composed of a plurality of detectors, a label may be given to a specific detector. Examples of the labels include "walking aid" and "rain gear." Similarly, labels may be given to a plurality of detectors constituting the second detector and the third detector.

Note that the detail specifying unit 46 does not necessarily need to include the first detector, the second detector, and the third detector, and may be configured to include only any of them. In such a case, the process may be terminated without specifying the details of the related object, unless it is estimated that the related object is a specific carrying method.

The process of identifying details of the related object is not limited to the method using the above-described detector, but may also be performed by searching a database that includes the carrying method as a search key.

Further, in addition to the carrying method estimated by the carrying method estimating unit 44, reference information may be taken into consideration to specify the details of the related object. The reference information may include, for example, attributes of the person. The person's attributes include information such as age, gender, and facial expression. Thereby, when reference information indicating that the person is an elderly person is acquired, it is possible to perform processing such as preferentially applying a detector labeled with a "walking aid".

The reference information is not limited to the example described above, and may include information regarding the environment. The information regarding the environment includes, for example, information indicating the presence or absence of locations, facilities, and equipment, such as intersections, station premises, and passageways with Braille blocks. As a result, when reference information indicating that a related object exists around a Braille block is obtained, processing can be performed such as giving priority to a detector labeled as "walking aid". I can do it.

Further, the reference information may include information obtained from an external information providing device. Examples of the information acquired from such an external information providing device include information regarding the weather and traffic. As a result, when information such as rain or snow is acquired, it is possible to perform processing such as preferentially applying a detector labeled with a "walking aid" label.

Next, a series of steps up to executing a predetermined process based on the estimated carrying method will be described using FIG. 13. FIG. 13 is a flowchart illustrating processing performed by the information processing device 40 of the fourth embodiment. Note that some explanations of processes that overlap with those of the first to third embodiments described above are omitted.

The information processing device 40 estimates the method of carrying the related object, as in the first to third embodiments described above (S401 to S404). Next, the detail specifying unit 46 determines whether the related object is held obliquely (S405). If it is determined that the related object is held obliquely, a process of specifying details of the related object using the first detector is performed (S406), and the flow ends.

On the other hand, if it is not determined that the related object is held diagonally, it is determined whether the related object is held vertically (S407). If it is determined that the related object is held vertically, a process of specifying details of the related object using the second detector (S408) is performed, and the flow ends.

Further, if it is determined in S407 that the related object is not held vertically, a process of specifying details of the related object using the third detector (S409) is performed, and the flow ends.

Note that in the flowchart illustrated in FIG. 13, the details of the related object are specified using either the first detector, the second detector, or the third detector. The flow may be configured with some exceptions. For example, in the process of S405, if it is not determined that the related object is being held diagonally, the flow may be ended without executing S407. Alternatively, a method different from S405 may be used to determine whether the related object is being carried at an angle to the ground. Similarly, a method different from S407 may be used to determine whether the related object is being carried perpendicular to the ground. Then, the details of the related object may be specified using a detector according to these determination results.

Next, an example of how the information processing device 40 of the fourth embodiment is used will be described using FIGS. 14A to 14D. 14A to 14D are diagrams for explaining usage examples of the information processing device 40 of the fourth embodiment.

In the usage example to be discussed here, a situation will be described assuming that a person A carrying a white cane C is photographed by the surveillance camera 210, as shown in FIG. 14A. The information processing device 40 cooperates with the surveillance camera 210 to acquire the image 100. The information processing device 40 then detects an area 101 containing a person and an area 102 containing objects other than the person from the image 100. As a result, as shown in FIG. 14B, an area 101 including a person and an area 102 including an object other than the person are detected.

The information processing device 40 identifies whether the detected person and the object other than the person are related objects based on the area 101 including the person and the area 102 including the object other than the person. Further, the information processing device 40 defines the region 102 including an object other than the person identified as a related object as the region 110 including the related object, and defines the first length 120 and the second length in the region 110 including the related object. 130 is obtained. Thereby, as shown in FIG. 14C, the first length 120 and the second length 130 in the region 110 including the related object can be obtained. In this example, since the first length 120 and the second length 130 are equal, the related object included in the related object containing area 110 is held diagonally by a person in the person containing area 101. Presumed.

Thereafter, the information processing device 40 selects a detector from among the first detector 170, the second detector 171, and the third detector 172 according to the carrying method, as shown in FIG. 14D. The first detector 170 in this example is a detector that is applied when the related object is estimated to be held at an angle, and can detect a white cane. On the other hand, the second detector 171 and the third detector 172 are detectors that are applied when the related object is estimated to be held vertically or horizontally, and are used to detect a walking cane or an umbrella. It is shown as a detector. In this example, since the information processing device 40 estimates that the related object is being held obliquely, it uses the first detector 170 to identify details of the related object. Through the above processing, the information processing device 40 identifies that the related object is a white cane.

In this way, according to the configuration of the fourth embodiment, the detail specifying unit 46 can specify the details of the related object according to the estimated carrying method. Thereby, details of the related object can be specified depending on the carrying method.

<Embodiment 5>
Embodiment 5 of the present disclosure will be described using FIGS. 15 to 17A to 17D. The information processing apparatus of this embodiment is different from any of the information processing apparatuses described in Embodiments 1 to 4 described above in that it further includes a notification section.

FIG. 15 is a block diagram showing the configuration of an information processing device 50 according to the fifth embodiment. The information processing device 50 includes at least an image acquisition section 51 , an object detection section 52 , a related object identification section 53 , a carrying method estimation section 54 , and a notification section 57 .

The notification unit 57 provides notification regarding the related object based on the method of carrying the related object. Further, the notification unit 57 provides notification regarding the related object when the estimated carrying method is a predetermined carrying method. There are various types of notifications, such as a warning or a request for assistance. Targets to be notified include people who have a predetermined relationship with the related object, people who are around the device that acquired the image 100, and the like. In addition to this, the notification may be sent to a person in a remote location such as a monitoring center. Further, the notification unit 57 may change the content of the notification depending on the method of carrying the related object. For example, the length, number of times, frequency, interval, etc. of notifications may be made different.

For example, the notification unit 57 may issue a notification when it is determined that the method of carrying the related object is a method of carrying that may cause harm to the surroundings. The carrying method that may cause harm may simply be a determination of whether a related object is being carried by a particular carrying method. Furthermore, in addition to the carrying method, the size of related objects and the degree of crowding may be taken into consideration. For example, when considering the size of a related object, the larger the related object, the more likely it is to be determined that it is likely to cause harm to the surroundings. Furthermore, when considering the degree of crowding, the more the degree of crowding indicates that the place is crowded, the more likely it is to be determined that there is a possibility of harm to the surroundings. Note that the degree of crowding may be calculated based on the number of people in the image 100, or the degree of crowding calculated by another device may be used.

The notification unit 57 may take the details of the related object into consideration and provide notification regarding the related object. Further, the content of the notification, the conditions for notification, and the target for notification may be changed depending on the details of the related object. For example, different warnings may be given for people who walk with a white cane vertically and those who walk horizontally. Note that the details of the related object may be specified by the method using the aforementioned detector, or may be specified by another method.

The notification unit 57 also provides notification via various notification devices. Various notification devices include, for example, audio output devices such as speakers, and display devices such as displays. Further, these notification devices may be installed at specific locations such as roads, facilities, monitoring centers, etc., or may be portable devices that can be moved or carried. Furthermore, it may have directionality that allows information to be transmitted only to a specific person. Note that various notification devices and the information processing device 50 may be configured integrally.

Next, a series of steps up to executing a predetermined process based on the estimated carrying method will be described using FIG. 16. FIG. 16 is a flowchart illustrating processing performed by the information processing device 50 of the fifth embodiment. Note that some descriptions of processes that overlap with those of the first to fourth embodiments described above are omitted.

The information processing device 50 estimates the method of carrying the related object, as in the first to fourth embodiments described above (S501 to S504). Then, the notification unit 57 determines whether the related object is being held horizontally (S505). If it is determined that the related object is being held horizontally, a notification process is performed (S506), and the flow ends. On the other hand, the flow also ends if it is determined in S505 that the related object is not held sideways.

Note that in the flowchart illustrated in FIG. 16, the notification is performed only when the related object is held horizontally in S505, but the notification may be performed even when the related object is not held horizontally. At that time, the content of the notification may be different. Further, the notification may be performed when it is determined that the related object is being held vertically or diagonally. Furthermore, the content of the notification may vary depending on the estimated type of carrying method. Further, a method different from S505 may be used to determine whether the related object is being carried horizontally with respect to the ground.

Next, an example of how the information processing device 50 of the fifth embodiment is used will be described using FIGS. 17A to 17D. 17A to 17D are diagrams for explaining usage examples of the information processing device 50 of the fifth embodiment.

In the example of use discussed here, a situation will be described assuming that a person A carrying an umbrella B is photographed by the surveillance camera 210, as shown in FIG. 17A. The information processing device 50 cooperates with the surveillance camera 210 to acquire the image 100. Then, the information processing device 50 detects an area 101 including the person A and an area 102 including an object other than the person from the image 100. As a result, as shown in FIG. 17B, an area 101 including the person A and an area 102 including objects other than the person are detected.

The information processing device 50 specifies whether the detected person and the object other than the person are related objects based on the area 101 including the person and the area 102 including the object other than the person. Furthermore, the information processing device 50 defines the region 102 that includes an object other than the person identified as the related object as the region 110 that includes the related object, and defines the first length 120 and the second length of the region 110 that includes the related object. 130 is obtained. Thereby, as shown in FIG. 17C, the first length 120 and the second length 130 in the region 110 including the related object can be obtained. In this example, since the second length 130 is longer than the first length 120, it is estimated that the related object is being held sideways by person A.

Thereafter, the information processing device 50 determines, based on the estimated carrying method, whether the related object is being carried in a carrying manner that may cause harm to the surroundings. As a result of the determination, it is determined that the related object is being carried in a manner that may cause harm to the surrounding area, so the information processing device 50 detects the surroundings of the surveillance camera 210 that acquired the image 100, as shown in FIG. 17D. Through a speaker 220 installed at , a warning is given to person A regarding how to carry umbrella B, which has been identified as a related object.

In this manner, according to the configuration of the fifth embodiment, the notification unit 57 can provide notification according to the estimated carrying method. This allows notifications to be made in accordance with the carrying method.

Note that the fifth embodiment of the present disclosure is not limited to the above-described aspect, and may be configured as follows, for example.
The information processing device 50 may notify a person registered in advance. For example, when the notification unit 57 determines to notify, the information processing device 50 notifies a person registered in advance. Further, the notification may be made only to those who satisfy the notification conditions from among the persons registered in advance.

The notification conditions may include, for example, when the notification unit 57 determines to issue a notification based on the image 100, the condition is that the user is in the vicinity of the device that generated the image 100. Further, the process of identifying a person who has been registered in advance may be performed in cooperation with the surveillance camera 210, using known face recognition technology. Furthermore, in addition to facial recognition, various technologies using location information such as geofencing may be used to identify whether a pre-registered person exists in the area surrounding the device that generated the image 100. good.

Further, the information processing device 50 may store the person detected when the notification unit 57 determines to issue the notification in the storage device in association with information regarding the notification. The information regarding the notification may include the date and time of notification, the content of the notification, the method of notification, and the like. Furthermore, when notifying the same person who is already stored in the storage device, the notification unit 57 may use a method different from the method of notification performed in the past. For example, when it is determined that it is necessary to alert a specific person, if there is a history of alerting that person through a loudspeaker in the past, then instead of alerting the person through a loudspeaker, For example, a security guard located at the location would request caution through various notification devices.

(About hardware configuration)
In each embodiment of the present disclosure, each component of each device represents a functional unit block. A part or all of each component of each device is realized by an arbitrary combination of an information processing device 500 and a program as shown in FIG. 18, for example. FIG. 18 is a block diagram illustrating an example of the hardware configuration of the information processing device 500 that implements each component of each device. Information processing device 500 includes the following configuration, as an example.
・CPU (Central Processing Unit) 501
・ROM (Read Only Memory) 502
・RAM (Random Access Memory) 503
・Program 504 loaded into RAM 503
- Storage device 505 that stores the program 504
- A drive device 507 that reads and writes from the recording medium 506
- Communication interface 508 connected to communication network 509
- Input/output interface 510 for inputting and outputting data
・Bus 511 connecting each component

Each component of each device in each embodiment is realized by the CPU 501 acquiring and executing a program 504 that realizes these functions. More specifically, the CPU 501 estimates a program for acquiring the image 100, a program for detecting an object from the image 100, a program for identifying an object that has a predetermined relationship with a person as a related object, and a method for carrying the related object. This is achieved by executing various programs such as programs and updating various parameters held in the RAM 503, the storage device 505, etc.

A program 504 that realizes the functions of each component of each device is stored, for example, in advance in a storage device 505 or ROM 502, and is read out by the CPU 501 as necessary. Note that the program 504 may be supplied to the CPU 501 via the communication network 509. Alternatively, the drive device 507 may read out a program stored in the recording medium 506 in advance and supply it to the CPU 501.

Furthermore, the program 504 can display the progress of processing or processing results via an output device. Furthermore, it is also possible to communicate with external devices via the communication interface. Note that the program 504 can also be recorded on a computer-readable (non-transitory) storage medium.

Each device can be realized not only in the above-mentioned configuration but also in various configurations. For example, each device may be realized by arbitrarily combining information processing device 500 and program 504, which are different for each component. Further, a plurality of components included in each device may be realized by an arbitrary combination of one information processing device 500 and the program 504.

In addition, some or all of the components of each device are realized by other general-purpose or dedicated circuits, processors, etc., or a combination thereof. These may be configured by a single chip or multiple chips connected via a bus.

A part or all of each component of each device may be realized by a combination of the circuits and the like described above and a program.

When some or all of the components of each device are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally located or distributed. Good too. For example, the information processing device, circuit, etc. may be realized in the form of an information processing system, such as a client and server system, a cloud computing system, etc., in which each is connected via a communication network.

The embodiments described above are preferred embodiments of the present disclosure, and the scope of the present disclosure is not limited to the embodiments described above. That is, it is possible for those skilled in the art to modify or substitute each of the embodiments described above without departing from the gist of the present disclosure, and to construct embodiments with various changes.

Part or all of the embodiments described above may be described as in the following supplementary notes, but are not limited to the following.

<Additional note 1>
an image acquisition unit that acquires an image;
an object detection unit that detects an area including a person and an area including an object other than the person from the image;
a related object identification unit that identifies an object that has a predetermined relationship with the person as a related object;
a carrying method estimation unit that estimates a carrying method of the related object based on a region including the related object;
An information processing device comprising:
<Additional note 2>
The carrying method estimation unit estimates the carrying method of the related object based on a first length in a region including the related object and a second length in a region including the related object.
The information processing device according to supplementary note 1.
<Additional note 3>
The carrying method estimation unit estimates a carrying method of the related object based on a ratio of the second length to the first length.
The information processing device according to supplementary note 2.
<Additional note 4>
The carrying method estimating unit estimates that the carrying method of the related object is vertically when the first length is longer than the second length by a predetermined length or more.
The information processing device according to supplementary note 2 or 3.
<Additional note 5>
The carrying method estimation unit estimates that the carrying method of the related object is horizontally when the second length indicates that the second length is longer than the first length by a predetermined length or more.
The information processing device according to any one of Supplementary Notes 2 to 4.
<Additional note 6>
The carrying method estimating unit estimates that the carrying method of the related object is to carry the related object diagonally, if the difference between the first length and the second length is shorter than the predetermined length.
The information processing device according to any one of Supplementary Notes 2 to 5.
<Additional note 7>
The carrying method estimation unit obtains a length along the vertical direction of the image as the first length in a region including the related object, and obtains a length along a direction perpendicular to the first length as the first length. obtained as the second length,
The information processing device according to any one of Supplementary Notes 2 to 6.
<Additional note 8>
comprising an area adjustment unit that corrects information regarding the area including the related object based on any of information regarding the image, information about the person, and information about objects other than the person;
The information processing device according to any one of Supplementary Notes 1 to 7.
<Additional note 9>
The information processing device according to appendix 7, wherein the information regarding the area including the related object includes the first length or the second length.
<Additional note 10>
The information regarding the image includes a moving direction of the person,
The area adjustment unit corrects information regarding the area including the related object based on a moving direction of the person and a reference direction.
The information processing device according to any one of appendix 8.
<Additional note 11>
comprising a details specifying unit that specifies details of the related object based on a method of carrying the related object;
The information processing device according to any one of Supplementary Notes 1 to 10.
<Additional note 12>
The details specifying unit specifies details of the related object when the carrying method is a predetermined carrying method.
The information processing device according to appendix 11.
<Additional note 13>
comprising a notification unit that provides a notification regarding the related object based on a method of carrying the related object;
The information processing device according to any one of Supplementary Notes 1 to 12.
<Additional note 14>
The notification unit provides a notification regarding the related object when the carrying method is a predetermined carrying method.
The information processing device according to appendix 13.
<Additional note 15>
an image acquisition means for acquiring an image;
object detection means for detecting an area including a person and an area including an object other than the person from the image;
identification means for identifying an object that has a predetermined relationship with a person as a related object;
carrying method estimating means for estimating a carrying method of the related object based on a region including the related object;
An information processing system equipped with.
<Additional note 16>
Get the image and
detecting an object from the image, which detects an area including a person and an area including an object other than the person;
Identify objects that have a predetermined relationship with a person as related objects,
estimating a method of carrying the related object based on a region including the related object;
Information processing method.
It should be noted that the forms of Supplementary Notes 15 to 16 above can be developed into the forms of Supplementary Notes 2 to 14, similar to Supplementary Note 1.

10, 20, 30, 40, 500 Information processing device 11, 21, 31, 41 Image acquisition unit 12, 22, 32, 42 Object detection unit 13, 23, 33, 43 Related object identification unit 14, 24, 34, 44 Carrying method estimation section 35 Area adjustment section 46 Detail specification section 57 Notification section 100 Image 501 CPU (Central Processing Unit)
502 ROM (Read Only Memory)
503 RAM (Random Access Memory)
504 Program 505 Storage device 506 Recording medium 507 Drive device 508 Communication interface 509 Communication network 510 Input/output interface 511 Bus

Claims (10)

an image acquisition unit that acquires an image;
an object detection unit that detects an area including a person and an area including an object other than the person from the image;
a related object identification unit that identifies an object that has a predetermined relationship with the person as a related object;
a carrying method estimation unit that estimates a carrying method of the related object based on a region including the related object;
An information processing device comprising: The carrying method estimation unit estimates the carrying method of the related object based on a first length in a region including the related object and a second length in a region including the related object.
The information processing device according to claim 1. The carrying method estimation unit estimates a carrying method of the related object based on a ratio of the second length to the first length.
The information processing device according to claim 2. The carrying method estimation unit obtains a length along the vertical direction of the image as the first length in a region including the related object, and obtains a length along a direction perpendicular to the first length as the first length. obtained as the second length,
The information processing device according to claim 2 or 3. comprising an area adjustment unit that corrects information regarding the area including the related object based on any of information regarding the image, information about the person, and information about objects other than the person;
An information processing device according to any one of claims 1 to 4. The information regarding the image includes a moving direction of the person,
The area adjustment unit corrects information regarding the area including the related object based on a moving direction of the person and a reference direction.
The information processing device according to claim 5. comprising a details specifying unit that specifies details of the related object based on a method of carrying the related object;
An information processing device according to any one of claims 1 to 6. comprising a notification unit that provides a notification regarding the related object based on a method of carrying the related object;
An information processing device according to any one of claims 1 to 7. an image acquisition means for acquiring an image;
object detection means for detecting an area including a person and an area including an object other than the person from the image;
identification means for identifying an object that has a predetermined relationship with a person as a related object;
carrying method estimating means for estimating a carrying method of the related object based on a region including the related object;
An information processing system equipped with. Get the image and
detecting an object from the image, which detects an area including a person and an area including an object other than the person;
Identify objects that have a predetermined relationship with a person as related objects,
estimating a method of carrying the related object based on a region including the related object;
Information processing method.

Images