JP2022095994A - Image processing system, image processing program, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of estimating the posture of a person with high accuracy based on a photographed image even when the installation height of a photographing device varies.

SOLUTION: An image processing system includes: an image acquisition part for acquiring an image obtained by photographing an entire photographing area by a photographing device installed at an installation position for overlooking a predetermined photographing area; a person region detection part for detecting a person region from an image; an information acquisition part for acquiring information on the height from a predetermined position of the installation position: and an attitude estimation part for estimating the posture of a person by machine learning based on the person region and information on the height.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an image processing system, an image processing program, and an image processing method.

In Japan, the longevity of life has become remarkable due to the improvement of living standards, the improvement of sanitary environment, and the improvement of medical standards due to the high economic growth after the war. For this reason, the aging society has a high aging rate, coupled with the decline in the birth rate. In such an aging society, it is expected that the number of people requiring long-term care will increase due to illness, injury, and aging.

In facilities such as hospitals and welfare facilities for the elderly, care recipients may fall while walking or fall from their beds and be injured. Therefore, a system for detecting the condition of the care recipient from the photographed image so that the staff such as the caregiver and the nurse can immediately rush to the care recipient when the care recipient becomes in such a state. Is under development. In order to detect the state of the care recipient or the like with such a system, it is necessary to detect the posture of the person to be detected with high accuracy from the captured image.

However, especially in an omnidirectional image taken with a wide-angle lens camera such as a fisheye lens camera, the appearance of the person in the image depends on the position of the person even if the person is the same person due to the distortion characteristics of the lens. It has the characteristic of changing.

Techniques for detecting the posture of a person from an image such as an omnidirectional image include those disclosed in the following Patent Documents 1 and 2.

In Patent Document 1, the personal ID of a person and the size of the head when standing upright in the image are registered in association with each other, the size of the image of the head of the person in the image is detected, and the detected image of the head is used. A technique for determining a person's posture based on size is disclosed. In Patent Document 2, a human region including a person is detected from an image, and the posture of the person is determined based on the image of the human region by a neural network or the like learned in advance by teacher data of a combination of the image of the human region and the posture. The estimation technique is disclosed.

Japanese Unexamined Patent Publication No. 2015-158952 Japanese Unexamined Patent Publication No. 2018-206321

However, the technique of Patent Document 1 has a problem that the posture cannot be estimated with high accuracy because the posture of the person is only determined based on the size of the image of the head on the image. The technique of Patent Document 2 has a problem that the estimation accuracy of the posture of a person based on an image is lowered when the height of the photographing device is changed due to a change in the installation location of the photographing device for photographing an image.

The present invention has been made to solve such a problem. That is, it is an object of the present invention to provide an image processing system, an image processing program, and an image processing method that can estimate the posture of a person with high accuracy based on the captured image even if the installation height of the photographing device fluctuates. ..

The above-mentioned problems of the present invention are solved by the following means.

(1) An image acquisition unit that acquires an image in which the entire imaging area is captured by an imaging device installed at an installation position that overlooks a predetermined imaging area, and a person area detection unit that detects a person area from the image. , An information acquisition unit that acquires height information from a predetermined position of the installation position, and a posture estimation unit that estimates a person's posture by machine learning based on the person area and the height information. An image processing system having.

(2) The posture estimation unit further includes a correction unit that corrects the person area based on the height information in accordance with the case where the height from the predetermined position is a preset reference height. Was previously learned as teacher data from the combination of the person area detected from the image taken by the image pickup device installed at the reference height and the posture of the person, and was corrected by the correction unit. The image processing system according to (1) above, which estimates the posture of a person based on the person area.

(3) The feature point estimation unit that estimates feature points related to the human body from the person area, and the height information according to the case where the height from the predetermined position is a preset reference height. Based on this, it further has a correction unit that corrects the feature points, and the posture estimation unit is estimated from the person area detected from the image taken by the photographing device installed at the reference height. The image processing system according to (1) above, wherein the combination of the feature points and the posture of the person is learned in advance as teacher data, and the posture of the person is estimated based on the feature points corrected by the correction unit. ..

(4) A procedure (a) for acquiring an image in which the entire shooting area is shot by a shooting device installed at an installation position overlooking a predetermined shooting area, and a procedure (b) for detecting a person area from the image. And the procedure (c) for acquiring the height information of the installation position from the predetermined position, and the procedure for estimating the posture of the person by machine learning based on the person area and the height information (). An image processing program for causing a computer to execute a process having d) and.

(5) The process further includes a procedure (e) for correcting the person area based on the height information in accordance with the case where the height from the predetermined position is a preset reference height. Then, in the procedure (d), the combination of the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person is learned in advance as teacher data. The image processing program according to (4) above, wherein the posture estimation unit estimates the posture of a person based on the person area corrected in the procedure (e).

(6) In the process, the height is matched with the procedure (f) of estimating a feature point related to the human body from the person area and the case where the height from the predetermined position is a preset reference height. Based on the above information, the procedure (g) for correcting the feature points is further provided, and in the procedure (d), the detection is performed from the image taken by the photographing device installed at the reference height. A person based on the feature points corrected in the procedure (g) by a posture estimation unit pre-learned using the combination of the feature points estimated from the person area and the posture of the person as teacher data. The image processing program according to (4) above, which estimates the posture of the above.

(7) An image processing method using an image processing system, the step (a) of acquiring an image in which the entire shooting area is taken by a shooting device installed at an installation position overlooking a predetermined shooting area, and the above. Based on the step (b) of detecting the person area from the image, the step (c) of acquiring the height information of the installation position from the predetermined position, the person area, and the height information. An image processing method comprising a step (d) of estimating a person's posture by machine learning.

(8) Further having a step (e) of correcting the person area based on the information of the height according to the case where the height from the predetermined position is a preset reference height, the step. In (d), the posture estimation unit learned in advance using the combination of the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person as teacher data. The image processing method according to (7) above, wherein the posture of the person is estimated based on the person area corrected in the step (e).

(9) The height information is used according to the step (f) of estimating the feature points related to the human body from the person area and the case where the height from the predetermined position is a preset reference height. Based on this, the person further includes a step (g) for correcting the feature points, and in the step (d), the person detected from the image taken by the photographing device installed at the reference height. The posture of the person is estimated based on the feature points corrected in the step (g) by the posture estimation unit which has been learned in advance using the combination of the feature points estimated from the region and the posture of the person as teacher data. The image processing method according to (7) above.

A person area is detected from the image of the entire shooting area taken by the shooting device installed at the installation position that overlooks a predetermined shooting area, and based on the height information in which the shooting device is installed and the person area, Estimate the posture of a person by machine learning. As a result, even if the installation height of the photographing device fluctuates, the posture of the person can be estimated with high accuracy based on the captured image.

It is a figure which shows the schematic structure of the image recognition system including the image recognition apparatus which concerns on embodiment. It is a block diagram which shows the hardware composition of an image recognition apparatus. It is a block diagram which shows the function of the control part of an image recognition apparatus. It is a figure which shows the person area detected in the image. It is explanatory drawing which shows the feature point. It is explanatory drawing for demonstrating the correction of a feature point by a correction part. It is explanatory drawing for explaining a feature point before and after a correction on an image. It is a flowchart which shows the operation of an image recognition apparatus. It is a block diagram which shows the function of the control part of an image recognition apparatus. It is a flowchart which shows the operation of an image recognition apparatus. It is a block diagram which shows the function of the control part of an image recognition apparatus. It is a flowchart which shows the operation of an image recognition apparatus.

Hereinafter, an image processing system, an image processing program, and an image processing method according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

(First Embodiment)
FIG. 1 is a diagram showing a schematic configuration of an image recognition system 10 including an image recognition device 100 according to an embodiment.

The image recognition system 10 includes an image recognition device 100, a photographing device 200, a communication network 300, and a mobile terminal 400. The image recognition device 100 is communicably connected to the photographing device 200 and the mobile terminal 400 by the communication network 300.

The image recognition device 100 receives an image taken by the photographing device 200 (hereinafter, also simply referred to as “image 600” (see FIG. 4 and the like)) from the photographing device 200, and receives the subject 500 who is a person from the image 600. The included area is detected as the person area 610. The image recognition device 100 can detect the person area 610 by detecting the area where the object (object) exists on the image 600 and estimating the category of the object included in the detected area. The area where the object exists is detected on the image 600 as a rectangle (candidate rectangle) including the object. The image recognition device 100 detects the person area 610 by detecting the candidate rectangles whose object category is presumed to be a person among the detected candidate rectangles. The image recognition device 100 detects the posture or behavior of the subject 500 based on the person area 610. Postures include standing, mid-waist, sitting, lying down, crouching, and sitting. Behaviors include getting up, getting out of bed, falling, and falling. As will be described later, the person area 610 is corrected based on the height of the installation position of the photographing device 200 in accordance with the case where the height of the installation position of the photographing device 200 is a preset reference height. In addition, an event related to the subject 500 can be detected from the estimated posture or behavior. The event is a change in the state recognized by the image recognition device 100 or the like with respect to the target person 500, and is an event that should be notified (notified) to the staff 80 such as getting up, getting out of bed, falling, and falling. .. When the image recognition device 100 detects an event, the image recognition device 100 transmits an event notification notifying the content of the event to the mobile terminal 400. The image recognition device 100 can detect the subject 500 as a person region 610 on the image 600 by a neural network (hereinafter referred to as “NN”). Examples of the method for detecting the target object by the NN include known methods such as Faster R-CNN, Fast R-CNN, and R-CNN. The image recognition device 100 is composed of a computer. The image recognition device 100 may be configured by, for example, a server.

The photographing device 200 is composed of, for example, a near-infrared camera, is installed at an installation position overlooking a predetermined photographing area, and photographs the entire photographing area. The installation position of the photographing device 200 is, for example, the ceiling of the living room of the subject 500. The predetermined photographing area is, for example, a three-dimensional area including the entire floor surface of the living room. Hereinafter, the photographing apparatus 200 will be described as being installed on the ceiling of the living room of the subject 500. The photographing device 200 irradiates near-infrared rays toward the photographing area by an LED (Light Emitting Device), and receives the reflected light of the near-infrared rays reflected by an object in the photographing area by a CMOS (Completion Metal Oxide Sensor) sensor. You can shoot the shooting area with. The image 600 may be a monochrome image having the reflectance of near infrared rays as each pixel.

The photographing device 200 may photograph the photographing area as a moving image having a frame rate of, for example, 15 fps to 30 fps. The image 600 includes a moving image and a still image. The photographing apparatus 200 transmits the image 600 to the image recognition apparatus 100 and the like.

The photographing device 200 may be configured by a sensor box having a computer. The sensor box is a device including a near-infrared camera, a body motion sensor, and the like. In this case, the image 600 is transmitted from the sensor box to the image recognition device 100. The sensor box may have a part or all of the functions of the image recognition device 100. The body movement sensor is a doppler shift type sensor that transmits and receives microwaves to and from the bed and detects the doppler shift of the microwave generated by the body movement (for example, respiratory movement) of the subject 500.

For the communication network 300, a network interface based on a wired communication standard such as Ethernet (registered trademark) may be used. For the communication network 300, a network interface based on a wireless communication standard such as Bluetooth (registered trademark) or 802.11 may be used. An access point 310 is provided in the communication network 300, and the mobile terminal 400 and the image recognition device 100 and the photographing device 200 are communicably connected by a wireless communication network.

The mobile terminal 400 receives the event notification from the image recognition device 100 and displays the content of the event notification. The event notification may include detection results of getting up, getting out of bed, falling, and falling, as well as detection results of microbody movement abnormalities and the like. The mobile terminal 400 may receive and display an image 600 from the photographing device 200 or the image recognition device 100. The mobile terminal 400 is composed of, for example, a smartphone.

FIG. 2 is a block diagram showing a hardware configuration of the image recognition device 100. The image recognition device 100 includes a control unit 110, a storage unit 120, a display unit 130, an input unit 140, and a communication unit 150. These components are connected to each other via the bus 160.

The control unit 110 is configured by a CPU (Central Processing Unit), and controls and performs arithmetic processing of each unit of the image recognition device 100 according to a program. The details of the function of the control unit 110 will be described later.

The storage unit 120 may be composed of a RAM (Random Access Memory), a ROM (Read Only Memory), and an SSD (Solid State Drive). The RAM temporarily stores programs and data as a work area of the control unit 110. The ROM stores various programs and various data in advance. The SSD stores various programs including an operating system and various data.

The display unit 130 is, for example, a liquid crystal display and displays various information.

The input unit 140 is composed of, for example, a touch panel and various keys. The input unit 140 is used for various operations and inputs.

The communication unit 150 is an interface for communicating with an external device. For communication, network interfaces according to standards such as Ethernet (registered trademark), SATA, PCI Express, USB, and IEEE 1394 can be used. In addition, a wireless communication interface such as Bluetooth (registered trademark), 802.11, 4G, etc. may be used for communication. The communication unit 150 receives the image 600 from the photographing device 200. The communication unit 150 transmits the event notification to the mobile terminal 400.

The details of the function of the control unit 110 will be described.

FIG. 3 is a block diagram showing the function of the control unit 110 of the image recognition device 100. The control unit 110 includes an image acquisition unit 111, a person area detection unit 112, a feature point estimation unit 113, a height information acquisition unit 114, a correction unit 115, and a posture estimation unit 116. The height information acquisition unit 114 constitutes an information acquisition unit.

The image acquisition unit 111 acquires the image 600 received from the photographing device 200 by the communication unit 150.

The person area detection unit 112 is composed of NN. The NN reflects the person area detection parameters obtained by learning to detect the person area. The person area detection unit 112 generates a feature map from which pixel features are extracted by a convolution calculation based on the image 600. The person area detection unit 112 detects the area where the object exists on the image 600 as a candidate rectangle from the feature map. The person area detection unit 112 can detect a candidate rectangle by a known technique using NN such as Faster R-CNN. The person area detection unit 112 calculates the reliability score for each predetermined category for each candidate rectangle. That is, the person area detection unit 112 calculates the reliability score for a predetermined category including the person. The confidence score is the likelihood for each predetermined category. The person area detection unit 112 can calculate the reliability score by a known technique using NN such as Faster R-CNN. Predetermined categories can be, for example, people, chairs, and equipment. The person area detection unit 112 detects each candidate rectangle in which the category with the highest reliability score is a person as the person area 610. The sum of the reliability scores calculated for each of the predetermined categories for one candidate rectangle is 1.

The person area detection unit 112 is preliminarily learned to estimate the person area 610 from the image 600 by using the teacher data of the combination of the image 600 and the person area 610 set as the correct answer for the image 600. .. As a result, the above-mentioned person area detection parameter is reflected in the person area detection unit 112.

FIG. 4 is a diagram showing a person region 610 detected in the image 600.

In the example of FIG. 4, the person area 610 is detected as a rectangular area surrounding the subject 500 who is a person. In this case, the person area 610 can be output together with the image 600 as the coordinates of two points forming any pair of rectangular pairs of vertices of the person area 610 in the image 600.

The feature point estimation unit 113 is composed of NN. The NN reflects the feature point estimation parameters obtained by learning for estimating the feature point 620 (hereinafter, also simply referred to as “feature point 620”) relating to the human body. The feature point estimation unit 113 estimates the feature point 620 based on the person area 610. The feature point 620 may include a joint point 621 and a pair of vertices 622 of the head rectangle 623. The head rectangle 623 is a rectangular area including the head of a person. The feature point estimation unit 113 can estimate the feature point 620 by a known technique using NN such as DeepPose. For DeepPose, see the known literature (Alexander Toshev, et al.
Details can be found in “DeepPose: Human Pose Optimization via Deep Neural Networks”, in CVPR, 2014).

FIG. 5 is an explanatory diagram showing a feature point 620. FIG. 5 also shows a person area 610 (more specifically, a frame of the person area 610).

In the example of FIG. 5, the white circles indicate the joint points 621, and the gray circles indicate the opposite vertices 622 of the head rectangle 623, respectively.

The feature point estimation unit 113 is previously trained to estimate the feature point 620 from the person area 610 by using the teacher data of the combination of the person area 610 and the feature point 620 set as the correct answer for the person area 610. Has been done. As a result, the feature point estimation unit 113 reflects the above-mentioned feature point estimation parameters.

The height information acquisition unit 114 acquires, for example, the height information of the installation position of the photographing apparatus 200, which is input by the user in the input unit 140, from a predetermined position. Hereinafter, the height of the installation position of the photographing device 200 from a predetermined position is also referred to as “height of the photographing device 200”, and the information of the height from the predetermined position is also referred to simply as “height information”. The predetermined position is an arbitrary and constant position, and may be, for example, a position on the floor surface.

The correction unit 115 is based on the height information in accordance with the case where the height of the photographing apparatus 200 is a preset reference height, and the feature point 620 (more specifically, the coordinates of the feature point 620 in the image 600). ) Is corrected.

FIG. 6 is an explanatory diagram for explaining the correction of the feature point 620 by the correction unit 115.

In FIG. 6, the photographing apparatus 200A is installed at an installation position 240 cm from the floor surface, which is a reference height. The photographing apparatus 200B is installed at an installation position 270 cm from the floor surface, which is higher than the reference height. When the same subject 500 fixed on the floor surface is photographed by the photographing device 200A and the photographing device 200B, the size of the subject 500 on the image 600 is increased even though the same subject 500 is photographed. different. Specifically, the subject 500 on the image 600 taken by the image pickup device 200B at a height of 270 cm from the floor surface is on the image 600 taken by the image pickup device 200A at a height of 240 cm from the floor surface. It is smaller than the subject 500. Such a change in the size of the subject 500 on the image 600 due to the change in the height of the photographing apparatus 200 is estimated by the posture estimation unit 116, which will be described later, of the posture of the subject 500 based on the feature point 620. It can reduce accuracy. Such a decrease in the estimation accuracy of the posture is caused by the teacher of the combination of the feature point 620 estimated from the image 600 taken by the photographing apparatus 200 of various heights and the posture set as the correct answer for the feature point 620. This can be prevented by training the posture estimation unit 116 using the data. However, the amount of teacher data required to maintain posture estimation accuracy increases. Therefore, even if the height of the photographing apparatus 200 fluctuates, the feature points 620 are corrected in order to realize highly accurate posture estimation without increasing the amount of required teacher data. Specifically, the feature point 620 is corrected according to the case where the image is taken by the image pickup device 200 having a reference height. That is, the feature points are so that the relative relationship between the distance and the position of each feature point 620 is the relative relationship between the distance and the position of each feature point 620 when the photographing device 200 is installed at the reference height. Correct 620.

As shown in FIG. 6, the height of the reference image plane with respect to the floor surface is α cm. Therefore, the distance between the reference image plane and the photographing device 200A is (240-α) cm, and the distance between the reference image plane and the photographing device 200B is (270-α) cm. Then, the ratio of the image 600 by the photographing apparatus 200A to the image 600 by the photographing apparatus 200B, which is the length that the object on the reference image plane is reflected in the image 600, is the ratio of (240-α) and (270-α). become. Therefore, the feature is such that the distance L from the center of the image 600 on the image 600 of the feature point 620 estimated from the image 600 captured by the photographing device 200B is the distance L'corrected by the following equation. The coordinates of the point 620 on the image 600 are corrected. The height of the reference image plane can be set to an appropriate constant value by experiment from the viewpoint of posture estimation accuracy.

L'= L × (270-α) / (240-α)
FIG. 7 is an explanatory diagram for explaining the feature points 620 before and after the correction on the image 600.

In FIG. 7, the center of the image 600 is indicated by a black dot, and the feature point 620 is indicated by a white circle. The feature point 620 whose distance from the center of the image 600 is L is the feature point before correction. The feature point 620 whose distance from the center of the image 600 is L'is the corrected feature point. As shown in FIG. 7, the coordinates of the feature point 620 are corrected by changing the distance from the center of the image 600 to the center of the image 600 without changing the direction of the feature point 620 with respect to the center of the image 600.

The posture estimation unit 116 is composed of NN. The posture estimation parameters obtained by learning for estimating the posture of a person are reflected in the NN. The posture estimation unit 116 estimates the posture of the subject 500 based on the corrected feature point 620.

The posture estimation unit 116 is preliminarily learned to estimate the posture from the feature point 620 by using the combination of the feature point 620 and the posture set as the correct answer for the feature point 620 as teacher data. As a result, the posture estimation unit 116 reflects the above-mentioned posture estimation parameters. The feature points 620 and the like used as the teacher data may be only those detected from the image 600 taken by the photographing apparatus 200 installed at the reference height. That is, even if the photographing device 200 in which the image 600 for which the posture is to be estimated is captured is installed at a height other than the reference height, the image captured by the photographing device 200 installed at a height different from the reference height. It is not necessary to newly prepare teacher data such as feature points 620 detected from 600.

The operation of the image recognition device 100 will be described.

FIG. 8 is a flowchart showing the operation of the image recognition device 100. This flowchart is executed by the control unit 110 according to the program stored in the storage unit 120.

The image acquisition unit 111 acquires an image 600 from the photographing device 200 via the communication unit 150 (S101).

The person area detection unit 112 detects the person area 610 from the image 600 (S102).

The feature point estimation unit 113 estimates the feature point 620 from the person area 610 (S103).

The correction unit 115 corrects the feature point 620 based on the height information of the photographing apparatus 200, in accordance with the case where the photographing apparatus 200 is assumed to be installed at the reference height (S104).

The posture estimation unit 116 estimates the posture of the subject 500 based on the corrected feature point 620 (S105).

This embodiment has the following effects.

The person area is detected from the image of the entire shooting area taken by the shooting device installed at the installation position that overlooks the predetermined shooting area, the feature points are estimated from the person area, and the height of the shooting device is the reference height. The feature points are corrected according to a certain case, and the posture of the person is estimated by machine learning based on the corrected feature points. As a result, even if the installation height of the imaging device fluctuates, the posture of the person can be estimated with high accuracy based on the captured image, and the posture estimation by machine learning due to the change in the installation height of the imaging device. It is possible to reduce the teacher data used for machine learning, which is necessary to prevent the decrease in accuracy.

(Second Embodiment)
A second embodiment of the present invention will be described. The differences between the present embodiment and the first embodiment are as follows. In the first embodiment, the feature points 620 estimated from the person area 610 are corrected based on the height information, and the posture is estimated based on the corrected feature points 620. On the other hand, in the present embodiment, the person area 610 is corrected based on the height information, and the posture is estimated based on the corrected person area 610. Since the present embodiment is the same as the first embodiment in other respects, duplicated description will be omitted or simplified.

FIG. 9 is a block diagram showing the function of the control unit 110 of the image recognition device 100. The control unit 110 includes an image acquisition unit 111, a person area detection unit 112, a height information acquisition unit 114, a correction unit 115, and a posture estimation unit 116.

The correction unit 115 corrects the person area 610 based on the height information in accordance with the case where the height of the photographing apparatus 200 is set as the reference height. Specifically, the correction of the coordinates of each feature point 620 performed in the first embodiment is performed for each pixel of the person region 610. As a result, each coordinate of the pixel of the person area is corrected, and the corrected person area 610 is calculated. Although the pixel density of the person area 610 may change due to the correction of the person area 610, a known conversion may be performed to return the corrected person area 610 to the pixel density of the person area 610 before correction.

The posture estimation unit 116 estimates the posture of the subject 500 based on the corrected person area 610.

The posture estimation unit 116 is preliminarily learned to estimate the posture from the person area 610 by using the combination of the person area 610 and the posture set as the correct answer for the person area 610 as teacher data. The person area 610 or the like used as the teacher data may be only those detected from the image 600 taken by the photographing device 200 installed at the reference height. That is, even if the photographing device 200 in which the image 600 for which the posture is to be estimated is taken is installed at a height other than the reference height, the image taken by the photographing device 200 installed at a height different from the reference height. It is not necessary to newly prepare teacher data such as the person area 610 detected from 600.

The operation of the image recognition device 100 will be described.

FIG. 10 is a flowchart showing the operation of the image recognition device 100. This flowchart is executed by the control unit 110 according to the program stored in the storage unit 120.

The image acquisition unit 111 acquires an image 600 from the photographing device 200 via the communication unit 150 (S201).

The person area detection unit 112 detects the person area 610 including the target person 500 from the image 600 (S202).

The correction unit 115 corrects the person area 610 based on the height information of the photographing device 200 according to the case where the photographing device 200 is installed at the reference height (S203).

The posture estimation unit 116 estimates the posture of the subject 500 based on the corrected person area 610 (S204).

This embodiment has the following effects.

The person area is detected from the image of the entire shooting area taken by the shooting device installed at the installation position that overlooks the predetermined shooting area, and the person area is corrected according to the case where the height of the shooting device is the reference height. Then, the posture of the person is estimated by machine learning based on the corrected person area. As a result, even if the installation height of the imaging device fluctuates, the posture of the person can be estimated with high accuracy based on the captured image, and the posture estimation by machine learning due to the change in the installation height of the imaging device. It is possible to reduce the teacher data used for machine learning, which is necessary to prevent the decrease in accuracy.

(Third Embodiment)
A third embodiment of the present invention will be described. The differences between the present embodiment and the first embodiment are as follows. In the first embodiment, the feature points 620 estimated from the person area 610 are corrected based on the height information, and the posture is estimated based on the corrected feature points 620. On the other hand, in the present embodiment, the posture is estimated by machine learning based on the person area 610 and the height information without correcting the feature points 620 and the like. Since the present embodiment is the same as the first embodiment in other respects, duplicated description will be omitted or simplified.

FIG. 11 is a block diagram showing the function of the control unit 110 of the image recognition device 100. The control unit 110 includes an image acquisition unit 111, a person area detection unit 112, a height information acquisition unit 114, and a posture estimation unit 116.

The posture estimation unit 116 estimates the posture of the subject 500 based on the person area 610 detected from the image 600 and the height information.

The posture estimation unit 116 uses the combination of the person area 610 and the height information and the posture set as the correct answer for the input of the person area 610 and the height information as teacher data, and obtains the posture from the person area 610 and the height information. Learning for estimation is performed in advance. The combination of the person area 610 and the height information used as the teacher data and the posture set as the correct answer for the input of the person area 610 and the height information is photographed by the photographing apparatus 200 installed at a plurality of heights. The one based on the image 600 is used. That is, as teacher data, input of the person area 610 and height information obtained based on the image 600 taken by the image pickup device 200 installed at various heights, and the person area 610 and the height information. Use the combination with the correct answer label of the posture set as the correct answer for the input.

The operation of the image recognition device 100 will be described.

FIG. 12 is a flowchart showing the operation of the image recognition device 100. This flowchart is executed by the control unit 110 according to the program stored in the storage unit 120.

The image acquisition unit 111 acquires an image 600 from the photographing device 200 via the communication unit 150 (S301).

The person area detection unit 112 detects the person area 610 including the target person 500 from the image 600 (S302).

The posture estimation unit 116 estimates the posture of the subject 500 based on the person area 610 and the height information (S303).

This embodiment has the following effects.

A person area is detected from the image of the entire shooting area taken by the shooting device installed at the installation position that overlooks a predetermined shooting area, and based on the height information in which the shooting device is installed and the person area, Estimate the posture of a person by machine learning. As a result, even if the installation height of the photographing device fluctuates, the posture of the person can be estimated with high accuracy based on the captured image.

The configuration of the image recognition system 10 described above is the main configuration described in explaining the features of the above-described embodiment, and is not limited to the above-mentioned configuration and may be variously modified within the scope of the claims. can. Further, it does not exclude the configuration provided in the general image recognition system.

For example, in the above-described embodiment, the reference height and the height of the reference image plane are set separately. However, the reference height and the height of the reference image plane may be the same.

Further, the function of the image recognition device 100 may be provided in the photographing device 200 configured by the sensor box or the mobile terminal 400.

Further, the image recognition device 100, the photographing device 200, and the mobile terminal 400 may each be configured by a plurality of devices, or any plurality of the devices may be configured as a single device.

Further, in the above-mentioned flowchart, some steps may be omitted or other steps may be added. Further, a part of each step may be executed at the same time, or one step may be divided into a plurality of steps and executed.

Further, the means and methods for performing various processes in the image recognition system 10 described above can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versail Disc) -ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. Further, the above program may be provided as a single application software, or may be incorporated into the software of a device such as a detection unit as a function.

10 Image recognition system,
100 image recognition device,
110 Control unit,
111 Image acquisition unit,
112 Person area detector,
113 Point estimation unit,
114 Height information acquisition unit,
115 correction part,
116 Posture estimation unit,
200 shooting equipment,
300 communication networks,
310 access points,
400 mobile terminals,
500 Targets,
600 images,
610 person area,
620 feature points,
621 Cut vertices,
622 Head rectangle paired vertices,
623 Head rectangle.

(1) A procedure (a) for acquiring a person area detected from an image in which the entire shooting area is taken by a shooting device installed at an installation position overlooking a predetermined shooting area, and a predetermined setting position. The teacher teaches the combination of the procedure (b) for acquiring the height information from the position and the posture set as the correct answer to the input of the person area corresponding to the information of the plurality of different heights and the information of the plurality of different heights. A computer is made to execute a process having the procedure (c) of inputting information on the person area and the height into the trained model for estimating the posture of the person trained as data and estimating the posture of the person. Image processing program for .
(2) A person area acquisition unit that acquires a person area detected from an image in which the entire shooting area is shot by a shooting device installed at an installation position that overlooks a predetermined shooting area, and a predetermined setting position of the person area acquisition unit. Teacher data is a combination of an information acquisition unit that acquires height information from a position, and a combination of a plurality of different height information and a posture set as a correct answer to the input of a person area corresponding to the plurality of different height information. Image processing having a trained model for estimating the posture of a person trained as, and a posture estimation unit for inputting information on the person area and the height into the trained model and estimating the posture of the person. system.
(3) A step (a) of acquiring a person area detected from an image in which the entire shooting area is taken by a shooting device installed at an installation position overlooking a predetermined shooting area, and a predetermined setting position. The teacher teaches the combination of the step (b) of acquiring the height information from the position and the posture set as the correct answer to the input of the person area corresponding to the plurality of different height information and the plurality of different height information. An image processing method comprising a step (c) of inputting information on the person area and the height into a trained model for estimating the posture of a person trained as data and estimating the posture of the person.

(1) A procedure (a) for detecting a person area from a captured image by a photographing device, and a procedure (b) for acquiring information on the height of the installation position of the photographing device from a predetermined position. And the combination of the posture set as the correct answer for the input of the person area and the height information is used as the teacher data, and the trained model for estimating the posture of the person from the person area and the height information is described above. It has a procedure (c) of inputting the person area detected in the procedure (a) and the height information acquired in the procedure (b) into the trained model and estimating the posture of the person. An image processing program that allows a computer to perform processing.
(2) A person area detection unit that detects a person area from an image captured by the photographing device, and an information acquisition unit that acquires information on the height of the installation position of the photographing device from a predetermined position. , A trained model that has been trained to estimate the posture of a person from the person area and height information, using the combination with the posture set as the correct answer for the input of the person area and height information as teacher data, and the person. An image processing system including a posture estimation unit that inputs the person area detected by the area detection unit and the height information acquired by the information acquisition unit into the trained model and estimates the posture of the person. ..
(3) A step (a) of detecting a person area from a captured image by a photographing device, and a step (b) of acquiring information on the height of the installation position of the photographing device from a predetermined position. , A trained model that has been trained to estimate the posture of a person from the person area and height information, using the combination with the posture set as the correct answer for the input of the person area and height information as teacher data. An image processing method comprising the step (c) of inputting the information of the person area detected in (a) and the height acquired in the step (b) and estimating the posture of the person.

Claims (9)

An image acquisition unit that acquires an image in which the entire imaging area is captured by an imaging device installed at an installation position that gives a bird's-eye view of a predetermined imaging area.
A person area detection unit that detects a person area from the image,
An information acquisition unit that acquires height information from a predetermined position of the installation position, and
A posture estimation unit that estimates the posture of a person by machine learning based on the person area and the height information, and a posture estimation unit.
Image processing system with. Further, it has a correction unit for correcting the person area based on the height information in accordance with the case where the height from the predetermined position is a preset reference height.
The posture estimation unit is pre-learned as teacher data from the combination of the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person, and the correction unit is used. Estimates the posture of the person based on the person area corrected by
The image processing system according to claim 1. A feature point estimation unit that estimates feature points related to the human body from the person area,
Further, it has a correction unit for correcting the feature points based on the height information in accordance with the case where the height from the predetermined position is a preset reference height.
The posture estimation unit uses a combination of the feature points estimated from the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person as teacher data. The posture of the person is estimated based on the feature points learned in advance and corrected by the correction unit.
The image processing system according to claim 1. The procedure (a) of acquiring an image in which the entire shooting area is shot by a shooting device installed at an installation position overlooking a predetermined shooting area, and
The procedure (b) for detecting a person area from the image and
The procedure (c) for acquiring height information of the installation position from a predetermined position and
The procedure (d) of estimating the posture of a person by machine learning based on the person area and the height information, and
An image processing program for causing a computer to execute a process having the above. The process further includes a procedure (e) for correcting the person area based on the height information in accordance with the case where the height from the predetermined position is a preset reference height.
In the procedure (d), the posture estimation learned in advance using the combination of the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person as teacher data. The unit estimates the posture of the person based on the person area corrected in the procedure (e).
The image processing program according to claim 4. In the process, the procedure (f) for estimating the feature points related to the human body from the person area and the procedure (f).
Further, the procedure (g) for correcting the feature points based on the height information is provided in accordance with the case where the height from the predetermined position is a preset reference height.
In the procedure (d), the combination of the feature points estimated from the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person is used as teacher data. The posture of the person is estimated based on the feature points corrected in the procedure (g) by the posture estimation unit learned in advance.
The image processing program according to claim 4. It is an image processing method by an image processing system.
At the stage (a) of acquiring an image in which the entire shooting area is shot by a shooting device installed at an installation position overlooking a predetermined shooting area,
The step (b) of detecting a person area from the image and
At the stage (c) of acquiring height information of the installation position from a predetermined position,
The step (d) of estimating the posture of a person by machine learning based on the person area and the height information, and
Image processing method having. Further, the step (e) for correcting the person area based on the height information is provided in accordance with the case where the height from the predetermined position is a preset reference height.
In the step (d), the posture estimation learned in advance using the combination of the person area detected from the image taken by the photographing device installed at the reference height and the posture of the person as teacher data. The unit estimates the posture of the person based on the person area corrected in the step (e).
The image processing method according to claim 7. At the stage (f) of estimating the feature points related to the human body from the person area,
Further, it has a step (g) of correcting the feature point based on the information of the height in accordance with the case where the height from the predetermined position is a preset reference height.
In the step (d), teacher data is a combination of the feature points estimated from the person area detected from the image taken by the image pickup device installed at the reference height and the posture of the person. The posture of the person is estimated based on the feature points corrected in the step (g) by the posture estimation unit learned in advance.
The image processing method according to claim 7.

Images