JP7031048B1 - Image processing methods, computer programs and image processing equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method, a computer program and an image processing apparatus which can be expected to perform image processing using a bounding box on a distorted image. An image processing method acquires a distorted image including distortion taken by a camera, generates a three-dimensional frame surrounding an object captured in the distorted image, and uses the distorted image based on the three-dimensional frame. Extract a partial image containing an object. The three-dimensional frame may be a shape in which two planar frames surrounding the object intersect. The two-dimensional coordinates of the object in the distorted image are converted into the three-dimensional coordinates in the virtual three-dimensional space, and two intersecting planar frame frames surrounding the object are generated in the three-dimensional space, and the generated frame 3 Inversely transform the 3D coordinates to 2D coordinates in the distorted image, select one of the two planar frames based on the size in the distorted image, and target from the distorted image based on the selected planar frame. A partial image including an object may be extracted. [Selection diagram] FIG. 10

Description

The present invention relates to an image processing method, a computer program, and an image processing apparatus that perform a process of detecting an object captured in an image taken by a camera.

In recent years, technologies such as machine learning and deep learning have advanced, and it has become possible to detect a specific object, such as a person or a car, from an image taken by a camera. This technology is used for surveillance cameras, in-vehicle cameras, and the like, and it is possible to determine, for example, whether or not a person is within the shooting range of the camera, and what kind of behavior the person is performing. With a surveillance camera or the like, it is desired to be able to shoot as wide a range as possible, and in many cases, shooting is performed using a lens with a wide angle of view (so-called wide-angle lens). An image taken with a wide-angle lens is an image in which a large amount of distortion occurs in the peripheral portion.

Patent Document 1 proposes an image processing system that corrects distortion of an image generated by photographing an object. This image processing system generates first image data by photographing an object illuminated by an illumination unit, a projection unit that projects a predetermined pattern on the object, and an object illuminated by the illumination unit without including the predetermined pattern. The image pickup device includes an image pickup unit that captures an object illuminated by the illumination unit together with a predetermined pattern to generate second image data, and a drive unit that moves the illumination unit and the image pickup unit relative to the object. .. Further, the image processing system detects the distortion amount of the pattern image corresponding to the predetermined pattern in the second image indicated by the second image data, and corrects the distortion of the first image indicated by the first image data based on the detected distortion amount. It is equipped with an image processing device to perform.

Japanese Unexamined Patent Publication No. 2019-192949

When the process of detecting an object from the image taken by the camera is performed, a rectangular frame surrounding the detected object, a so-called bounding box, is superimposed and displayed on the captured image in order to present the detection result to the user. Is done. However, when displaying a bounding box for a distorted image taken by a camera, the size of the bounding box surrounding the object becomes large because the object is distorted, and something other than the object gets inside the bounding box. In some cases, the accuracy of the bounding box deteriorated. Even when the camera shoots through a normal lens, for example, the captured image may be distorted, and particularly when the camera shoots through a wide-angle lens, the image may be greatly distorted.

The present invention has been made in view of such circumstances, and an object thereof is an image processing method and a computer program which can be expected to perform image processing using a bounding box on a distorted image. And to provide an image processing apparatus.

In the image processing method according to one embodiment, a distorted image including distortion taken by a camera is acquired, a three-dimensional frame surrounding an object reflected in the acquired distorted image is generated, and the generated three-dimensional shape is obtained. A partial image including the object is extracted from the distorted image based on the frame, and the three-dimensional frame is a shape in which two planar frames surrounding the object intersect .

In the case of one embodiment, it can be expected that image processing using a bounding box is performed on a distorted image.

It is a schematic diagram for demonstrating the outline of the information processing system which concerns on this embodiment. It is a block diagram which shows the structure of the server apparatus which concerns on this embodiment. It is a block diagram which shows the structure of the terminal apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the bounding box generation process performed by the server apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the bounding box adjustment process performed by the server apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the bounding box adjustment process performed by the server apparatus which concerns on this embodiment. It is a schematic diagram which shows the display example of the image by the terminal apparatus. It is a flowchart which shows the procedure of the process performed by the server apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the bounding box generation process for image extraction performed by the server apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating image extraction based on a bounding box. It is a flowchart which shows the procedure of the image extraction processing performed by the server apparatus which concerns on this embodiment.

A specific example of the information processing system according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

<System configuration>
FIG. 1 is a schematic diagram for explaining an outline of an information processing system according to the present embodiment. In the information processing system according to the present embodiment, for example, a camera 1 installed in a commercial facility or a public facility photographs the surroundings, a server device 3 detects a person from the captured image, and bounding surrounds the detected person. This is a system in which a box is superimposed on a captured image and displayed on the terminal device 5.

The camera 1 according to the present embodiment is a camera that shoots through a wide-angle lens having a wide angle of view, for example, a camera having a horizontal angle of view of about 180 °. The camera 1 shoots at a frequency of several to several tens of times per second, and transmits the image obtained by the shooting to the server device 3. The camera 1 communicates with the server device 3 via, for example, a LAN (Local Area Network), a wireless LAN, a mobile phone communication network, or a wired or wireless network such as the Internet, and transmits a captured image to the server device 3. .. In the present embodiment, the camera 1 shoots through a wide-angle lens, but the present invention is not limited to this, and the camera 1 may shoot through a lens different from the wide-angle lens.

The server device 3 receives a photographed image transmitted from one or a plurality of cameras 1 and performs a process of detecting a person photographed in the received photographed image. The server device 3 generates a bounding box 101 surrounding the person detected from the captured image, and generates an image for display in which the generated bounding box 101 is superimposed on the original captured image. The server device 3 transmits the generated display image to the terminal device 5.

The terminal device 5 is a device used by a user who performs monitoring or the like using the camera 1, and may be configured by using a general-purpose information processing device such as a PC (personal computer), a smartphone, or a tablet-type terminal device. The terminal device 5 receives the image transmitted from the server device 3 and displays the received image on the display unit. The image transmitted from the server device 3 to the terminal device 5 and displayed on the display unit is an image taken by the camera 1, and if a person is present in the image, the bounding box 101 surrounding the person is superimposed. It is an image.

In the present embodiment, the server device 3 performs a process of detecting a person captured in the captured image, a process of generating a bounding box 101 surrounding the detected person, and the like, but the present invention is limited to this. is not it. For example, the camera 1 performs a process of detecting a person, a process of generating a bounding box 101, and the like, and a photographed image on which the bounding box 101 is superimposed is directly transmitted to a terminal via the server device 3 or without the server device 3. It may be transmitted to the device 5. Further, in the present embodiment, the server device 3 transmits the captured image in which the bounding box 101 is superimposed to the terminal device 5, and the terminal device 5 that receives the image is displayed on the display unit, but the present invention is limited to this. is not it. For example, when the server device 3 includes a display unit, a captured image in which the bounding box 101 is superimposed on the display unit of the server device 3 may be displayed. Further, for example, when the camera 1 includes a display unit, the camera 1 may generate and superimpose the bounding box 101 to display a captured image in which the bounding box 101 is superimposed on its own display unit.

<Device configuration>
FIG. 2 is a block diagram showing a configuration of the server device 3 according to the present embodiment. The server device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In the present embodiment, it is assumed that the processing is performed by one server device, but a plurality of server devices may perform the processing in a distributed manner.

The processing unit 31 is configured by using an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory) and the like. Has been done. The processing unit 31 reads and executes the server program 32a stored in the storage unit 32 to acquire an image taken by the camera 1, a process of detecting a person captured in the image, and a process of detecting the detected person. Various processes such as a process of generating a surrounding bounding box and a process of superimposing the bounding box on a captured image and transmitting it to the terminal device 5 are performed.

The storage unit 32 is configured by using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In the present embodiment, the storage unit 32 stores the server program 32a executed by the processing unit 31, and has learned YOLO (You Only Look Once) learning used for the process of detecting a person from the image taken by the camera 1. The model 32b is stored.

In the present embodiment, the server program (program product) 32a is provided in a mode of being recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. Remember. However, the server program 32a may be written to the storage unit 32, for example, at the manufacturing stage of the server device 3. Further, for example, the server program 32a may be acquired by the server device 3 by communication, which is distributed by another remote server device or the like. For example, in the server program 32a, the writing device may read what was recorded on the recording medium 99 and write it in the storage unit 32 of the server device 3. The server program 32a may be provided in the form of distribution via the network, or may be provided in the form recorded on the recording medium 99.

The YOLO learning model 32b is a learning model based on a CNN (Convolutional Neural Network), detects a specific object from an input image, types of the detected object, a bounding box surrounding the detected object, and the like. It is a learning model in which machine learning is performed so as to output the information of. The YOLO learning model 32b according to the present embodiment is machine-learned in advance so as to detect a human head captured in an image and output information on a bounding box surrounding the detected human head. Since the learning model of YOLO is an existing technology, detailed description of its structure, algorithm, etc. will be omitted. Further, in the present embodiment, the human head is detected by using the learning model of YOLO, but the present invention is not limited to this, and for example, R-CNN (Regions with Convolutional Neural Networks), Faster R-CNN or SSD. A learning model such as (Single Shot multibox Detector) may be used. The storage unit 32 stores information on the structure of the learning model constituting the YOLO learning model 32b, information on internal parameters determined by machine learning, and the like.

The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 33 communicates with the camera 1 and the terminal device 5 via the network N. The communication unit 33 transmits the data given from the processing unit 31 to another device, and gives the data received from the other device to the processing unit 31.

The storage unit 32 may be an external storage device connected to the server device 3. Further, the server device 3 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 3 is not limited to the above configuration, and may include, for example, a reading unit for reading information stored in a portable storage medium, an input unit for receiving operation input, a display unit for displaying an image, and the like. ..

Further, in the server device 3 according to the present embodiment, the processing unit 31 reads out and executes the server program 32a stored in the storage unit 32, whereby the image acquisition unit 31a, the person detection unit 31b, the coordinate conversion unit 31c, and the like. The bounding box (abbreviated as BBox in FIG. 2) generation unit 31d, the coordinate inverse conversion unit 31e, the bounding box adjustment unit 31f, the image superimposition unit 31g, the image transmission unit 31h, etc. are combined with the processing unit 31 as software-like functional units. It will be realized. In this figure, as the functional unit of the processing unit 31, the functional unit that performs processing on the image captured by the camera 1 is shown, and the functional unit related to other processing is omitted.

The image acquisition unit 31a acquires an image (photographed image) taken by the camera 1 by receiving the image transmitted by the camera 1 by the communication unit 33. In the present embodiment, the camera 1 shoots through a wide-angle lens, and the captured image acquired by the image acquisition unit 31a is an image (distorted image) having distortion in the peripheral portion. The image acquisition unit 31a temporarily stores the acquired captured image in the storage unit 32. The captured image temporarily stored in the storage unit 32 by the image acquisition unit 31a is erased from the storage unit 32 at an appropriate timing after the bounding box is superimposed and transmitted to the terminal device 5 by the subsequent image processing. It's okay.

The human detection unit 31b performs a process in which the image acquisition unit 31a detects the head of a person captured in the captured image acquired from the camera 1. In the present embodiment, the human detection unit 31b performs the detection process using the YOLO learning model 32b stored in the storage unit 32. The human detection unit 31b inputs the distorted captured image acquired from the camera 1 by the image acquisition unit 31a into the YOLO learning model 32b. The YOLO learning model 32b detects the head of the person captured in the input image, generates a bounding box surrounding the detected head of the person, and outputs the information of the generated bounding box.

The bounding box output by the YOLO learning model 32b in the present embodiment is a rectangular frame that surrounds the detected human body. This frame is represented using, for example, the coordinates of two points on the two-dimensional plane of the captured image. For example, the frame is represented by the coordinates (x1, y1) of the upper left corner portion and the coordinates (x2, y2) of the lower right corner portion, and the YOLO learning model 32b has these (x1, y1) and (x2, y2). Outputs the coordinates of as the bounding box information. When a plurality of people are captured in the captured image, the YOLO learning model 32b generates a bounding box surrounding the head for each person and outputs the coordinates of each bounding box.

The human detection unit 31b acquires the information of the bounding box output by the YOLO learning model 32b in response to the input of the captured image. Further, in the present embodiment, the human detection unit 31b calculates the coordinates of the center of the human head captured in the photographed image based on the information of the bounding box acquired from the YOLO learning model 32b. For example, when the YOLO learning model 32b outputs the coordinates (x1, y1) of the upper left corner portion and the coordinates (x2, y2) of the lower right corner portion of the bounding box, the human detection unit calculates the coordinates of the midpoint of these two points. By doing so, the coordinates of the center of the human head are calculated. The human detection unit 31b gives the calculated coordinates of the center of the human head to the coordinate conversion unit 31c.

The coordinate conversion unit 31c performs a predetermined coordinate conversion process on the coordinates of the center of the human head detected by the person detection unit 31b. The coordinates given from the person detection unit 31b to the coordinate conversion unit 31c are two-dimensional coordinates in the distorted captured image captured by the camera 1. The coordinate conversion unit 31c performs a process of converting the two-dimensional coordinates into three-dimensional coordinates in the three-dimensional virtual space. The details of the coordinate conversion process performed by the coordinate conversion unit 31c will be described later.

The bounding box generation unit 31d is a three-dimensional bounding box that surrounds the person's head based on the three-dimensional coordinates in the three-dimensional virtual space of the center (centroid) of the person's head converted by the coordinate conversion unit 31c. To generate. In the present embodiment, the three-dimensional bounding box generated by the bounding box generation unit 31d is, for example, a rectangular parallelepiped or a cube. Further, the bounding box generation unit 31d according to the present embodiment generates a three-dimensional bounding box that surrounds the person's body (the portion below the neck). That is, the bounding box generation unit 31d generates two three-dimensional bounding boxes that surround the head and the body other than the head, respectively, for the person detected from the captured image. When generating a bounding box that surrounds the body, the bounding box generation unit 31d assumes that the height of the target person is a predetermined value (for example, 160 cm, 170 cm, etc.) and determines the height (length) of the bounding box. decide.

The coordinate inverse conversion unit 31e performs a process of converting the three-dimensional coordinates of the bounding box generated by the bounding box generation unit 31d into two-dimensional coordinates. The conversion process from the three-dimensional coordinates to the two-dimensional coordinates performed by the coordinate inverse conversion unit 31e corresponds to the inverse conversion of the conversion process from the two-dimensional coordinates to the three-dimensional coordinates by the coordinate conversion unit 31c. The bounding box that has been converted by the coordinate inverse conversion unit 31e corresponds to a person who projects a three-dimensional bounding box onto a two-dimensional plane of a captured image taken by the camera 1 to make it two-dimensional. The details of the coordinate conversion process performed by the coordinate inverse conversion unit 31e will be described later.

The bounding box adjusting unit 31f performs a process of adjusting the size of the bounding box based on the conversion result of the three-dimensional bounding box to the two-dimensional coordinates by the coordinate inverse conversion unit 31e. As described above, when the bounding box generation unit 31d generates the bounding box that surrounds the body of the person shown in the captured image, the height of the bounding box is determined on the assumption that the height of the person is a predetermined value. , Actual height may differ from the assumed height. Therefore, the bounding box adjusting unit 31f adjusts the height of the bounding box. In the present embodiment, the bounding box adjusting unit 31f has a bounding box surrounding the head converted into two-dimensional coordinates by the coordinate inverse conversion unit 31e, and a head generated by the human detection unit 31b using the YOLO learning model 32b. Compare the size with the surrounding bounding box, and adjust the height of the surrounding bounding box based on the comparison result.

The image superimposing unit 31g performs image processing for superimposing the final bounding box whose height has been adjusted by the bounding box adjusting unit 31f on the captured image of the camera 1 acquired by the image acquiring unit 31a. The image superimposing unit 31g gives an image in which a bounding box is superimposed on the captured image to the image transmitting unit 31h.

The image transmitting unit 31h causes the terminal device 5 to display this image by performing a process of transmitting the image given from the image superimposing unit 31g to the predetermined terminal device 5 by the communication unit 33. In the present embodiment, the server device 3 performs image processing for superimposing the bounding box on the captured image, but the present invention is not limited to this. For example, the server device 3 uses the captured image and the bounding box information as a terminal device. It may be transmitted to 5 and the terminal device 5 may superimpose the bounding box on the captured image and display it.

FIG. 3 is a block diagram showing the configuration of the terminal device 5 according to the present embodiment. The terminal device 5 according to the present embodiment includes a processing unit 51, a storage unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, an operation unit 55, and the like. The terminal device 5 is a device used by a user who performs a business such as monitoring based on an image taken by a camera 1, and may be configured by using an information processing device such as a personal computer, a smartphone, or a tablet terminal device, for example. ..

The processing unit 51 is configured by using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. By reading and executing the program 52a stored in the storage unit 52, the processing unit 51 performs various processes such as receiving the captured image of the camera 1 transmitted from the server device 3 and displaying it on the display unit. ..

The storage unit 52 is configured by using a storage device such as a hard disk or a flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data required for processing by the processing unit 51. In the present embodiment, the storage unit 52 stores the program 52a executed by the processing unit 51. In the present embodiment, the program 52a is distributed by a remote server device or the like, which is acquired by the terminal device 5 by communication and stored in the storage unit 52. However, the program 52a may be written in the storage unit 52, for example, at the manufacturing stage of the terminal device 5. For example, in the program 52a, the terminal device 5 may read the program 52a recorded on the recording medium 98 such as a memory card or an optical disk and store it in the storage unit 52. For example, in the program 52a, the writing device may read out what was recorded on the recording medium 98 and write it in the storage unit 52 of the terminal device 5. The program 52a may be provided in a mode of distribution via a network, or may be provided in a mode recorded on a recording medium 98.

The communication unit 53 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 53 communicates with the server device 3 via the network N. The communication unit 53 transmits the data given from the processing unit 51 to another device, and gives the data received from the other device to the processing unit 51.

The display unit 54 is configured by using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing of the processing unit 51. The operation unit 55 accepts the user's operation and notifies the processing unit 51 of the accepted operation. For example, the operation unit 55 accepts a user's operation by an input device such as a mechanical button or a touch panel provided on the surface of the display unit 54. Further, for example, the operation unit 55 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be removable with respect to the terminal device 5.

Further, in the terminal device 5 according to the present embodiment, the processing unit 51 reads out and executes the program 52a stored in the storage unit 52, so that the image receiving unit 51a, the display processing unit 51b, and the like serve as software-like functional units. It is realized in the processing unit 51. The program 52a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or a web browser.

The image receiving unit 51a performs a process of receiving the image transmitted by the server device 3 by the communication unit 53. The image receiving unit 51a temporarily stores the received image in, for example, a storage unit 52. In the present embodiment, the image received by the image receiving unit 51a from the server device 3 is an image in which the bounding box is superimposed on the captured image of the camera 1.

The display processing unit 51b performs processing for displaying various characters, images, and the like on the display unit 54. In the present embodiment, the display processing unit 51b displays on the display unit 54 an image received by the image receiving unit 51a from the server device 3 and an image in which the bounding box is superimposed on the captured image of the camera 1. In the present embodiment, the camera 1 repeatedly shoots at a frequency of several tens of times per second, that is, shoots a moving image. The server device 3 superimposes a bounding box on each captured image captured by the camera 1 at a frequency of several tens of times per second and transmits the bounding box to the terminal device 5, and the display processing unit 51b of the terminal device 5 is the server device 3. By displaying the image received from the server at a frequency of several tens of times per second, a moving image on which the bounding box is superimposed is displayed.

<Bounding box generation process>
FIG. 4 is a schematic diagram for explaining the bounding box generation process performed by the server device 3 according to the present embodiment. In this figure, the quadrangle with reference numeral 110 indicates a photographed image taken by the camera 1. The camera 1 according to the present embodiment shoots through a wide-angle lens, and the captured image 110 is an image (distorted image) including distortion due to the lens. The server device 3 that has acquired the captured image 110 from the camera 1 uses the YOLO learning model 32b to perform a process of detecting the head of a person included in the captured image 110. By performing the detection process using the YOLO learning model 32b, the server device 3 can acquire information such as the coordinates of the bounding box 111 that surrounds the human head included in the captured image 110 as the detection result. The server device 3 calculates the coordinates (two-dimensional coordinates) of the center point 112 (center point of the bounding box) of the human head based on the information such as the coordinates of the bounding box 111 acquired from the YOLO learning model 32b.

Next, the server device 3 converts the coordinates of the center point 112 of the human head included in the captured image 110 into the two-dimensional coordinates of the corresponding points 113 in the two-dimensional plane obtained by removing the distortion from the captured image 110. Since the image processing for correcting an image including distortion due to the lens is a conventional technique, detailed description thereof will be omitted. However, for example, based on the following approximate expression using a lens distortion model, a point (xd, yd) and distortion-free image points (x, y) can be converted to each other.

The server device 3 sets the two-dimensional coordinates of the point 112 corresponding to the center of the human head included in the captured image 110 including the distortion as a point in the two-dimensional plane from which the distortion is removed based on the above equation (1). Convert to 113 two-dimensional coordinates. In the present embodiment, the server device 3 does not need to perform coordinate conversion on all the points included in the captured image 110, and only needs to convert the coordinates of the point 112 corresponding to the center of the human head.

Next, the server device 3 performs a process of converting the two-dimensional coordinates of the point 113 corresponding to the center of the human head into the three-dimensional coordinates in the three-dimensional virtual space. The conversion from 2D coordinates to 3D coordinates performed at this time generally corresponds to the reverse conversion process of the conversion process performed when a virtual camera is set in the 3D virtual space and a 2D image is generated. It is a thing. Therefore, first, the conversion process from the three-dimensional coordinates to the two-dimensional coordinates will be described, and then the conversion process from the two-dimensional coordinates to the three-dimensional coordinates to which this conversion process is applied in the opposite direction will be described.

There are two types of three-dimensional coordinates, the world coordinate system and the camera coordinate system. The world coordinate system defines the coordinates of XYZ with a specific position in the three-dimensional virtual space as the origin. The camera coordinate system defines the coordinates of XYZ with the camera arranged in the three-dimensional virtual space as the origin. Assuming that the coordinates of the world coordinate system are (Xw, Yw, Zw) and the coordinates of the camera coordinate system are (Xc, Yc, Zc), the conversion from the world coordinate system to the camera coordinate system is an external variable (extrinsic parameters) of the camera. ) Can be used by the following equation (2).

Further, assuming that the coordinates of the points in the two-dimensional plane are (u, v), the conversion from the camera coordinate system to the two-dimensional coordinates uses a constant called the internal variables (intrinsic parameters) of the camera and uses the following (3). It can be done by an expression.

In the above equations (2) and (3), the internal variables and external variables of the camera are constants determined based on the characteristics of the camera, the position of the camera, and the like. Further, the constant c in the equation (3) is a numerical value representing a scale or a ratio or the like, and is determined according to the system configuration or the like. Based on the above equations (2) and (3), the conversion from the three-dimensional coordinates of the world coordinate system to the two-dimensional coordinates of the two-dimensional plane can be performed by the following equation (4).

By substituting any point (Xw, Yw, Zw) in the world coordinate system into the above equation (4), the corresponding two-dimensional coordinates (u, v) can be uniquely specified. It is expected that the two-dimensional coordinates will be converted to the three-dimensional coordinates by performing the inverse calculation of the equation (4), that is, the calculation of the product of the inverse matrix of the matrix of the internal and external variables of the camera and the two-dimensional coordinates. However, since the number of dimensions increases in the conversion from two-dimensional coordinates to three-dimensional coordinates, there is insufficient information to uniquely identify the three-dimensional coordinates. Therefore, in the information processing system according to the present embodiment, it is assumed that the height of the person detected from the captured image 110 is a predetermined value (for example, 175 cm). That is, the server device 3 according to the present embodiment assumes that the value of Zw is a predetermined value in the equation (4), and calculates the values of Xw and Yw of the three-dimensional coordinates from the two-dimensional coordinates (u, v). Then, the two-dimensional coordinates are converted into the three-dimensional coordinates.

The server device 3 converts the two-dimensional coordinates of the point 113 on the two-dimensional plane from which the distortion has been removed into the three-dimensional coordinates of the corresponding point 114 in the world coordinate system based on the above equation (4). In FIG. 4, the conversion from the two-dimensional coordinates to the three-dimensional coordinates of the camera coordinate system and the conversion from the three-dimensional coordinates of the camera coordinate system to the three-dimensional coordinates of the world coordinate system are shown separately. The server device 3 may collectively perform these two steps of coordinate conversion.

Next, the server device 3 has a three-dimensional bounding box 115 that surrounds the human head detected from the captured image 110 and a three-dimensional bounding box 116 that surrounds the body below the head in the three-dimensional space of the world coordinate system. Performs the process of generating and. First, the server device 3 regards the point 114 obtained by the above coordinate conversion as the center (center of gravity) of the human head, and generates a rectangular parallelepiped frame of a predetermined size around the point 114 to generate the head. A three-dimensional bounding box 115 that surrounds the portion is generated. In this example, the three-dimensional bounding box 115 that surrounds the head is a cube of 20 cm × 20 cm × 20 cm.

The server device 3 generates a three-dimensional bounding box 116 that surrounds the body by generating a rectangular parallelepiped frame of a predetermined size below the three-dimensional bounding box 115 that surrounds the head. In this example, the height of a person is 175 cm, a gap of 5 cm is provided between the person and the three-dimensional bounding box 115 that surrounds the head, and the three-dimensional bounding box 116 that surrounds the body is a rectangular parallelepiped type of 50 cm × 50 cm × 150 cm. It is supposed to be.

Next, the server device 3 converts the three-dimensional bounding boxes 115 and 116 generated in the three-dimensional space of the world coordinate system into the three-dimensional bounding boxes 117 and 118 of the three-dimensional space of the camera coordinate system, and further converts the two-dimensional plane. Converted to the bounding boxes 119 and 120 in. At this time, the server device 3 may directly convert to the three-dimensional bounding boxes 115 and 116 of the world coordinate system and the bounding boxes 119 and 120 in the two-dimensional plane by using the above equation (4). The server device 3 performs the coordinate conversion of the equation (4) on the plurality of vertices included in the three-dimensional bounding boxes 115 and 116, and connects the plurality of vertices on the two-dimensional plane after the coordinate conversion to the bounding box 119. And 120 may be generated, and the bounding boxes 119 and 120 may be generated by performing coordinate conversion on all points such as vertices and borders constituting the three-dimensional bounding boxes 115 and 116.

The server device 3 then performs a process of converting the distortion-free two-dimensional plane bounding boxes 119 and 120 into the distortion-free bounding boxes 121 and 122 on the image including the distortion of the wide-angle lens. At this time, the server device 3 converts the coordinates of the bounding box based on the above equation (1). The server device 3 performs the coordinate conversion of the equation (1) on the plurality of vertices included in the bounding boxes 119 and 120 on the distortion-free two-dimensional plane, and the plurality of vertices on the distorted image after the coordinate conversion. The bounding boxes 121 and 122 may be generated by connecting the above, or the bounding boxes 121 and 122 may be generated by performing coordinate conversion on all points such as vertices and borders constituting the bounding boxes 119 and 120. In the example shown in FIG. 4, coordinate transformation is performed for all the points of the bounding box, whereby the bounding boxes 121 and 122 including the distorted lines are obtained.

The bounding boxes 121 and 122 generated by the above processing are shapes in which three-dimensional bounding boxes 115 and 116 having a three-dimensional shape such as a cube and a rectangular parallelepiped are projected onto a two-dimensional plane. The server device 3 may shape the three-dimensional bounding box such as a cube or a rectangular parallelepiped into a planar bounding box such as a square or a rectangle.

<Bounding box adjustment process>
The bounding box generated by the above-mentioned bounding box generation process is generated assuming that a person detected from the captured image is standing and the height is a predetermined value (for example, 175 cm). It is thought that the difference in height between people is within a few tens of centimeters, but when a person is sitting, the height from the ground to the head may be 50 cm or more lower than the height, and the person sitting. The height of the generated bounding box may be inappropriate. Therefore, the server device 3 according to the present embodiment performs a process of adjusting the height of the bounding box (bounding box surrounding the body).

5 and 6 are schematic views for explaining the bounding box adjustment process performed by the server device 3 according to the present embodiment. The server device 3 has the size (second size) of the bounding box 111 given by the YOLO learning model 32b to the human head captured in the captured image 110 captured by the camera 1, and the human head in the world coordinate system. The size (first size) of the bounding box 121 generated based on the three-dimensional bounding box 115 surrounding the portion is compared, and the height of the bounding box 122 surrounding the human body is adjusted based on the comparison result.

In the example shown in FIG. 5, the camera 1 is installed at a position higher than the person to be photographed, and the camera 1 takes a bird's-eye view of the surroundings. The captured image 110 obtained by photographing the camera 1 can be regarded as an image projected on a plane perpendicular to the optical axis of the camera 1, and is indicated by a straight line with a reference numeral 110 in the drawing. In the illustrated situation, the head of a person standing at a position A near the camera 1 and the head of a person sitting at a position B far from the camera 1 are captured at the same position in the captured image 110. Become. However, the head of the person at the position A near the camera 1 is larger than the head of the person at the position B far from the camera 1 in the captured image 110.

In the information processing system according to the present embodiment, it is assumed that the person detected from the captured image 110 when the bounding box is generated stands at a predetermined height. Therefore, the bounding box generated for the person captured in the photographed image 110 is suitable for the person standing at the position A in FIG. 5, but the person actually sitting at the position B is the photographed image. It may have been copied to 110.

When the person photographed in the photographed image 110 is a person sitting in the position B, as shown in the upper part of FIG. 6, the bounding box 111 surrounding the head of the person detected by the YOLO learning model 32b from the photographed image 110 is formed. It is smaller than the bounding box 121 generated based on the three-dimensional bounding box 115. On the other hand, when the person captured in the captured image 110 is a person standing at position A, as shown in the lower part of FIG. 6, it surrounds the head of the person detected by the YOLO learning model 32b from the captured image 110. The bounding box 111 and the bounding box 121 generated based on the three-dimensional bounding box 115 have substantially the same size.

The server device 3 according to the present embodiment generates a two-dimensional bounding box 121 superimposed on the captured image 110 based on the three-dimensional bounding box 115, and then determines the size (first size) of the bounding box 121. The size (second size) of the bounding box 111 surrounding the human head detected by the YOLO learning model 32b is compared. The size comparison can be made, for example, by comparing the area, the length of the longest side, the length of the diagonal line, and the like. Since the bounding box 121 has a three-dimensional shape, the server device 3 may convert it into a plane-shaped (rectangular or square) bounding box and compare the sizes. The server device 3 can convert the three-dimensional bounding box 121 into a planar shape, for example, by generating the smallest square or rectangular frame containing the bounding box 121. When the sizes of the two bounding boxes 111 and 121 are substantially the same, the server device 3 determines that the generated bounding boxes 121 and 122 are appropriate, and adopts the generated bounding boxes 121 and 122.

If the size of the bounding box 111 of the YOLO learning model 32b is smaller than the size of the bounding box 121 generated based on the three-dimensional bounding box 115, the server device 3 is inappropriate for the generated bounding boxes 121 and 122. Judging that, adjust the bounding box. The server device 3 sets the height (height from the ground to the head) of the person captured in the captured image 110 from a predetermined height (for example, 170 cm) of the standing person to the head of the sitting person from the ground. By changing the height to a predetermined height (for example, 100 cm) and performing the above-mentioned binding box generation process, the bounding boxes 121 and 122 on the captured image 110 suitable for a sitting person are generated.

By changing the estimated value of the height of the head, the values of Xw and Yw of the three-dimensional coordinates in the world coordinate system calculated by the equation (4) also change. Therefore, the process of adjusting the bounding box described above is also a process of adjusting the position of a person in the world coordinate system. Therefore, the server device 3 increases or decreases the estimated value of the height of the head of the person captured in the captured image 110, for example, the size of the bounding box 121 generated from the three-dimensional bounding box 115, and the YOLO learning model 32b. By searching for the height of the head when the size of the bounding box 111 of the above matches (the difference is less than or equal to the threshold value), the height of the person (height from the ground to the head) in the world coordinate system is accurate. It can be estimated well, and the position of a person in the world coordinate system can be estimated accurately.

In addition to the method of searching for values that match the sizes of the two bounding boxes 121 and 111 by repeatedly increasing and decreasing the height estimates, the height and position of a person can be estimated by learning by machine learning, for example. A method using a model can be adopted. The learning model is preliminarily machine-learned to accept, for example, an image with the size of two bounding boxes 121, 111 or an image with two bounding boxes 121, 111 as input and output the height of the target person. It can be made.

In the present embodiment, the calculation is performed using numerical values such as 170 cm or 100 cm for the height from the ground to the head, but these numerical values are examples and are not limited thereto. These numerical values may be determined in advance by the designer of the information processing system according to the present embodiment and stored in the storage unit 32 of the server device 3 as default values for processing together with the server program 32a. Further, these numerical values may be input from the user when the server device 3 performs processing. For example, when the user inputs a numerical value, the server device 3 can perform processing using this numerical value, and when the numerical value is not input, the server device 3 can perform processing using a default value.

The server device 3 that has completed the adjustment of the bounding boxes 121 and 122 generates an image in which the obtained bounding boxes 121 and 122 are superimposed on the captured image 110 by the camera 1. The server device 3 transmits the generated image to a predetermined terminal device 5, and causes the display unit 54 of the terminal device 5 to display this image. As a result, the user who uses the terminal device 5 can confirm the image of the person photographed on the captured image 110 of the camera 1 surrounded by the bounding boxes 121 and 122 on the terminal device 5.

FIG. 7 is a schematic diagram showing an example of displaying an image by the terminal device 5. In the illustrated display example, there are two people in the room photographed by the camera 1, and the bounding box surrounding the head and the bounding box surrounding the body are displayed superimposed on each person. ing. The camera 1 repeatedly takes pictures at a frequency of, for example, several tens of times per second, and the image displayed by the terminal device 5 is also updated at the same frequency. That is, in the present embodiment, the camera 1 captures a moving image, and the terminal device 5 displays the moving image. The bounding box that surrounds each person moves following the movement of the person when the person moves in the image.

Further, in the illustrated example, two points indicating the positional relationship between the two detected persons are shown in the square area provided at the lower right of the screen displayed on the display unit 54 of the terminal device 5. In this example, the server device 3 estimates the position of a person based on the size comparison of the above-mentioned bounding box, and the coordinates (Xw) from the three-dimensional coordinates in the world coordinate system of each person obtained as a result of the estimation to the two-dimensional coordinates of XY. , Yw). The server device 3 plots the points corresponding to each person in a square area based on the two-dimensional coordinates (Xw, Yw) of each person, so that the user can estimate the position of the person detected from the captured image. Can be presented.

<Flow chart>
FIG. 8 is a flowchart showing a procedure of processing performed by the server device 3 according to the present embodiment. The image acquisition unit 31a of the processing unit 31 of the server device 3 according to the present embodiment communicates with the camera 1 by the communication unit 33, and acquires the captured image taken by the camera 1 (step S1). The person detection unit 31b of the processing unit 31 detects a person copied in the captured image acquired in step S1 by using the YOLO learning model 32b stored in the storage unit 32 (step S2). The human detection unit 31b calculates the coordinates of the center point of the human head based on the information such as the position of the bounding box surrounding the human head output by the YOLO learning model 32b (step S3).

The coordinate conversion unit 31c of the processing unit 31 removes the distortion by performing a calculation for removing the distortion of the center point of the head in the captured image calculated in step S3 using the above equation (1). It is converted into the two-dimensional coordinates in the two-dimensional plane (step S4). The coordinate conversion unit 31c assumes that the detected person is standing and the height is a predetermined value (for example, 170 cm), and based on the above equation (4), the two-dimensional coordinates converted in step S4 are world-wide. Convert to three-dimensional coordinates of the coordinate system (step S5).

The bounding box generation unit 31d of the processing unit 31 is a cubic frame of a predetermined size (for example, 20 cm × 20 cm × 20 cm) centered on the three-dimensional coordinates converted in step S5 in the three-dimensional virtual space of the world coordinate system. By generating the above, a three-dimensional bounding box surrounding the detected person's head is generated (step S6). The bounding box generation unit 31d generates a rectangular parallelepiped frame of a predetermined size (for example, 50 cm × 50 cm × 150 cm) below the bounding box that surrounds the head generated in step S6. A bounding box that surrounds the body (the part below the head) is generated (step S7).

The coordinate inverse conversion unit 31e of the processing unit 31 converts the two generated bounding boxes into a bounding box on a two-dimensional plane based on the above equation (4) (step S8). The coordinate inverse conversion unit 31e performs a calculation of applying distortion to the bounding box on the distortion-free two-dimensional plane converted in step S8 using the above equation (1), thereby causing distortion in two dimensions. It is converted into a bounding box on a plane (captured image) (step S9).

In the bounding box adjusting unit 31f of the processing unit 31, the size of the bounding box surrounding the head of the detected person among the bounding boxes converted in step S9 is the head of the person output by the YOLO learning model 32b in step S2. It is determined whether or not the size of the bounding box surrounding the portion is substantially the same (the difference is within the threshold value) (step S10). When the sizes of the two bounding boxes are different (the difference exceeds the threshold value) (S10: NO), the bounding box adjusting unit 31f sets the height of the detected person, that is, the height from the ground to the head to a predetermined value (for example, 170 cm). To another value (step S11), and the process returns to step S4.

If it is possible to determine whether the detected person is standing or sitting and the height accuracy is sufficient in two ways, standing and sitting, the bounding box is adjusted in step S11. The portion 31f is changed to a predetermined value (for example, 100 cm) as the height of the head when a person is sitting. In this case, the processing unit 31 may change the height and perform the processing in steps S4 to S9, and then proceed to the processing in step S12 without performing the determination in step S10.

On the other hand, when estimating the height and coordinates of the detected person in more detail, the bounding box adjusting unit 31f increases or decreases the estimated value of the person's height by a predetermined value (for example, 1 cm) in step S11. Then, the processes of steps S4 to S11 are repeated until the sizes of both bounding boxes become substantially the same according to the determination in step S10. The bounding box adjusting unit 31f reduces the estimated value of the height of a person when, for example, the size of the bounding box converted in step S9 is larger than the size of the bounding box by the YOLO learning model 32b. On the other hand, when the size of the bounding box converted in step S9 is larger than the size of the bounding box by the YOLO learning model 32b, the bounding box adjusting unit 31f increases the estimated value of the height of the person.

When it is determined that the sizes of both bounding boxes are substantially the same (S10: YES), the image superimposing unit 31g of the processing unit 31 superimposes the bounding box converted in step S9 on the captured image acquired in step S1. The resulting image is generated (step S12). The image transmission unit 31h of the processing unit 31 transmits the image generated in step S12 to the predetermined terminal device 5 by the communication unit 33 (step S13), and ends the processing.

<Image extraction>
In the information processing system according to the present embodiment, the bounding box generated by the server device 3 from the captured image of the camera 1 is used for various processes other than the process of superimposing the bounding box on the captured image and displaying it on the terminal device 5. Can be expected. For example, the server device 3 can perform a process of extracting an image area including a person from a captured image based on the generated bounding box.

In the above-mentioned bounding box generation process, the server device 3 generated two, a bounding box surrounding the head of the person captured in the photographed image and a bounding box surrounding the body. When extracting an image, the server device 3 according to the present embodiment generates one bounding box that surrounds the whole body of a person instead of the above two bounding boxes. First, the process of generating the bounding box for image extraction will be described.

FIG. 9 is a schematic diagram for explaining a bounding box generation process for image extraction performed by the server device 3 according to the present embodiment. The server device 3 detects the human head using the YOLO learning model 32b for the captured image 110 acquired from the camera 1. The server device 3 sets the coordinates of the human head in the distorted photographed image as the two-dimensional coordinates on the two-dimensional plane from which the distortion is removed, the three-dimensional coordinates in the three-dimensional space of the camera coordinate system, and the three-dimensional coordinates of the world coordinate system. Converts to 3D coordinates in space in order. Since the coordinate conversion process up to this point is the same as the process described with reference to FIG. 4, detailed description thereof will be omitted.

The server device 3 generates two rectangular planar bounding boxes 141 and 142 so as to surround the point 114 corresponding to the head obtained by the above coordinate conversion process in the three-dimensional space of the world coordinate system. .. The first bounding box 141 is, for example, a rectangular frame of 50 cm in the x direction, 0 cm in the y direction, and 175 cm in the z direction in three-dimensional coordinates, and is located at the center position in the x direction and 10 cm from the top in the z direction. It is arranged in a three-dimensional space so as to include the point 114. The second bounding box 142 is, for example, a rectangular frame of 0 cm in the x direction, 50 cm in the y direction, and 175 cm in the z direction of the three-dimensional coordinates, and is located at the center position in the y direction and 10 cm from the top in the z direction. It is arranged in a three-dimensional space so as to include the point 114. Due to this arrangement, the two bounding boxes 141 and 142 intersect (orthogonally).

The shapes and sizes of the above two bounding boxes 141 and 142 are examples, and the present invention is not limited to these. The shape of the bounding boxes 141 and 142 does not have to be a quadrangle, and may be, for example, a triangle or a polygon having a pentagon or more. In addition, numerical values such as 50 cm and 175 cm for the bounding boxes 141 and 142 can be changed as appropriate. These numerical values may be determined in advance by the designer of the information processing system according to the present embodiment and stored in the storage unit 32 of the server device 3 as default values for processing together with the server program 32a. Further, these numerical values may be input from the user when the server device 3 performs processing. For example, when the user inputs a numerical value, the server device 3 can perform processing using this numerical value, and when the numerical value is not input, the server device 3 can perform processing using a default value.

Further, when the server device 3 performs the process of estimating the height of a person (height from the ground to the head) by the method shown in FIGS. 5 and 6, the heights of the two bounding boxes 141 and 142 are high. (Length in the z direction) can be the estimated height.

The server device 3 converts the bounding boxes 141 and 142 generated in the three-dimensional space of the world coordinate system into the three-dimensional bounding boxes 143 and 144 in the three-dimensional space of the camera coordinate system, and further converts the bounding boxes 145 in the two-dimensional plane. , 146. At this time, the server device 3 may directly convert the bounding boxes 141 and 142 of the world coordinate system into the bounding boxes 145 and 146 in the two-dimensional plane by using the above equation (4).

The server device 3 converts the distortion-free two-dimensional plane bounding boxes 145 and 146 into a distortion-free bounding box on the image of the wide-angle lens. At this time, the server device 3 can convert the coordinates of the bounding box based on the above equation (1). Next, the server device 3 selects one of the two bounding boxes obtained by the coordinate transformation, and sets the selected one as the bounding box 147 for image extraction. The server device 3 can compare, for example, the width or area of two bounding boxes on an image containing distortion, and select a bounding box having a large width or area.

The server device 3 uses the bounding box 147 for image extraction generated in the above procedure to extract an image area including a person captured in this image from a captured image including distortion acquired from the camera 1. conduct. Note that the server device 3 does not generate the bounding box 147 for image extraction by the procedure shown in FIG. 9, but is used for image extraction based on the two bounding boxes 121 and 122 generated by the procedure shown in FIG. You may generate a bounding box for. In this case, the server device 3 may integrate the bounding box 121 surrounding the human head and the bounding box 122 surrounding the body to generate one two-dimensional (rectangular) bounding box for image extraction. can. The server device 3 can generate a bounding box for image extraction, for example, by generating a minimum rectangular frame including a bounding box 121 surrounding the head and a bounding box 122 surrounding the body.

FIG. 10 is a schematic diagram for explaining image extraction based on a bounding box. In the upper part of FIG. 10, a magnified image of a person and its surroundings taken in a photographed image taken by the camera 1 is shown. Since the camera 1 took a picture through a wide-angle lens, in the illustrated image, the person is not upright with respect to the ground, and the person is shown in the image in a state of being tilted (distorted) at an angle. In this image, the bounding box 131 shown by a solid line rectangle and the bounding box 132 shown by a broken line rectangle are shown superimposed. The solid line bounding box 131 is an image extraction bounding box 131 generated by the server device 3 according to the present embodiment according to the procedure shown in FIG. The broken line bounding box 132 is a bounding box 132 when the entire person is detected by the method of YOLO and a bounding box is attached.

The server device 3 can extract an image area in which a person is captured from a distorted photographed image by extracting the image in the bounding box 131 generated for the photographed image. The image extracted based on the bounding box 131 is shown in the lower left of FIG. The server device 3 performs a process of correcting the inclination (for example, an image rotation process) of the image extracted from the bounding box 131 that is inclined with respect to the captured image. In the lower left of FIG. 10, an image after tilt correction with respect to the extracted image is shown. The image extracted based on the bounding box 131 generated by the server device 3 according to the present embodiment is an image in which a detected person stands along the vertical and horizontal directions (vertical direction and horizontal direction) of the image. The process of correcting the inclination of the extracted image may be performed by a method other than the rotation of the image.

On the other hand, the YOLO bounding box 132 is a rectangular bounding box along the vertical and horizontal directions of a distorted captured image. An image when an image area is extracted from the captured image based on the YOLO bounding box 132 is shown in the lower right of FIG. The image extracted based on the YOLO bounding box 132 is such that the detected person is tilted (distorted) in the image. Also, the extracted image has a large number of pixels belonging to something other than humans (such as the background) (compared to the image extracted based on the bounding box 131) and contains a lot of information about things other than humans. It becomes an image.

The server device 3 uses the image extracted based on the bounding box 131, for example, a process of identifying who is the person copied in this image, a process of recognizing the behavior of the person copied in this image, or a process of recognizing the behavior of the person copied in this image. It can be used as input information for various processes such as a process of tracking a person captured in an image. The image extracted based on the bounding box 131 generated by the server device 3 according to the present embodiment is more detected by a person for the total number of pixels than the image extracted based on the bounding box 132 of YOLO. It can be expected that the ratio of the number of pixels to be occupied will be high, and the accuracy of subsequent processing can be expected to be further improved.

FIG. 11 is a flowchart showing a procedure of image extraction processing performed by the server device 3 according to the present embodiment. The processing unit 31 of the server device 3 according to the present embodiment acquires a photographed image taken by the camera 1 in the image acquisition unit 31a (step S31), and uses the YOLO learning model 32b in the person detection unit 31b. Detection is performed (step S32). The human detection unit 31b calculates the coordinates of the center point of the human head based on the information such as the position of the bounding box surrounding the human head output by the YOLO learning model 32b (step S33). The coordinate conversion unit 31c of the processing unit 31 converts the calculated coordinates of the center point of the head into two-dimensional coordinates in the two-dimensional plane from which distortion has been removed (step S34), and the two-dimensional coordinates are converted into 3 in the world coordinate system. Convert to dimensional coordinates (step S35).

The bounding box generation unit 31d of the processing unit 31 generates two rectangular planar bounding boxes including the three-dimensional coordinates of the center point of the head converted in step S35 in the three-dimensional virtual space of the world coordinate system. (Step S36). The two bounding boxes can be, for example, a rectangular planar frame of 50 cm × 0 cm × 175 cm and a rectangular planar frame of 0 cm × 50 cm × 175 cm.

The coordinate inverse conversion unit 31e of the processing unit 31 converts the two generated bounding boxes into a bounding box on a two-dimensional plane (step S37). The coordinate inverse conversion unit 31e converts the two bounding boxes on the distorted two-dimensional plane converted in step S37 into two bounding boxes on the distorted two-dimensional plane (captured image) (step S38). ).

The processing unit 31 selects one of the two bounding boxes on the captured image generated in step S38, which has a larger width or area, for image extraction, and uses the bounding box for image extraction from the captured image. The enclosed image area is extracted (step S39). The processing unit 31 corrects the inclination (step S40) by performing a process such as rotation on the extracted image area, and ends the image extraction process.

<Summary>
In the information processing system according to the present embodiment having the above configuration, the server device 3 acquires a captured image (distorted image) including distortion captured by the camera 1 through a wide-angle lens, and the object is copied to the acquired captured image. Detect (person). The server device 3 converts the two-dimensional coordinates of the detected person into three-dimensional coordinates in a virtual three-dimensional space, and generates a three-dimensional frame (three-dimensional bounding box) surrounding the object in the three-dimensional space. The server device 3 reversely converts the three-dimensional coordinates of the generated three-dimensional frame into the two-dimensional coordinates in the captured image captured by the camera 1. The server device 3 superimposes the flat frame (flat bounding box) obtained by these on the distorted captured image taken by the camera 1 and displays it on the terminal device 5 or the like. As a result, the server device 3 can be expected to superimpose a bounding box suitable for a distorted captured image and present the detection result of the object from the captured image to the user.

Further, the server device 3 according to the present embodiment converts the two-dimensional coordinates of the object in the distorted captured image captured by the camera 1 into the two-dimensional coordinates in the distorted image. The server device 3 converts the two-dimensional coordinates in the distorted image into the three-dimensional coordinates of the camera coordinate system centered on the camera 1, and converts the three-dimensional coordinates of the camera coordinate system into the three-dimensional coordinates of the world coordinate system. To generate a three-dimensional frame that surrounds the object in the world coordinate system. As a result, the server device 3 can be expected to accurately generate a three-dimensional frame surrounding the object captured in the captured image from the distorted captured image captured through the wide-angle lens. In the present embodiment, the server device 3 converts the two-dimensional coordinates to the three-dimensional coordinates of the camera coordinate system and the three-dimensional coordinates of the camera coordinate system to the three-dimensional coordinates of the world coordinate system (4). Although it is performed collectively based on the formula, it is not limited to this, and each coordinate conversion may be performed individually.

Further, the server device 3 according to the present embodiment detects the human head captured in the captured image including distortion, converts the two-dimensional coordinates of the human head into three-dimensional coordinates, and in the three-dimensional virtual space. A first three-dimensional frame surrounding the human head is generated, a second three-dimensional frame surrounding the human body is generated, and the first three-dimensional frame and the second three-dimensional frame are combined to form a three-dimensional frame surrounding the person. Generate. This can be expected to reduce the processing load of the coordinate conversion performed by the server device 3.

Further, the server device 3 according to the present embodiment generates a first three-dimensional frame of a predetermined size surrounding the human head, generates a second three-dimensional frame of a predetermined size surrounding the human body, and first. A three-dimensional frame and a second three-dimensional frame are combined to generate a three-dimensional frame that surrounds a person, and the three-dimensional coordinates of this three-dimensional frame are converted into two-dimensional coordinates in the two-dimensional plane of the captured image. The server device 3 calculates the size of the part corresponding to the first three-dimensional frame in the plane frame converted into two-dimensional coordinates, and calculates the size of the plane frame based on the detection result of the human head from the captured image including distortion. Comparison with the size is performed, and the height of the second three-dimensional frame is adjusted based on the comparison result. As a result, the server device 3 can be expected to adjust the second three-dimensional frame generated as a predetermined size to a size suitable for the height or posture of the detected person.

Further, in the server device 3 according to the present embodiment, the size of the portion corresponding to the first three-dimensional frame in the two-dimensional plane frame converted from the three-dimensional frame is the detection result of the human head from the captured image. If it is larger than the size of the based plane frame, the height of the second three-dimensional frame in the three-dimensional virtual space is reduced. As a result, the server device 3 can be expected to accurately adjust the second three-dimensional frame generated as a predetermined size to a size suitable for the height or posture of the detected person.

Further, the server device 3 according to the present embodiment detects an object captured in the captured image including distortion by using the YOLO algorithm. The YOLO algorithm has an accurate track record as an algorithm for detecting an object from an image, and a bounding box can be attached to the detected object. Therefore, the server device 3 according to the present embodiment is used. It is an algorithm suitable for the processing to be performed. However, the server device 3 may detect an object from the captured image by using an algorithm other than YOLO.

Further, in the information processing system according to the present embodiment, the server device 3 acquires a captured image (distorted image) including distortion captured by the camera 1 through a wide-angle lens, and the object (person) copied on the acquired captured image. A three-dimensional frame (bounding box) surrounding the above is generated, superimposed on the generated three-dimensional frame photographed image, and displayed on the display unit 54 of the terminal device 5. As a result, the information processing system can be expected to superimpose a bounding box suitable for a distorted captured image and present the existence of the object captured in the captured image to the user.

Further, the information processing system according to the present embodiment extracts a partial image including an object from a photographed image including distortion based on the generated three-dimensional frame. As a result, it can be expected that processing such as face recognition or action recognition can be performed accurately based on the extracted partial image.

Further, in the three-dimensional frame generated by the information processing system according to the present embodiment, a first frame that detects a person captured in a photographed image as an object and surrounds the person's head and the body of this person. Includes a second frame surrounding. By generating a three-dimensional frame including such a plurality of frames, it is possible to extract and use an image suitable for processing such as face recognition or action recognition.

Further, in the information processing system according to the present embodiment, the server device 3 generates a plane frame in the two-dimensional plane of the captured image based on the three-dimensional bounding box generated in the three-dimensional virtual space of the world coordinate system, and captures the image. The size of the image is compared with the plane frame directly generated by an algorithm such as YOLO. The server device 3 can estimate the height of the object from the ground, the position in the three-dimensional virtual space, and the like based on the comparison result.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Camera 3 Server device 5 Terminal device 31 Processing unit 31a Image acquisition unit 31b Human detection unit 31c Coordinate conversion unit 31d Bounding box generation unit 31e Coordinate reverse conversion unit 31f Bounding box adjustment unit 31g Image superimposition unit 31h Image transmission unit 32 Storage unit 32a Server program 32b YOLO learning model 33 Communication unit 51 Processing unit 51a Image reception unit 51b Display processing unit 52 Storage unit 52a Program 53 Communication unit 54 Display unit 55 Operation unit 98,99 Recording medium 101 Bounding box N network

Claims (5)

Acquires a distorted image including the distortion taken by the camera,
A three-dimensional frame surrounding the object copied in the acquired distorted image is generated.
A partial image including the object is extracted from the distorted image based on the generated three-dimensional frame.
The three-dimensional frame is a shape in which two planar frames surrounding the object intersect.
Image processing method. The two-dimensional coordinates of the object in the distorted image are converted into three-dimensional coordinates in a virtual three-dimensional space.
In the three-dimensional space, two intersecting planar frames surrounding the object are generated.
The generated three-dimensional coordinates of the frame are inversely converted into the two-dimensional coordinates in the distorted image.
Select one of the two planar shaped frames based on the size in the distorted image.
A partial image including the object is extracted from the distorted image based on the frame of the selected planar shape.
The image processing method according to claim 1 . Correct the tilt of the extracted partial image,
The image processing method according to claim 1 or 2 . On the computer
Acquires a distorted image including the distortion taken by the camera,
A three-dimensional frame surrounding the object copied in the acquired distorted image is generated.
Based on the generated three-dimensional frame, a process of extracting a partial image including the object from the distorted image is executed .
The three-dimensional frame is a computer program in which two planar frames surrounding the object intersect . An acquisition unit that acquires a distorted image including distortion taken by the camera,
A generation unit that generates a three-dimensional frame surrounding the object copied in the acquired distorted image, and a generation unit.
A processing unit for extracting a partial image including the object from the distorted image based on the generated three-dimensional frame is provided .
The three-dimensional frame is an image processing device having a shape in which two planar frames surrounding the object intersect .

Images