JP2023016187A - Image processing method, computer program, and image processing device - Google Patents

Abstract

To provide an image processing method, a computer program and an image processing device for generating a bounding box that encloses an object shown in a distorted image.SOLUTION: An image processing method pertaining to the present embodiment includes: acquiring a distorted image which is photographed by a camera and includes distortion; detecting an object shown in the acquired distorted image; converting the two-dimensional coordinates of the object in the distorted image to three-dimensional coordinates in a virtual three-dimensional space; generating a solid frame that encloses the object in the three-dimensional space; inversely converting the three-dimensional coordinates of the generated sold frame to two-dimensional coordinates in the distorted image; and superimposing a planar frame obtained by converting the solid frame to two-dimensional coordinates on top of the distorted image.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing method, a computer program, and an image processing apparatus for detecting an object captured in an image captured by a camera.

In recent years, advances in techniques such as machine learning and deep learning have made it possible to detect a specific object, such as a person or a vehicle, from an image captured by a camera. This technology is used in surveillance cameras, in-vehicle cameras, etc., and can determine, for example, whether or not a person exists within the imaging range of the camera, and what kind of behavior the person is taking. 2. Description of the Related Art Surveillance cameras and the like are desired to be capable of photographing as wide a range as possible, and are often photographed using a lens with a wide angle of view (so-called wide-angle lens). An image captured using a wide-angle lens is an image in which distortion is more pronounced toward the periphery.

Japanese Unexamined Patent Application Publication No. 2002-200002 proposes an image processing system that corrects distortion of an image generated by photographing an object. This image processing system includes an illumination unit that illuminates an object, a projection unit that projects a predetermined pattern onto the object, and an image of the object illuminated by the illumination unit that does not include the predetermined pattern to generate first image data. an imaging device that has an imaging unit that captures an image of an object illuminated by the illumination unit together with a predetermined pattern to generate second image data; and a driving unit that relatively moves the illumination unit and the imaging unit with respect to the object. . Further, the image processing system detects the distortion amount of the pattern image corresponding to the predetermined pattern in the second image indicated by the second image data, and corrects the distortion of the first image indicated by the first image data based on the detected distortion amount. and an image processing device that performs

JP 2019-192949 A

When an object is detected from an image captured by a camera, a rectangular frame surrounding the detected object, a so-called bounding box, is displayed superimposed on the captured image in order to present the detection results to the user. is done. However, when a bounding box is displayed for a distorted image captured by a camera, the size of the bounding box surrounding the object becomes large because the object is distorted, and objects other than the object enter the bounding box. In some cases, the precision of the bounding box was degraded. The captured image may also be distorted when the camera takes a picture, for example, through a normal lens, and in particular when the camera takes a picture through a wide-angle lens, the image may be greatly distorted.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and aims to provide an image processing method capable of generating a bounding box surrounding an object captured in a distorted image. It is to provide a program and an image processing device.

An image processing method according to one embodiment acquires a distorted image captured by a camera and includes distortion, detects an object appearing in the acquired distorted image, and calculates two-dimensional coordinates of the object in the distorted image. , transforming into three-dimensional coordinates in a virtual three-dimensional space, generating a three-dimensional frame surrounding the object in the three-dimensional space, and inverting the three-dimensional coordinates of the generated three-dimensional frame to the two-dimensional coordinates in the distorted image; A planar frame obtained by transforming the three-dimensional frame into two-dimensional coordinates is superimposed on the distorted image.

According to one embodiment, it is expected to generate a bounding box around the object for the distorted image.

1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment; FIG. 1 is a block diagram showing the configuration of a server device according to an embodiment; FIG. 2 is a block diagram showing the configuration of a terminal device according to this embodiment; FIG. FIG. 4 is a schematic diagram for explaining bounding box generation processing performed by the server device according to the present embodiment; FIG. 4 is a schematic diagram for explaining bounding box adjustment processing performed by the server device according to the present embodiment; FIG. 4 is a schematic diagram for explaining bounding box adjustment processing performed by the server device according to the present embodiment; FIG. 4 is a schematic diagram showing an example of image display by a terminal device; 4 is a flow chart showing the procedure of processing performed by the server device according to the present embodiment; FIG. 5 is a schematic diagram for explaining bounding box generation processing for image extraction performed by the server device according to the present embodiment; FIG. 4 is a schematic diagram for explaining image extraction based on bounding boxes; 5 is a flow chart showing the procedure of image extraction processing performed by the server device according to the present embodiment;

A specific example of an information processing system according to an embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

<System configuration>
FIG. 1 is a schematic diagram for explaining an outline of an information processing system according to this embodiment. In the information processing system according to the present embodiment, for example, a camera 1 installed in a commercial facility or a public facility takes pictures of the surroundings, a server device 3 detects a person from the taken image, and a bounding image surrounding the detected person is detected. This system superimposes a box on a photographed image and displays it on the terminal device 5 .

The camera 1 according to the present embodiment is a camera that takes pictures through a wide-angle lens with a wide angle of view, for example, a camera with a horizontal angle of view of about 180°. The camera 1 shoots several times to several tens of times per second, and transmits the images obtained by the shooting to the server device 3 . The camera 1 communicates with a server device 3 via a wired or wireless network such as a LAN (Local Area Network), a wireless LAN, a mobile phone communication network, or the Internet, and transmits captured images to the server device 3. . In the present embodiment, the camera 1 takes pictures through a wide-angle lens, but the invention is not limited to this, and pictures may be taken through a lens different from the wide-angle lens.

The server device 3 receives captured images transmitted from one or a plurality of cameras 1 and performs processing for detecting a person appearing in the received captured images. The server device 3 generates a bounding box 101 surrounding the person detected from the captured image, and generates a display image in which the generated bounding box 101 is superimposed on the original captured image. The server device 3 transmits the generated display image to the terminal device 5 .

The terminal device 5 is a device used by a user who performs monitoring using the camera 1, and may be configured using a general-purpose information processing device such as a PC (personal computer), a smart phone, or a tablet terminal device. The terminal device 5 receives the image transmitted from the server device 3 and displays the received image on the display unit. The image transmitted from the server device 3 to the terminal device 5 and displayed on the display unit is the image captured by the camera 1. If a person exists in the image, a bounding box 101 surrounding the person is superimposed. This is an image.

In the present embodiment, the server device 3 performs the process of detecting a person in the captured image, the process of generating the bounding box 101 surrounding the detected person, and the like, but the present invention is limited to this. isn't it. For example, the camera 1 performs processing for detecting a person, processing for generating the bounding box 101, and the like, and the photographed image superimposed with the bounding box 101 is directly transmitted to the terminal via the server device 3 or not via the server device 3. It may be sent to device 5 . In the present embodiment, the server device 3 transmits the photographed image on which the bounding box 101 is superimposed to the terminal device 5, and the terminal device 5 receives it and displays it on the display unit. isn't it. For example, if the server device 3 has a display unit, the captured image with the bounding box 101 superimposed thereon may be displayed on the display unit of the server device 3 . Further, for example, when the camera 1 has a display unit, the camera 1 may generate and superimpose the bounding box 101 and display the photographed image with the bounding box 101 superimposed on its own display unit.

<Device configuration>
FIG. 2 is a block diagram showing the configuration of the server device 3 according to this embodiment. The server device 3 according to this embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In this embodiment, the explanation is given assuming that the processing is performed by one server device, but the processing may be performed by a plurality of server devices in a distributed manner.

The processing unit 31 is configured using an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory) and a RAM (Random Access Memory). It is By reading and executing the server program 32a stored in the storage unit 32, the processing unit 31 performs processing for acquiring an image captured by the camera 1, processing for detecting a person in the image, processing for detecting the detected person. Various processes such as a process of generating a surrounding bounding box and a process of superimposing the bounding box on the captured image and transmitting the captured image to the terminal device 5 are performed.

The storage unit 32 is configured using, for example, a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31 . In the present embodiment, the storage unit 32 stores a server program 32a executed by the processing unit 31, and a learned YOLO (You Only Look Once) used for processing for detecting a person from an image captured by the camera 1. It stores the model 32b.

In the present embodiment, the server program (program product) 32a is provided in a form recorded in a recording medium 99 such as a memory card or an optical disk, and the server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. Remember. However, the server program 32a may be written in the storage unit 32 during the manufacturing stage of the server device 3, for example. Further, for example, the server program 32a may be distributed by another remote server device or the like and acquired by the server device 3 through communication. For example, the server program 32 a may be recorded in the recording medium 99 and read by a writing device and written in the storage unit 32 of the server device 3 . The server program 32 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99 .

The YOLO learning model 32b is a learning model based on CNN (Convolutional Neural Network), detects a specific object from the input image, and determines the type of the detected object, the bounding box surrounding the detected object, etc. It is a learning model that has undergone machine learning so as to output the information of The YOLO learning model 32b according to the present embodiment is machine-learned in advance so as to detect a person's head in an image and output information about a bounding box surrounding the detected person's head. Since the YOLO learning model is an existing technology, detailed descriptions of its structure, algorithms, etc. will be omitted. In this embodiment, the YOLO learning model is used to detect the human head, but the present invention is not limited to this. A learning model such as (Single Shot multibox Detector) may be used. The storage unit 32 stores information on the structure of the learning model that constitutes the YOLO learning model 32b, information on internal parameters determined by machine learning, and the like.

The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In this embodiment, the communication unit 33 communicates with the camera 1 and the terminal device 5 via the network N. FIG. The communication unit 33 transmits data received from the processing unit 31 to other devices, and provides the processing unit 31 with data received from other devices.

Note that the storage unit 32 may be an external storage device connected to the server device 3 . The server device 3 may be a multicomputer including a plurality of computers, or may be a virtual machine virtually constructed by software. The server device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives operation inputs, or a display unit that displays images. .

Further, in the server device 3 according to the present embodiment, the processing unit 31 reads out and executes the server program 32a stored in the storage unit 32, so that the image acquisition unit 31a, the human detection unit 31b, the coordinate conversion unit 31c, A bounding box (abbreviated as BBox in FIG. 2) generation unit 31d, coordinate inverse transformation unit 31e, bounding box adjustment unit 31f, image superimposition unit 31g, image transmission unit 31h, etc. are included in the processing unit 31 as software functional units. Realized. In the figure, as the functional units of the processing unit 31, the functional units that perform processing on the image captured by the camera 1 are illustrated, and the functional units related to other processing are omitted.

The image acquisition unit 31 a acquires an image (captured image) captured by the camera 1 by receiving an image transmitted by the camera 1 through the communication unit 33 . In the present embodiment, the camera 1 shoots through a wide-angle lens, and the shot image acquired by the image acquisition unit 31a is an image with distortion in the peripheral portion (distorted image). The image acquisition unit 31a temporarily stores the acquired captured image in the storage unit 32 . Note that the captured image temporarily stored in the storage unit 32 by the image acquisition unit 31a is deleted from the storage unit 32 at an appropriate timing after the bounding box is superimposed by subsequent image processing and transmitted to the terminal device 5. you can

The human detection unit 31b performs a process of detecting the head of a person in the photographed image acquired by the image acquisition unit 31a from the camera 1 . In the present embodiment, the human detection unit 31b uses the YOLO learning model 32b stored in the storage unit 32 to perform detection processing. The human detection unit 31b inputs the distorted shot image acquired from the camera 1 by the image acquisition unit 31a to the YOLO learning model 32b. The YOLO learning model 32b detects a human head in the input image, generates a bounding box surrounding the detected human head, and outputs information about the generated bounding box.

In this embodiment, the bounding box output by the YOLO learning model 32b is a rectangular frame that surrounds the detected human body. This frame is represented, for example, by using the coordinates of two points on the two-dimensional plane of the captured image. For example, the frame is represented by the coordinates (x1, y1) of the upper left corner and the coordinates (x2, y2) of the lower right corner, and the YOLO learning model 32b uses these (x1, y1) and (x2, y2) Output the coordinates of as bounding box information. Note that when a plurality of people are shown in the captured image, the YOLO learning model 32b generates a bounding box surrounding the head of each person and outputs the coordinates of each bounding box.

The human detection unit 31b acquires bounding box information output by the YOLO learning model 32b in response to the input of the captured image. Further, in the present embodiment, the human detection unit 31b calculates the coordinates of the center of the head of the person captured in the captured image based on the bounding box information acquired from the YOLO learning model 32b. For example, when the YOLO learning model 32b outputs the coordinates (x1, y1) of the upper left corner and the coordinates (x2, y2) of the lower right corner of the bounding box, the human detection unit calculates the coordinates of the midpoint of these two points. By doing so, the coordinates of the center of the person's head are calculated. The person detection unit 31b provides the calculated coordinates of the center of the person's head to the coordinate conversion unit 31c.

The coordinate transformation unit 31c performs predetermined coordinate transformation processing on the coordinates of the center of the person's head detected by the person detection unit 31b. The coordinates given from the human detection unit 31b to the coordinate transformation unit 31c are two-dimensional coordinates in the distorted captured image captured by the camera 1. FIG. The coordinate transformation unit 31c performs a process of transforming the two-dimensional coordinates into three-dimensional coordinates in a three-dimensional virtual space. The details of the coordinate conversion processing performed by the coordinate conversion section 31c will be described later.

The bounding box generation unit 31d generates a three-dimensional bounding box surrounding the person's head based on the three-dimensional coordinates in the three-dimensional virtual space of the center (center of gravity) of the person's head transformed by the coordinate transformation unit 31c. to generate The three-dimensional bounding box generated by the bounding box generator 31d in the present embodiment has, for example, a rectangular parallelepiped or cube shape. Furthermore, the bounding box generator 31d according to the present embodiment generates a three-dimensional bounding box surrounding the body of this person (the part below the neck). That is, the bounding box generator 31d generates two three-dimensional bounding boxes respectively surrounding the head and the body other than the head for the person detected from the photographed image. When generating the bounding box surrounding the body, the bounding box generation unit 31d assumes that the height of the target person is a predetermined value (for example, 160 cm, 170 cm, etc.), and sets the height (length) of the bounding box to decide.

The coordinate inverse transformation unit 31e performs a process of transforming the three-dimensional coordinates of the bounding box generated by the bounding box generation unit 31d into two-dimensional coordinates. The conversion processing from three-dimensional coordinates to two-dimensional coordinates performed by the coordinate inverse conversion unit 31e corresponds to the inverse conversion of the conversion processing from two-dimensional coordinates to three-dimensional coordinates by the coordinate conversion unit 31c. The bounding box transformed by the coordinate inverse transformation unit 31e corresponds to a two-dimensional image obtained by projecting a three-dimensional bounding box onto the two-dimensional plane of the captured image captured by the camera 1. FIG. The details of the coordinate conversion processing performed by the coordinate inverse conversion unit 31e will be described later.

The bounding box adjustment unit 31f performs processing for adjusting the size of the bounding box based on the result of conversion of the three-dimensional bounding box into two-dimensional coordinates by the coordinate inverse conversion unit 31e. As described above, when the bounding box generator 31d generates a bounding box surrounding the body of the person in the captured image, the height of the bounding box is determined on the assumption that the height of the person is a predetermined value. , the actual height may differ from the assumed height. Therefore, the bounding box adjuster 31f adjusts the height of the bounding box. In the present embodiment, the bounding box adjustment unit 31f combines the bounding box surrounding the head converted into two-dimensional coordinates by the coordinate inverse conversion unit 31e and the head generated by the human detection unit 31b using the YOLO learning model 32b. The size of the bounding box surrounding the body is compared, and the height of the bounding box surrounding the body is adjusted based on the result of this comparison.

The image superimposing unit 31g performs image processing to superimpose the final bounding box whose height is adjusted by the bounding box adjusting unit 31f on the captured image of the camera 1 acquired by the image acquiring unit 31a. The image superimposing unit 31g supplies the image obtained by superimposing the bounding box on the captured image to the image transmitting unit 31h.

The image transmission unit 31h causes the communication unit 33 to transmit the image given from the image superimposition unit 31g to the predetermined terminal device 5, thereby causing the terminal device 5 to display the image. In the present embodiment, the server device 3 performs image processing for superimposing the bounding box on the captured image, but the present invention is not limited to this. 5, and the terminal device 5 may superimpose the bounding box on the captured image and display it.

FIG. 3 is a block diagram showing the configuration of the terminal device 5 according to this embodiment. The terminal device 5 according to the present embodiment includes a processing unit 51, a storage unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, an operation unit 55, and the like. The terminal device 5 is, for example, a device used by a user who performs tasks such as monitoring based on images captured by the camera 1, and can be configured using an information processing device such as a personal computer, a smartphone, or a tablet terminal device. .

The processing unit 51 is configured using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. By reading and executing the program 52a stored in the storage unit 52, the processing unit 51 receives an image captured by the camera 1 transmitted from the server device 3 and performs various processes such as displaying the image on the display unit. .

The storage unit 52 is configured using a storage device such as a hard disk or flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data necessary for the processing of the processing unit 51 . In the present embodiment, the storage unit 52 stores a program 52a executed by the processing unit 51. FIG. In this embodiment, the program 52a is distributed by a remote server device or the like, and the terminal device 5 acquires it through communication and stores it in the storage unit 52. FIG. However, the program 52a may be written in the storage unit 52 during the manufacturing stage of the terminal device 5, for example. For example, the program 52a may be stored in the storage unit 52 after the terminal device 5 reads the program 52a recorded in the recording medium 98 such as a memory card or an optical disk. For example, the program 52 a may be recorded in the recording medium 98 and read by a writing device and written in the storage unit 52 of the terminal device 5 . The program 52 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 98 .

The communication unit 53 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 53 communicates with the server device 3 via the network N. FIG. The communication unit 53 transmits the data given from the processing unit 51 to other devices, and gives the data received from the other devices to the processing unit 51 .

The display unit 54 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 51 . The operation unit 55 receives user operations and notifies the processing unit 51 of the received operations. For example, the operation unit 55 accepts user operations through input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 54 . Further, for example, the operation unit 55 may be an input device such as a mouse and a keyboard, and these input devices may be detachable from the terminal device 5 .

In the terminal device 5 according to the present embodiment, the processing unit 51 reads out and executes the program 52a stored in the storage unit 52, so that the image receiving unit 51a, the display processing unit 51b, and the like function as software functional units. It is implemented in the processing unit 51 . The program 52a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or web browser.

The image receiving unit 51 a performs processing for receiving an image transmitted by the server device 3 at the communication unit 53 . The image receiving unit 51a temporarily stores the received image in the storage unit 52, for example. In the present embodiment, the image received by the image receiving unit 51a from the server device 3 is an image obtained by superimposing a bounding box on the captured image of the camera 1. FIG.

The display processing unit 51b performs processing for displaying various characters or images on the display unit 54. FIG. In the present embodiment, the display processing unit 51b displays on the display unit 54 the image received by the image receiving unit 51a from the server device 3, or the image obtained by superimposing the bounding box on the captured image of the camera 1. FIG. In this embodiment, the camera 1 repeats photographing at a frequency of about several tens of times per second, that is, photographing moving images. The server device 3 superimposes a bounding box on each captured image captured by the camera 1 at a frequency of about several tens of times per second, and transmits the bounding box to the terminal device 5 . By displaying the image received from the terminal at a frequency of about several tens of times per second, a moving image on which the bounding box is superimposed is displayed.

<Bounding box generation processing>
FIG. 4 is a schematic diagram for explaining bounding box generation processing performed by the server device 3 according to the present embodiment. A rectangle denoted by reference numeral 110 in the drawing indicates an image captured by the camera 1 . The camera 1 according to the present embodiment shoots through a wide-angle lens, and the shot image 110 is an image including distortion due to the lens (distorted image). The server device 3 that has acquired the photographed image 110 from the camera 1 uses the YOLO learning model 32b to perform processing for detecting a person's head included in this photographed image 110 . By performing detection processing using the YOLO learning model 32b, the server device 3 can acquire information such as the coordinates of the bounding box 111 surrounding the person's head included in the captured image 110 as a detection result. The server device 3 calculates the coordinates (two-dimensional coordinates) of the center point 112 of the human head (the center point of the bounding box) based on information such as the coordinates of the bounding box 111 obtained from the YOLO learning model 32b.

Next, the server device 3 converts the coordinates of the center point 112 of the person's head included in the photographed image 110 into the two-dimensional coordinates of the corresponding point 113 on the two-dimensional plane from which the distortion is removed from the photographed image 110 . Image processing for correcting an image including distortion due to a lens is a conventional technique, so a detailed description thereof will be omitted. yd) and points (x,y) of the undistorted image and back.

The server device 3 calculates the two-dimensional coordinates of the point 112 corresponding to the center of the person's head included in the captured image 110 including distortion, based on the above equation (1), as a point on the two-dimensional plane from which the distortion has been removed. 113 two-dimensional coordinates. In this embodiment, the server device 3 does not need to perform coordinate conversion for all points included in the captured image 110, and only needs to convert the coordinates of the point 112 corresponding to the center of the person's head.

Next, the server device 3 converts the two-dimensional coordinates of the point 113 corresponding to the center of the person's head into three-dimensional coordinates in the three-dimensional virtual space. The conversion from two-dimensional coordinates to three-dimensional coordinates performed at this time corresponds to inverse conversion processing of the conversion processing generally performed when generating a two-dimensional image by setting a virtual camera in a three-dimensional virtual space. It is. For this reason, the conversion processing from three-dimensional coordinates to two-dimensional coordinates will be described first, and then the conversion processing from two-dimensional coordinates to three-dimensional coordinates will be described by applying this conversion processing in the opposite direction.

There are two types of three-dimensional coordinates: a world coordinate system and a camera coordinate system. The world coordinate system defines XYZ coordinates with a specific position in the three-dimensional virtual space as the origin. The camera coordinate system defines XYZ coordinates with a camera arranged in a three-dimensional virtual space as an origin. Let the coordinates of the world coordinate system be (Xw, Yw, Zw) and the coordinates of the camera coordinate system be (Xc, Yc, Zc). ), it can be performed by the following equation (2).

Further, if the coordinates of a point on a two-dimensional plane are (u, v), the transformation from the camera coordinate system to the two-dimensional coordinates is performed using constants called intrinsic parameters of the camera, as follows (3) can be done with the formula

In the above equations (2) and (3), the internal and external variables of the camera are constants determined based on the characteristics of the camera, the position of the camera, and the like. Also, the constant c in the formula (3) is a numerical value representing a scale or a ratio, and is determined according to the system configuration and the like. Based on the above formulas (2) and (3), the conversion from the three-dimensional coordinates of the world coordinate system to the two-dimensional coordinates of the two-dimensional plane can be performed by the following formula (4).

By substituting an arbitrary point (Xw, Yw, Zw) in the world coordinate system into the above equation (4), the corresponding two-dimensional coordinates (u, v) can be uniquely specified. It is expected that the two-dimensional coordinates will be converted to three-dimensional coordinates by performing the inverse operation of the equation (4), that is, the operation of calculating the product of the inverse matrix of the matrix of the internal and external variables of the camera and the two-dimensional coordinates. However, since the number of dimensions increases when converting from two-dimensional coordinates to three-dimensional coordinates, information is insufficient to uniquely identify the three-dimensional coordinates. Therefore, in the information processing system according to the present embodiment, it is assumed that the height of a person detected from captured image 110 is a predetermined value (for example, 175 cm). That is, the server device 3 according to the present embodiment assumes that the value of Zw in the equation (4) is a predetermined value, and calculates the values of Xw and Yw of the three-dimensional coordinates from the two-dimensional coordinates (u, v). , the two-dimensional coordinates are converted into three-dimensional coordinates.

The server device 3 transforms the two-dimensional coordinates of the point 113 on the two-dimensional plane from which the distortion has been removed into the three-dimensional coordinates of the corresponding point 114 in the world coordinate system based on the above equation (4). In FIG. 4, conversion from two-dimensional coordinates to three-dimensional coordinates in the camera coordinate system and conversion from three-dimensional coordinates in the camera coordinate system to three-dimensional coordinates in the world coordinate system are shown separately. The server device 3 may collectively perform these two stages of coordinate transformation.

Next, the server device 3 creates a three-dimensional bounding box 115 surrounding the head of the person detected from the captured image 110 and a three-dimensional bounding box 116 surrounding the body below the head in the three-dimensional space of the world coordinate system. Perform processing to generate and First, the server device 3 considers the point 114 obtained by the coordinate transformation described above to be the center (center of gravity) of the person's head, and generates a rectangular parallelepiped frame of a predetermined size around the point 114 so that the head Generate a three-dimensional bounding box 115 that encloses the part. In this example, the three-dimensional bounding box 115 surrounding the head is a cubic shape of 20 cm×20 cm×20 cm.

The server device 3 generates a three-dimensional bounding box 116 surrounding the body by generating a rectangular parallelepiped frame of a predetermined size below the three-dimensional bounding box 115 surrounding the head. In this example, the height of a person is 175 cm, the three-dimensional bounding box 116 surrounding the body is a 50 cm × 50 cm × 150 cm rectangular parallelepiped with a gap of 5 cm between it and the three-dimensional bounding box 115 surrounding the head. and

Next, the server device 3 converts the three-dimensional bounding boxes 115 and 116 generated in the three-dimensional space of the world coordinate system into three-dimensional bounding boxes 117 and 118 in the three-dimensional space of the camera coordinate system, and further converts them into two-dimensional planes. to bounding boxes 119 and 120 in . At this time, the server device 3 may directly convert to the three-dimensional bounding boxes 115 and 116 in the world coordinate system and the bounding boxes 119 and 120 in the two-dimensional plane using the above equation (4). The server device 3 performs the coordinate transformation of formula (4) on a plurality of vertices contained in the three-dimensional bounding boxes 115 and 116, and connects the plurality of vertices on the two-dimensional plane after the coordinate transformation to create a bounding box 119. , and 120 may be generated, or the bounding boxes 119 and 120 may be generated by performing coordinate transformation on all points such as the vertices and frame lines forming the three-dimensional bounding boxes 115 and 116 .

Next, the server device 3 converts the bounding boxes 119 and 120 on the two-dimensional plane without distortion into the bounding boxes 121 and 122 on the image including the distortion of the wide-angle lens. At this time, the server device 3 transforms the coordinates of the bounding box based on the above equation (1). The server device 3 performs the coordinate transformation of formula (1) on a plurality of vertices contained in the bounding boxes 119 and 120 on the undistorted two-dimensional plane, and converts the vertices on the distorted image after the coordinate transformation. may be connected to generate the bounding boxes 121 and 122 , or the coordinate transformation may be performed for all points such as the vertices and frame lines forming the bounding boxes 119 and 120 to generate the bounding boxes 121 and 122 . In the example shown in FIG. 4, coordinate transformation is performed on all points of the bounding box, resulting in bounding boxes 121 and 122 containing distorted lines.

It should be noted that the bounding boxes 121 and 122 generated by the above-described processing have shapes obtained by projecting the three-dimensional bounding boxes 115 and 116, such as cubes and rectangular parallelepipeds, onto a two-dimensional plane. The server device 3 may shape the bounding box having a three-dimensional shape such as a cube or rectangular parallelepiped into a bounding box having a planar shape such as a square or a rectangle, for example.

<Bounding box adjustment processing>
The bounding box generated by the bounding box generation process described above is generated by assuming that the person detected from the captured image is standing and has a predetermined height (for example, 175 cm). The difference in height between people is considered to be within a few tens of centimeters, but when a person is sitting, the height from the ground to the head may be 50 cm or more lower than the height of the person sitting. The height of the bounding box generated for may be incorrect. Therefore, the server device 3 according to the present embodiment performs processing for adjusting the height of the bounding box (the bounding box surrounding the body).

5 and 6 are schematic diagrams for explaining bounding box adjustment processing performed by the server device 3 according to the present embodiment. The server device 3 stores the size (second size) of the bounding box 111 given by the YOLO learning model 32b to the human head captured in the captured image 110 captured by the camera 1, and the human head in the world coordinate system. The size (first size) of the bounding box 121 generated based on the three-dimensional bounding box 115 surrounding the body is compared with the size (first size) of the bounding box 121, and the height of the bounding box 122 surrounding the human body is adjusted based on the comparison result.

In the example shown in FIG. 5, the camera 1 is installed at a position higher than the person to be photographed, and photographs the surroundings in a bird's-eye view. A photographed image 110 obtained by photographing by the camera 1 can be regarded as an image projected onto a plane perpendicular to the optical axis of the camera 1, and is indicated by a straight line with reference numeral 110 in the figure. In the illustrated situation, the head of a person standing at a position A close to the camera 1 and the head of a person sitting at a position B far from the camera 1 appear at the same position in the captured image 110. Become. However, the person's head at position A closer to camera 1 appears larger in captured image 110 than the person's head at position B farther from camera 1 .

In the information processing system according to the present embodiment, it is assumed that the person detected from the captured image 110 is standing with a predetermined height when the bounding box is generated. Therefore, the bounding box generated for the person in the captured image 110 is suitable for the person standing at position A in FIG. There is a possibility that it is photographed on 110.

When the person photographed in the photographed image 110 is a person sitting at position B, as shown in the upper part of FIG. It is smaller than the bounding box 121 generated based on the 3D bounding box 115 . On the other hand, if the person photographed in the photographed image 110 is a person standing at position A, as shown in the lower part of FIG. The bounding box 111 and the bounding box 121 generated based on the three-dimensional bounding box 115 have approximately the same size.

After generating the two-dimensional bounding box 121 superimposed on the captured image 110 based on the three-dimensional bounding box 115, the server device 3 according to the present embodiment generates the size (first size) of the bounding box 121, The size (second size) of the bounding box 111 surrounding the human head detected by the YOLO learning model 32b is compared. Size comparisons can be made, for example, by comparing areas, longest side lengths, or diagonal lengths. Since the bounding box 121 has a three-dimensional shape, the server device 3 may convert it into a planar bounding box (rectangular or square) and compare the sizes. The server device 3 can convert the three-dimensional bounding box 121 into a planar shape, for example, by generating a minimum square or rectangular frame that encloses the bounding box 121 . If the two bounding boxes 111 and 121 have approximately the same size, the server device 3 determines that the generated bounding boxes 121 and 122 are appropriate and adopts them.

If the size of the bounding box 111 of the YOLO learning model 32b is smaller than the size of the bounding box 121 generated based on the three-dimensional bounding box 115, the server device 3 determines that the generated bounding boxes 121 and 122 are inappropriate. and adjust the bounding box. The server device 3 calculates the height (the height from the ground to the head) of the person photographed in the photographed image 110 from the predetermined height (for example, 170 cm) of the standing person, and the height from the ground to the head of the sitting person. By changing the height to a predetermined height (for example, 100 cm) and performing the above-described bounding box generation processing, bounding boxes 121 and 122 on the photographed image 110 suitable for a sitting person are generated.

By changing the estimated value of the height of the head, the values of the three-dimensional coordinates Xw and Yw in the world coordinate system calculated by the equation (4) also change. Therefore, the process of adjusting the bounding box described above is also the process of adjusting the position of the person in the world coordinate system. For this reason, the server device 3 increases or decreases, for example, the estimated value of the height of the person's head captured in the captured image 110, the size of the bounding box 121 generated from the three-dimensional bounding box 115, and the size of the YOLO learning model 32b. By searching for the height of the head when the size of the bounding box 111 matches (the difference is less than the threshold), the height of the person (the height from the ground to the head) in the world coordinate system can be obtained with accuracy It is possible to estimate well and accurately estimate the position of a person in the world coordinate system.

In addition, the height and position of a person can be estimated by repeating the increase and decrease of the estimated height value to search for a value that matches the sizes of the two bounding boxes 121 and 111. For example, machine learning can be used to estimate the height and position of the person. A method using a model may be employed. The learning model accepts, for example, the size of the two bounding boxes 121 and 111 or the image with the two bounding boxes 121 and 111 as input, and machine learning is performed in advance so that the height of the target person is output. can be made.

In this embodiment, the height from the ground to the head is calculated using numerical values such as 170 cm or 100 cm, but these numerical values are examples and are not limited to these. These numerical values may be determined in advance by a designer or the like of the information processing system according to the present embodiment, and stored in the storage unit 32 of the server device 3 as default values for processing together with the server program 32a. Further, these numerical values may be input by the user when the server device 3 executes processing. For example, if the user inputs a numerical value, the server apparatus 3 can use this numerical value for processing, and if no numerical value is input, use a default value for processing.

After adjusting the bounding boxes 121 and 122 , the server device 3 generates an image in which the obtained bounding boxes 121 and 122 are superimposed on the image 110 captured by the camera 1 . The server device 3 transmits the generated image to the predetermined terminal device 5 and causes the display section 54 of the terminal device 5 to display this image. As a result, the user using the terminal device 5 can confirm, on the terminal device 5 , an image in which the person captured in the captured image 110 of the camera 1 is surrounded by the bounding boxes 121 and 122 .

FIG. 7 is a schematic diagram showing an example of image display by the terminal device 5. As shown in FIG. In the illustrated display example, there are two people in the room photographed by the camera 1, and a bounding box surrounding the head of each person and a bounding box surrounding the body are superimposed on each person. ing. The camera 1 repeatedly takes images, for example, at a frequency of about several tens of times per second, and the image displayed by the terminal device 5 is also updated at about the same frequency. That is, in this embodiment, the camera 1 captures moving images, and the terminal device 5 displays the moving images. A bounding box surrounding each person moves to follow the movement of the person when the person moves in the image.

In the illustrated example, two points indicating the detected positional relationship of two persons are shown in a square area provided at the lower right of the screen displayed on the display unit 54 of the terminal device 5 . In this example, the server device 3 estimates the positions of people based on the size comparison of the bounding boxes described above, and uses the two-dimensional coordinates (Xw , Yw). Based on the two-dimensional coordinates (Xw, Yw) of each person, the server device 3 plots the points corresponding to each person in a square area, and informs the user of the estimation result of the position of the person detected from the captured image. can be presented.

<Flowchart>
FIG. 8 is a flow chart showing the procedure of processing performed by the server device 3 according to the present embodiment. The image acquisition unit 31a of the processing unit 31 of the server device 3 according to the present embodiment communicates with the camera 1 through the communication unit 33 and acquires the captured image captured by the camera 1 (step S1). The person detection unit 31b of the processing unit 31 uses the YOLO learning model 32b stored in the storage unit 32 to detect the person appearing in the captured image acquired in step S1 (step S2). The person detection unit 31b calculates the coordinates of the center point of the person's head based on the information such as the position of the bounding box surrounding the person's head output by the YOLO learning model 32b (step S3).

The coordinate transformation unit 31c of the processing unit 31 removes the distortion from the coordinates of the center point of the head in the captured image calculated in step S3 using the above equation (1). are converted into two-dimensional coordinates on the two-dimensional plane (step S4). The coordinate transformation unit 31c assumes that the detected person is standing and has a predetermined height (for example, 170 cm), and transforms the two-dimensional coordinates transformed in step S4 into the world based on the above equation (4). Convert to three-dimensional coordinates of the coordinate system (step S5).

The bounding box generation unit 31d of the processing unit 31 generates a cubic frame of a predetermined size (for example, 20 cm×20 cm×20 cm) around the three-dimensional coordinates converted in step S5 in the three-dimensional virtual space of the world coordinate system. to generate a three-dimensional bounding box surrounding the detected human head (step S6). The bounding box generation unit 31d generates a rectangular parallelepiped frame of a predetermined size (for example, 50 cm x 50 cm x 150 cm) below the bounding box surrounding the head generated in step S6. A bounding box surrounding the body (the part below the head) is generated (step S7).

The coordinate inverse transforming unit 31e of the processing unit 31 transforms the two generated bounding boxes into bounding boxes on a two-dimensional plane based on the above equation (4) (step S8). The coordinate inverse transforming unit 31e converts the undistorted two-dimensional plane bounding box transformed in step S8 into a distorted two-dimensional plane by performing an operation for imparting distortion using the above equation (1). It is converted into a bounding box on a plane (photographed image) (step S9).

The bounding box adjustment unit 31f of the processing unit 31 adjusts the size of the bounding box surrounding the detected human head among the bounding boxes converted in step S9 to the size of the human head output by the YOLO learning model 32b in step S2. It is determined whether or not the size is substantially the same as the size of the bounding box surrounding the part (the difference is within a threshold) (step S10). If both bounding boxes have different sizes (the difference exceeds the threshold) (S10: NO), the bounding box adjuster 31f sets the height of the detected person, that is, the height from the ground to the head, to a predetermined value (for example, 170 cm). to another value (step S11), and the process returns to step S4.

If it is possible to determine whether the detected person is standing or sitting, and if two types of height accuracy are sufficient for standing and sitting, bounding box adjustment is performed in step S11. The part 31f changes the height of the head of a person sitting down to a predetermined value (for example, 100 cm). In this case, the processing section 31 may proceed to step S12 without performing the determination of step S10 after performing the processing of steps S4 to S9 after changing the height.

On the other hand, when estimating the height and coordinates of the detected person in more detail, in step S11, the bounding box adjustment unit 31f increases or decreases the estimated value of the person's height by a predetermined value (for example, 1 cm). Then, the processing of steps S4 to S11 is repeated until the sizes of both bounding boxes are substantially the same as determined in step S10. For example, when the size of the bounding box converted in step S9 is larger than the size of the bounding box obtained by the YOLO learning model 32b, the bounding box adjuster 31f reduces the estimated height of the person. On the other hand, if the size of the bounding box converted in step S9 is larger than the size of the bounding box obtained by the YOLO learning model 32b, the bounding box adjuster 31f increases the estimated height of the person.

When it is determined that both bounding boxes have approximately the same size (S10: YES), the image superimposing unit 31g of the processing unit 31 superimposes the bounding box converted in step S9 on the captured image acquired in step S1. The resulting image is generated (step S12). The image transmission unit 31h of the processing unit 31 transmits the image generated in step S12 to the predetermined terminal device 5 through the communication unit 33 (step S13), and ends the process.

<Image extraction>
In the information processing system according to the present embodiment, the bounding box generated by the server device 3 from the captured image of the camera 1 is used for various processes other than the process of superimposing it on the captured image and displaying it on the terminal device 5. can be expected. For example, the server device 3 can perform processing for extracting an image region including a person from the captured image based on the generated bounding box.

In the bounding box generation process described above, the server device 3 generates two bounding boxes, one surrounding the head of the person captured in the captured image, and the other bounding box surrounding the body of the person. When extracting an image, the server device 3 according to the present embodiment generates one bounding box surrounding the whole body of a person instead of the two bounding boxes described above. First, the process of generating a bounding box for image extraction will be described.

FIG. 9 is a schematic diagram for explaining bounding box generation processing for image extraction performed by the server device 3 according to the present embodiment. The server device 3 detects a person's head using the YOLO learning model 32b for the captured image 110 acquired from the camera 1 . The server device 3 converts the coordinates of the human head in the distorted photographed image into two-dimensional coordinates on a two-dimensional plane from which distortion is removed, three-dimensional coordinates in the three-dimensional space of the camera coordinate system, and three-dimensional coordinates in the world coordinate system. Transform in order to three-dimensional coordinates in space. Note that the coordinate conversion processing up to this point is the same as the processing described with reference to FIG. 4, so detailed description thereof will be omitted.

In the three-dimensional space of the world coordinate system, the server device 3 generates two rectangular planar bounding boxes 141 and 142 so as to enclose the point 114 corresponding to the head obtained by the coordinate transformation process described above. . The first bounding box 141 is, for example, a rectangular frame 50 cm in the three-dimensional coordinate x direction, 0 cm in the y direction, and 175 cm in the z direction. It is placed in three-dimensional space so that the point 114 is included. The second bounding box 142 is, for example, a rectangular frame with three-dimensional coordinates of 0 cm in the x direction, 50 cm in the y direction, and 175 cm in the z direction. It is placed in three-dimensional space so that the point 114 is included. With this arrangement, the two bounding boxes 141 and 142 intersect (perpendicular).

The shape, size, etc. of the two bounding boxes 141 and 142 described above are merely examples, and are not limited to these. The shape of the bounding boxes 141 and 142 may not be square, but may be, for example, a triangle or a polygon with pentagons or more. Numerical values such as 50 cm and 175 cm for the bounding boxes 141 and 142 can be changed as appropriate. These numerical values may be determined in advance by a designer or the like of the information processing system according to the present embodiment, and stored in the storage unit 32 of the server device 3 as default values for processing together with the server program 32a. Further, these numerical values may be input by the user when the server device 3 executes processing. For example, if the user inputs a numerical value, the server apparatus 3 can use this numerical value for processing, and if no numerical value is input, use a default value for processing.

When the server apparatus 3 performs processing for estimating a person's height (the height from the ground to the head) by the method shown in FIGS. (z-direction length) can be the estimated height.

The server device 3 transforms the bounding boxes 141 and 142 generated in the three-dimensional space of the world coordinate system into three-dimensional bounding boxes 143 and 144 in the three-dimensional space of the camera coordinate system, and furthermore, the bounding box 145 in the two-dimensional plane. , 146. At this time, the server device 3 may directly transform the bounding boxes 141 and 142 of the world coordinate system into the bounding boxes 145 and 146 on the two-dimensional plane using the above equation (4).

The server device 3 converts the bounding boxes 145 and 146 on the two-dimensional plane without distortion into bounding boxes on the image including the distortion of the wide-angle lens. At this time, the server device 3 can transform the coordinates of the bounding box based on the above equation (1). Next, the server device 3 selects one of the two bounding boxes obtained by the coordinate transformation, and uses the selected one as the bounding box 147 for image extraction. The server device 3 can, for example, compare the width or area of two bounding boxes on an image including distortion, and select the bounding box with the larger width or area.

The server device 3 uses the bounding box 147 for image extraction generated by the above procedure to extract an image area including a person in the captured image including the distortion acquired from the camera 1. conduct. Note that the server device 3 does not generate the image extraction bounding box 147 by the procedure shown in FIG. 9, but based on the two bounding boxes 121 and 122 generated by the procedure shown in FIG. may generate a bounding box for In this case, the server device 3 can integrate the bounding box 121 surrounding the person's head and the bounding box 122 surrounding the person's body to generate one two-dimensional (rectangular) bounding box for image extraction. can. The server device 3 can generate a bounding box for image extraction by generating a minimum rectangular frame including, for example, a bounding box 121 surrounding the head and a bounding box 122 surrounding the body.

FIG. 10 is a schematic diagram for explaining image extraction based on bounding boxes. The upper part of FIG. 10 shows an enlarged image of a person and its surroundings captured in the captured image captured by the camera 1 . Since the camera 1 has taken a picture through a wide-angle lens, in the illustrated image, the person is not standing upright with respect to the ground, but is obliquely tilted (distorted) in the image. In this image, a bounding box 131 indicated by a solid-line rectangle and a bounding box 132 indicated by a dashed-line rectangle are superimposed. A solid-line bounding box 131 is a bounding box 131 for image extraction generated by the server device 3 according to the present embodiment according to the procedure shown in FIG. A dashed bounding box 132 is a bounding box 132 obtained by detecting the whole person and attaching a bounding box by the YOLO technique.

By extracting the image within the bounding box 131 generated for the captured image, the server device 3 can extract an image area in which a person is shown from the distorted captured image. An image extracted based on the bounding box 131 is shown in the lower left of FIG. The server device 3 performs a tilt correction process (for example, an image rotation process) on the image extracted from the bounding box 131 that is tilted with respect to the captured image. The bottom left of FIG. 10 shows an image after tilt correction for the extracted image. The image extracted based on the bounding box 131 generated by the server device 3 according to the present embodiment is an image in which the detected person stands along the vertical and horizontal directions of the image. Note that the process of correcting the tilt of the extracted image may be performed by a method other than image rotation.

On the other hand, the YOLO bounding box 132 is a rectangular bounding box along the vertical and horizontal directions of the distorted photographed image. An image obtained by extracting an image area based on the YOLO bounding box 132 from the captured image is shown in the lower right of FIG. The image extracted based on the YOLO bounding box 132 is one in which the detected person is tilted (distorted) in the image. Also, the extracted image has more pixels (compared to the image extracted based on the bounding box 131) belonging to non-human objects (e.g. background) and contains more information about non-human objects. becomes an image.

The server device 3 extracts the image extracted based on the bounding box 131, for example, a process of identifying who the person shown in this image is, a process of recognizing the behavior of the person shown in this image, or It can be used as input information for various processes such as the process of tracking a person in an image. The image extracted based on the bounding box 131 generated by the server device 3 according to the present embodiment has more people detected than the image extracted based on the bounding box 132 of YOLO. It can be expected that the ratio of the number of pixels occupied will increase, and it can be expected that the accuracy of subsequent processing will be improved.

FIG. 11 is a flow chart showing the procedure of image extraction processing performed by the server device 3 according to the present embodiment. The processing unit 31 of the server device 3 according to the present embodiment acquires the captured image captured by the camera 1 with the image acquisition unit 31a (step S31), and the human detection unit 31b detects a person using the YOLO learning model 32b. Detection is performed (step S32). The person detection unit 31b calculates the coordinates of the central point of the person's head based on the information such as the position of the bounding box surrounding the person's head output by the YOLO learning model 32b (step S33). The coordinate transformation unit 31c of the processing unit 31 transforms the calculated coordinates of the center point of the head into two-dimensional coordinates on the two-dimensional plane from which the distortion has been removed (step S34), and converts the two-dimensional coordinates to 3 Convert to dimensional coordinates (step S35).

The bounding box generation unit 31d of the processing unit 31 generates two rectangular planar bounding boxes containing the three-dimensional coordinates of the center point of the head transformed in step S35 in the three-dimensional virtual space of the world coordinate system. (Step S36). The two bounding boxes can be, for example, a rectangular planar frame of 50 cm×0 cm×175 cm and a rectangular planar frame of 0 cm×50 cm×175 cm.

The coordinate inverse transformation unit 31e of the processing unit 31 transforms the two generated bounding boxes into bounding boxes on a two-dimensional plane (step S37). The coordinate inverse transformation unit 31e transforms the two bounding boxes on the undistorted two-dimensional plane transformed in step S37 into two bounding boxes on the distorted two-dimensional plane (captured image) (step S38). ).

The processing unit 31 selects, for image extraction, one of the two bounding boxes on the captured image generated in step S38, which has a larger width or area, and extracts the bounding box from the captured image using the bounding box for image extraction. The enclosed image area is extracted (step S39). The processing unit 31 performs processing such as rotation on the extracted image region to correct the tilt (step S40), and ends the image extraction processing.

<Summary>
In the information processing system according to the present embodiment having the above configuration, the server device 3 acquires a photographed image including distortion (distorted image) photographed by the camera 1 through a wide-angle lens, and the object photographed in the acquired photographed image to detect (a person) The server device 3 converts the two-dimensional coordinates of the detected person into three-dimensional coordinates in a virtual three-dimensional space, and generates a three-dimensional frame (a three-dimensional bounding box) surrounding the object in the three-dimensional space. The server device 3 inversely transforms the generated three-dimensional coordinates of the three-dimensional frame into two-dimensional coordinates in the captured image captured by the camera 1 . The server device 3 superimposes the obtained planar frame (planar bounding box) on the distorted photographed image photographed by the camera 1 and displays it on the terminal device 5 or the like. As a result, the server device 3 can be expected to superimpose a bounding box suitable for a distorted photographed image and present the detection result of the object from the photographed image to the user.

In addition, the server device 3 according to the present embodiment converts the two-dimensional coordinates of the object in the distorted captured image captured by the camera 1 into the two-dimensional coordinates in the image from which the distortion is removed. The server device 3 transforms the two-dimensional coordinates in the image from which the distortion has been removed into three-dimensional coordinates in a camera coordinate system centered on the camera 1, and transforms the three-dimensional coordinates in the camera coordinate system into three-dimensional coordinates in the world coordinate system. to generate a three-dimensional frame surrounding the object in the world coordinate system. As a result, the server device 3 can be expected to accurately generate a three-dimensional frame that encloses the object in the photographed image from the distorted photographed image photographed through a wide-angle lens. In the present embodiment, the server device 3 performs transformation from two-dimensional coordinates to three-dimensional coordinates in the camera coordinate system and transformation from three-dimensional coordinates in the camera coordinate system to three-dimensional coordinates in the world coordinate system (4). Although it is collectively performed based on the formula, it is not limited to this, and each coordinate transformation may be performed individually.

In addition, the server device 3 according to the present embodiment detects the human head captured in the photographed image including distortion, converts the two-dimensional coordinates of the human head into three-dimensional coordinates, and converts the coordinates in the three-dimensional virtual space. A first three-dimensional frame surrounding a person's head is generated, a second three-dimensional frame surrounding a person's body is generated, and the first three-dimensional frame and the second three-dimensional frame are combined to form a three-dimensional frame surrounding the person. Generate. As a result, it can be expected that the processing load of the coordinate conversion performed by the server device 3 can be reduced.

Further, the server device 3 according to the present embodiment generates a first three-dimensional frame of a predetermined size surrounding a person's head, generates a second three-dimensional frame of a predetermined size surrounding a person's body, and generates a first three-dimensional frame. The 3D frame and the second 3D frame are combined to generate a 3D frame surrounding the person, and the 3D coordinates of this 3D frame are transformed into the 2D coordinates on the 2D plane of the captured image. The server device 3 calculates the size of the portion corresponding to the first three-dimensional frame in the planar frame converted into the two-dimensional coordinates, and calculates the size of the planar frame based on the detection result of the human head from the photographed image including distortion. A size comparison is performed, and the height of the second three-dimensional frame is adjusted based on the comparison result. As a result, the server device 3 can be expected to adjust the second stereoscopic frame generated as a predetermined size to a size suitable for the height or posture of the detected person.

Further, in the server device 3 according to the present embodiment, the size of the portion corresponding to the first three-dimensional frame in the two-dimensional plane frame converted from the three-dimensional frame is determined according to the detection result of the human head from the captured image. If the size is larger than the size of the base frame, the height of the second solid frame in the three-dimensional virtual space is reduced. As a result, the server device 3 can be expected to accurately adjust the second stereoscopic frame generated as a predetermined size to a size suitable for the height or posture of the detected person.

In addition, the server device 3 according to the present embodiment uses the YOLO algorithm to detect an object appearing in a photographed image including distortion. The YOLO algorithm has a proven track record as an algorithm for detecting an object from an image with high accuracy, and can attach a bounding box to the detected object. It is an algorithm suitable for the processing to be performed. However, the server device 3 may use an algorithm other than YOLO to detect the object from the captured image.

Further, in the information processing system according to the present embodiment, the server device 3 acquires a photographed image including distortion (distorted image) photographed by the camera 1 through a wide-angle lens, and the object (person) photographed in the acquired photographed image is obtained. is generated and displayed on the display unit 54 of the terminal device 5 so as to be superimposed on the generated stereoscopic frame captured image. As a result, the information processing system can be expected to superimpose a bounding box suitable for the distorted captured image and present the existence of the target object captured in the captured image to the user.

Further, the information processing system according to the present embodiment extracts a partial image including the target object from the captured image including distortion based on the generated three-dimensional frame. As a result, it can be expected that processing such as face recognition or action recognition can be performed with high accuracy based on the extracted partial images.

Further, the three-dimensional frame generated by the information processing system according to the present embodiment includes a first frame surrounding the person's head and a body of the person, which is detected as an object in the photographed image. and a second frame surrounding the . By generating such a three-dimensional frame including a plurality of frames, it becomes possible to extract and use an image suitable for processing such as face recognition or action recognition.

Further, in the information processing system according to the present embodiment, the server device 3 generates a plane frame on a two-dimensional plane of the photographed image based on the solid bounding box generated in the three-dimensional virtual space of the world coordinate system. A size comparison is made with a plane frame directly generated from the image by an algorithm such as YOLO. The server device 3 can estimate the height of the object from the ground, the position in the three-dimensional virtual space, etc. based on the comparison result.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the above-described meaning, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 camera 3 server device 5 terminal device 31 processing unit 31a image acquisition unit 31b human detection unit 31c coordinate transformation unit 31d bounding box generation unit 31e coordinate reverse transformation unit 31f bounding box adjustment unit 31g image superimposition unit 31h image transmission unit 32 storage unit 32a Server program 32b YOLO learning model 33 communication unit 51 processing unit 51a image reception unit 51b display processing unit 52 storage unit 52a program 53 communication unit 54 display unit 55 operation unit 98, 99 recording medium 101 bounding box N network

An image processing method according to one embodiment acquires a distorted image captured by a camera and includes distortion, detects an object appearing in the acquired distorted image, and calculates two-dimensional coordinates of the object in the distorted image. transforming the distorted image into two-dimensional coordinates in the distortion-removed image, and transforming the two-dimensional coordinates of the object in the distortion-removed image into three-dimensional coordinates in a camera coordinate system centered on the camera; transforming the three-dimensional coordinates of the object in the camera coordinate system into three-dimensional coordinates in the world coordinate system , generating a three-dimensional frame surrounding the object in the three-dimensional space of the world coordinate system; The three-dimensional coordinates of the frame are inversely transformed into two-dimensional coordinates in the distorted image, and the planar frame obtained by transforming the three-dimensional frame into two-dimensional coordinates is superimposed on the distorted image.

Claims (10)

Acquire a distorted image containing the distortion captured by the camera,
Detecting an object appearing in the acquired distorted image,
transforming the two-dimensional coordinates of the object in the distorted image into three-dimensional coordinates in a virtual three-dimensional space;
generating a three-dimensional frame surrounding the object in the three-dimensional space;
inversely transforming the generated three-dimensional coordinates of the three-dimensional frame into two-dimensional coordinates in the distorted image;
superimposing a plane frame obtained by transforming the three-dimensional frame into two-dimensional coordinates on the distorted image;
Image processing method. converting the two-dimensional coordinates of the object in the distorted image into two-dimensional coordinates in an image obtained by removing the distortion from the distorted image;
transforming the two-dimensional coordinates of the object in the image from which the distortion has been removed into three-dimensional coordinates in a camera coordinate system centered on the camera;
transforming the three-dimensional coordinates of the object in the camera coordinate system into three-dimensional coordinates in the world coordinate system;
generating a three-dimensional frame surrounding the object in the world coordinate system;
The image processing method according to claim 1. the object is a person,
detecting a person's head in the distorted image;
transforming the two-dimensional coordinates of the person's head into three-dimensional coordinates;
generating a first three-dimensional frame surrounding the person's head in the three-dimensional space;
generating a second three-dimensional frame surrounding the human body in the three-dimensional space;
combining the first three-dimensional frame and the second three-dimensional frame to generate a three-dimensional frame surrounding the person;
3. The image processing method according to claim 1 or 2. generating the first three-dimensional frame of a predetermined size surrounding the person's head;
generating the second three-dimensional frame of a predetermined size surrounding the person's body;
combining the first three-dimensional frame and the second three-dimensional frame to generate a three-dimensional frame surrounding the person;
inversely transforming the three-dimensional coordinates of the three-dimensional frame into two-dimensional coordinates in the distorted image;
calculating a first size, which is the size of a portion corresponding to the first three-dimensional frame, of a plane frame obtained by converting the three-dimensional frame into two-dimensional coordinates;
generating a planar frame surrounding the human head detected from the distorted image;
Calculate the second size, which is the size of the generated plane frame,
adjusting the height of the second three-dimensional frame based on the comparison result of the first size and the second size;
4. The image processing method according to claim 3. reducing the height of the second three-dimensional frame when the first size is larger than the second size;
5. The image processing method according to claim 4. estimating the position of the object based on a comparison result of the first size and the second size;
6. The image processing method according to claim 4 or 5. displaying the estimated position of the object on a display together with the distorted image on which the plane frame is superimposed;
7. The image processing method according to claim 6. Detecting the object in the distorted image with a YOLO (You Only Look Once) algorithm;
The image processing method according to any one of claims 1 to 7. to the computer,
Acquire a distorted image containing the distortion captured by the camera,
Detecting an object appearing in the acquired distorted image,
transforming the two-dimensional coordinates of the object in the distorted image into three-dimensional coordinates in a virtual three-dimensional space;
generating a three-dimensional frame surrounding the object in the three-dimensional space;
inversely transforming the generated three-dimensional coordinates of the three-dimensional frame into two-dimensional coordinates in the distorted image;
A computer program for superimposing a plane frame obtained by transforming the three-dimensional frame into two-dimensional coordinates on the distorted image. an acquisition unit that acquires a distorted image captured by a camera and containing distortion;
a detection unit that detects an object appearing in the acquired distorted image;
a transformation unit that transforms the two-dimensional coordinates of the object in the distorted image into three-dimensional coordinates in a virtual three-dimensional space;
a generation unit that generates a three-dimensional frame surrounding the object in the three-dimensional space;
an inverse transformation unit for inversely transforming the generated three-dimensional coordinates of the three-dimensional frame into two-dimensional coordinates in the distorted image; An image processing device comprising: a superimposing unit;

Images