JP2020201879A - Image processing apparatus and image processing program - Google Patents

Abstract

To use a fisheye image in machine learning with high recognition accuracy while considering distortion in the image.SOLUTION: An image processing apparatus comprises: an acquisition unit that acquires a fisheye image photographed with photographing means including a fisheye lens; a division unit that divides the fisheye image into a plurality of divided images in the circumferential direction; a conversion unit that converts the plurality of divided images into rectangular plane images; a composition unit that connects the plurality of plane images to create a rectangular composite plane image; and a first inference unit that infers an object captured in the composite plane image based on the composite plane image and a first inference model that is learned through machine learning related to the plane image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and an image processing program.

In recent years, research and development of machine learning (including inference) using captured images has been actively carried out. Further, conventionally, in addition to a general captured image, there is a fisheye image which is an image captured in a wide range using a fisheye lens. It would be beneficial in many ways if fisheye images could be used for machine learning.

Japanese Unexamined Patent Publication No. 2016-48834 Japanese Unexamined Patent Publication No. 2012-253723 JP-A-2010-217984 International Publication No. 2013/001941 Special Table 2014-519091 Japanese Unexamined Patent Publication No. 2012-226645 Japanese Unexamined Patent Publication No. 2012-23546

However, since the subject of the fisheye image is distorted differently depending on the position in the image, it is not easy to use it for machine learning with high recognition accuracy.

Therefore, the present invention has been made in view of the above circumstances, and an image processing device and an image processing program capable of using a fish-eye image for machine learning with high recognition accuracy in consideration of distortion in an image are provided. The purpose is to provide.

The image processing apparatus according to the first aspect of the present invention includes an acquisition unit that acquires a fisheye image captured by an imaging means including a fisheye lens, and a division unit that divides the fisheye image into a plurality of divided images in the circumferential direction. , A conversion unit that converts each of the plurality of divided images into a rectangular plane image, a composition unit that connects the plurality of the plane images to create a rectangular composite plane image, and a machine related to the composite plane image and the plane image. It includes a first inference model that has been learned by learning, and a first inference unit that infers an object in the composite plane image based on the first inference model.

Further, in the image processing apparatus, for example, when the compositing unit joins a plurality of the plane images to create a rectangular composite plane image, the ratio of the length to the width of the composite plane image is 1. A rectangular composite plane image is created by joining a plurality of the plane images so as to approach 1: 1.

Further, in the image processing apparatus, for example, when the fisheye image is divided into a plurality of divided images in the circumferential direction, the divided portion is based on statistical information about the moving object in the past fisheye image. The fisheye image is divided into a plurality of divided images in the circumferential direction so that the divided positions are not as large as the region.

Further, in the image processing apparatus, for example, based on the statistical information, the divided portion sets the fish-eye image in the circumferential direction by preventing the two regions in the fish-eye image having many moving objects from becoming divided positions. When dividing into two divided images, if the angle between the straight lines from the center of the fisheye image to the center of each of the two regions is less than 90 degrees, the two regions should be on the same side. The fish-eye image is divided in the circumferential direction by a single line passing through the center of the fish-eye image, and when the angle is 90 degrees or more, the fish so that the two regions are on opposite sides. The eye image is divided in the circumferential direction by a single line passing through the center of the fisheye image.

Further, in the image processing apparatus, for example, the divided portion is a region having many moving objects based on heat map information which is information obtained by classifying the regions in the fisheye image according to the appearance frequency of the moving objects based on the statistical information. The fish-eye image is divided into a plurality of divided images in the circumferential direction so that the divided positions are not so large.

Further, in the image processing device, for example, the image processing device has been learned by a central image extraction unit that extracts a central image including the center of the optical axis from the fisheye image, the central image, and machine learning about the central image. A second inference model and a second inference unit that infers the object shown in the central image based on the second inference model are further provided.

Further, in the image processing device, for example, the acquisition unit acquires the fisheye image photographed by the photographing means installed so as to photograph the area to be photographed vertically downward.

The image processing program according to the second aspect of the present invention divides the computer into a acquisition unit for acquiring a fisheye image captured by an imaging means provided with a fisheye lens, and the fisheye image into a plurality of divided images in the circumferential direction. A division unit, a conversion unit that converts each of the plurality of divided images into a rectangular plane image, a composition unit that connects the plurality of the plane images to create a rectangular composite plane image, the composite plane image, and a plane. It functions as a first inference model that has been learned by machine learning about an image, and as a first inference unit that infers an object appearing in the composite plane image based on the first inference model.

According to the first aspect and the second aspect of the present invention, the fisheye image can be used for machine learning with high recognition accuracy in consideration of the distortion in the image.

FIG. 1 is a block diagram showing a functional configuration of the image processing apparatus of the embodiment. FIG. 2 is a schematic view of an example of a fisheye image of the embodiment. FIG. 3 is a diagram showing an extraction region of a central image in the fisheye image example of the embodiment. FIG. 4 is a diagram showing an extraction region of the outer peripheral side image in the fisheye image example of the embodiment. FIG. 5 is a diagram showing a composite plane image ((a)) created based on the outer peripheral side image and an extracted central image ((b)) in the embodiment. FIG. 6 is a flowchart showing the first processing by the image processing apparatus of the embodiment. FIG. 7 is a flowchart showing details of the division position setting process in the first process of FIG. FIG. 8 is an explanatory diagram of an example of setting the division position of the outer peripheral side image when there is one person concentration area in the heat map in the embodiment. FIG. 9 is an explanatory diagram of a first setting example of the division position of the outer peripheral side image when there are two person concentration areas in the heat map in the embodiment. FIG. 10 is an explanatory diagram of a second setting example of the division position of the outer peripheral side image when there are two person concentration areas in the heat map in the embodiment. FIG. 11 is a flowchart showing a second process by the image processing apparatus of the embodiment.

Hereinafter, an image processing apparatus and an embodiment of an image processing program will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a functional configuration of the image processing device 1 of the embodiment. The image processing device 1 is a computer device that executes machine learning (including inference) using captured images. In this machine learning, for example, image processing is performed on a captured image obtained by a camera, and learning is performed by detecting a person or a face in the image. Further, as machine learning, for example, deep learning (deep learning) is used.

Deep learning is machine learning using a multi-layered neural network, and its effectiveness has been confirmed in many fields. For example, deep learning has achieved high recognition accuracy comparable to humans in image / voice recognition.

In deep learning, the characteristics of the identification target are automatically learned in the neural network by performing learning (supervised learning) using the teacher data regarding the identification target. Then, in deep learning, the identification target is identified (inferred) using a neural network that has learned the features.

For example, taking the detection of a person in an image as an inference process, in deep learning, a large number of images of the entire person are used as learning images for supervised learning to obtain the characteristics of the entire person in the image. Automatically learn from neural networks. Alternatively, taking face detection in an image as an example of inference processing, in deep learning, a large number of images showing a person's face are used as learning images for supervised learning, so that the face of the person in the image is captured. The features are automatically learned from the neural network.

In deep learning, by using a neural network whose features have been learned in this way, it is possible to generate a trained inference model capable of inference processing such as identifying an identification target in an image.

It would be beneficial in various respects if a fisheye image, which is an image taken over a wide area using a fisheye lens, could be used for machine learning such as deep learning. However, since the subject of the fisheye image is distorted differently depending on the position in the image, it is not easy to use it for machine learning with high recognition accuracy. In the following, a fisheye image taken by a fisheye camera (hereinafter, also simply referred to as “camera”) installed so as to shoot a shooting target area vertically downward will be described as an example.

In such a fisheye image, the distortion method (projection angle, size) of the subject differs depending on the position in the image, and therefore the recognition characteristics also differ, so that an error occurs in the recognition accuracy. Further, when this fisheye image is expanded in a plane, the subject is greatly stretched as it is closer to the center of the image, which further causes an error in recognition accuracy. Therefore, in the following, a technique capable of using a fisheye image for machine learning with high recognition accuracy in consideration of distortion in the image will be described.

As shown in FIG. 1, the image processing device 1 includes a processing unit 2, a storage unit 3, an input unit 4, a display unit 5, and a communication unit 6.

The processing unit 2 is composed of, for example, a CPU (Central Processing Unit), and has a functional configuration realized by executing an image processing program stored in the storage unit 3, such as an acquisition unit 21, a central image extraction unit 22, and an outer peripheral side. It includes an image extraction unit 23, a division unit 24, a conversion unit 25, a composition unit 26, a first inference unit 27, and a second inference unit 28. In the following, for the processing other than each unit 21 to 28 in the processing unit 2, the operating subject is referred to as "processing unit 2".

The acquisition unit 21 acquires a fisheye image captured by a camera 100 (photographing means) provided with a fisheye lens and stores it in the storage unit 3.

The central image extraction unit 22 extracts a central image including the center of the optical axis (for example, a rectangular central image) from the fisheye image. The outer peripheral side image extraction unit 23 extracts an outer peripheral side image existing on the outer peripheral side from a predetermined region including the center of the optical axis from the fisheye image.

The dividing unit 24 divides the outer peripheral side image into a plurality of divided images in the circumferential direction. For example, when the outer peripheral side image is divided into a plurality of divided images in the circumferential direction, the division unit 24 does not have a divided position as much as a region having many moving objects based on statistical information about moving objects in the past fisheye image. In this way, the outer peripheral side image is divided into a plurality of divided images in the circumferential direction.

Further, for example, the division unit 24 divides the outer peripheral side image into two divided images in the circumferential direction based on the statistical information so that the two regions having many moving objects in the fisheye image do not become the divided positions. At that time, if the angle between the straight lines from the center of the fish-eye image to the center of each of the two regions is less than 90 degrees, the outer peripheral side image is displayed as a fish-eye so that the two regions are on the same side. It is divided in the circumferential direction by a single line passing through the center of the image. Further, when the angle is 90 degrees or more, the dividing portion 24 divides the outer peripheral side image in the circumferential direction by a single line passing through the center of the fisheye image so that the two regions are on opposite sides. To do.

Further, for example, the division unit 24 does not have a division position as much as a region having many moving objects, based on heat map information which is information obtained by classifying the regions in the fisheye image based on the appearance frequency of moving objects based on statistical information. Then, the outer peripheral side image is divided into a plurality of divided images in the circumferential direction.

The conversion unit 25 converts each of the plurality of divided images into a rectangular flat image. When the compositing unit 26 joins a plurality of plane images to create a rectangular composite plane image, the ratio of the length to the width of the composite plane image approaches 1: 1 (that is, a plurality of splices). Of the methods of joining, the method of joining that has the closest ratio of vertical to horizontal length to 1: 1), and joins multiple flat images to create a rectangular composite flat image.

The first inference unit 27 infers an object appearing in the composite plane image based on the composite plane image and the first inference model learned by machine learning about the plane image.

The second inference unit 28 infers the object shown in the central image based on the central image and the second inference model learned by machine learning about the central image. Details of the processing of each part 21 to 28 will be described later with reference to FIGS. 2 and 2.

The storage unit 3 is composed of semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory (Flash Memory), HDD (Hard Disk Drive), and the like.

The storage unit 3 stores, for example, statistical information, heat map information, a first inference model, a second inference model, and the like. As described above, the statistical information is statistical information regarding moving objects in past fisheye images. The heat map information is information in which regions in the fisheye image are classified according to the appearance frequency of moving objects based on statistical information.

The first inference model is an inference model that has been learned by machine learning about a plane image. The second inference model is an inference model trained by machine learning on the central image. That is, in the image processing device 1, the first inference model for the plane image obtained by converting the outer peripheral side image and the central image are taken into consideration that the subject is distorted differently depending on the position in the image in the fisheye image. A second inference model for is prepared separately. This enables highly accurate object recognition.

The input unit 4 is a means for the user to input various information, and is composed of, for example, a keyboard, a mouse, a touch panel, and the like.

The display unit 5 is a means for displaying various information, and is composed of, for example, a liquid crystal display, a touch panel, or the like.

The communication unit 6 is a means for communicating with an external device such as a camera 100, and is composed of, for example, a communication interface or the like.

FIG. 2 is a schematic view of an example of a fisheye image of the embodiment. The fisheye image F1 in FIG. 2 is an image obtained by photographing the image target area vertically downward with the camera 100. In the fisheye image F1, the angle and size of a person appearing differ depending on the position in the image. Specifically, the closer to the center of the image, the larger the person appears to be taken from the top of the head, and the farther from the center of the image, the smaller the person appears to be taken from the horizontal direction. It is reflected.

Here, FIG. 3 is a diagram showing an extraction region of the central image in the fisheye image example of the embodiment. In the fisheye image F1 of FIG. 3, the region R1 is a region corresponding to the central image. That is, the central image extraction unit 22 extracts the image corresponding to the region R1 from the fisheye image F1 as the central image.

Further, FIG. 4 is a diagram showing an extraction region of the outer peripheral side image in the fisheye image example of the embodiment. In the fisheye image F1 of FIG. 4, the region in which the region R2 and the region R3 are combined (hereinafter, referred to as the region R23) is a region corresponding to the outer peripheral side image. That is, the outer peripheral side image extraction unit 23 extracts the image corresponding to the region R23 from the fisheye image F1 as the outer peripheral side image. The region R23 is the region excluding the central circle (a circle centered on the entire center point) inscribed in the region R1 from the fisheye image F1.

Further, the dividing unit 24 divides, for example, the outer peripheral side image corresponding to the area R23 into two divided images in the circumferential direction (a divided image corresponding to the area R2 and a divided image corresponding to the area R3).

Further, FIG. 5 is a diagram showing a composite plane image ((a)) created based on the outer peripheral side image and an extracted central image ((b)) in the embodiment. In FIG. 5A, the image of the region R3a is a plane image corresponding to the divided image of the region R3 of FIG. That is, the conversion unit 25 converts the divided image of the region R3 of FIG. 4 into a rectangular flat image of the region R3a of FIG. 5A by developing the plane.

Further, in FIG. 5A, the image of the region R2a is a plane image corresponding to the divided image of the region R2 of FIG. That is, the conversion unit 25 converts the divided image of the region R2 of FIG. 4 into a rectangular flat image of the region R2a of FIG. 5A by developing the plane.

Further, the compositing unit 26 creates a rectangular composite plane image by connecting the plane image of the region R2a and the plane image of the region R3a. At that time, the composite unit 26 creates the composite plane image so that the ratio of the length to the width of the composite plane image approaches 1: 1. By doing so, the recognition accuracy of the person when the composite plane image is used for machine learning is improved. The first inference unit 27 infers the object shown in the composite plane image based on the composite plane image of FIG. 5A and the first inference model.

Further, in FIG. 5B, the central image of the region R1 is the same as the central image of the region R1 of FIG. Then, the second inference unit 28 infers the object shown in the central image based on the central image and the second inference model of FIG. 5 (b).

Next, with reference to FIG. 6, the first processing by the image processing apparatus 1 of the embodiment will be described. FIG. 6 is a flowchart showing a first process by the image processing device 1 of the embodiment. This first process is a preprocess before performing the inference process.

In step S1, the processing unit 2 starts the image processing application stored in the storage unit 3.

Next, in step S2, the processing unit 2 determines whether or not the camera is connected to the camera 100, proceeds to step S3 in the case of Yes, and ends the processing in the case of No.

In step S3, the processing unit 2 acquires the functional information of the camera stored in the storage unit 3.

Next, in step S4, the processing unit 2 determines whether or not the connected camera 100 is a fisheye camera based on the function information of the camera acquired in step S3, and if Yes, proceeds to step S6. If No, the process proceeds to step S5.

In step S5, the processing unit 2 sets no correction for the image processing for the captured image to be acquired, and proceeds to step S10.

In step S6, the processing unit 2 determines whether the installation position of the camera 100 is the ceiling or the side surface based on the camera installation information stored in the storage unit 3, and if it is the ceiling, proceeds to step S8, and if it is the side surface. If so, the process proceeds to step S7.

In step S7, the processing unit 2 sets the plane expansion processing for the image processing for the fisheye image to be acquired, and proceeds to step S10.

In step S8, the processing unit 2 sets a division method (for example, division into a central image and two outer peripheral side images) for the fisheye image to be acquired.

Next, in step S9, the processing unit 2 executes the division position setting process. Here, FIG. 7 is a flowchart showing the details of the division position setting process in step S9 in the first process of FIG.

In step S21, the processing unit 2 determines whether or not the division position is set to the automatic setting, and if Yes, proceeds to step S22, and if No, proceeds to step S25.

In step S22, the processing unit 2 determines whether or not the set value is specified, and if Yes, the process proceeds to step S24, and if No, the process proceeds to step S23.

In step S23, the processing unit 2 sets the division position to an initial value (for example, if the outer peripheral side image is divided into two, the horizontal direction), and ends the processing.

In step S24, the processing unit 2 sets the division position to a designated value (value specified by the user) and ends the processing.

In step S25, the processing unit 2 sets each value such as the camera resolution and the number of frames.

Next, in step S26, the acquisition unit 21 of the processing unit 2 starts acquiring the fisheye image from the camera 100.

Next, in step S27, the processing unit 2 sets the counter value to “0”.

Next, the processing unit 2 determines whether or not the counter value is "3", and if Yes, the process proceeds to step S23, and if No, the process proceeds to step S29.

In step S29, the processing unit 2 starts timer measurement. Next, in step S30, the processing unit 2 determines whether or not there is a moving object (for example, the movement of a person) in the fisheye image, and if Yes, the process proceeds to step S31, and if No, the process proceeds to step S32. move on.

In step S31, the processing unit 2 reflects the heat map information stored in the storage unit 3.

In step S32, the processing unit 2 determines whether or not a time-out (predetermined value has elapsed) has occurred, and if Yes, the process proceeds to step S33, and if No, the process returns to step S30.

In step S33, the processing unit 2 increments (adds 1) the counter value. Next, in step S34, the processing unit 2 determines whether or not there is a red display area (person concentration area) in the heat map information, and if Yes, the process proceeds to step S35, and if No, the process returns to step S28. ..

In step S35, the processing unit 2 determines whether or not there are a plurality of red display areas (person concentration areas), and if Yes, the process proceeds to step S37, and if No, the process proceeds to step S36.

In step S36, the processing unit 2 sets the division position according to the setting procedure when there is one red display area (person concentration area). Here, FIG. 8 shows a setting example of the division position of the outer peripheral side image (the notation of the curved portion in the center of the image is omitted. The same applies to FIGS. 9 and 10) when there is one person concentration area in the heat map in the embodiment. It is explanatory drawing of.

As shown in FIG. 8A, it is assumed that the person concentration area in the heat map is the area H1. In that case, for example, as shown in FIG. 8B, a line perpendicular to the line L1 connecting the center point P1 of the region H1 and the overall center point C and passing through the overall center point C is divided into lines L2. Set as.

Returning to FIG. 7, in step S37, the processing unit 2 sets the division position according to the setting procedure when there are a plurality of red display areas (person concentration areas). Here, FIG. 9 is an explanatory diagram of a first setting example of the division position of the outer peripheral side image when there are two person concentration areas in the heat map in the embodiment.

As shown in FIG. 9A, it is assumed that there are two areas of concentration of people in the heat map, areas H2 and H3. In that case, for example, as shown in FIG. 9B, first, the line connecting the center point P2 of the region H2 and the overall center point C is defined as the line L3. Further, the line connecting the center point P3 of the region H3 and the overall center point C is defined as the line L4. Then, it is assumed that the angle between the line L3 and the line L4 is less than 90 degrees.

In that case, the line that bisects the angle between the lines L3 and L4 is defined as the line L5. Then, a line perpendicular to the line L5 and passing through the entire center point C is set as the line L6 at the division position. By doing so, the balance of the positions of a plurality of people in the image is improved, and the recognition accuracy of the person in machine learning is improved.

Further, FIG. 10 is an explanatory diagram of a second setting example of the division position of the outer peripheral side image when there are two person concentration areas in the heat map in the embodiment. As shown in FIG. 10A, it is assumed that there are two areas of concentration of people in the heat map, areas H4 and H5. In that case, for example, as shown in FIG. 10B, first, the line connecting the center point P4 of the region H4 and the overall center point C is defined as the line L7. Further, the line connecting the center point P5 of the region H5 and the overall center point C is defined as the line L8. Then, it is assumed that the angle between the line L7 and the line L8 is 90 degrees or more.

In that case, the line that bisects the angle between the lines L7 and L8 is set as the line L9 at the division position. By doing so, the balance of the positions of a plurality of people in the image is improved, and the recognition accuracy of the person in machine learning is improved.

Next, the second processing by the image processing apparatus 1 of the embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing a second process by the image processing apparatus of the embodiment. This second process is an inference process.

In step S41, the acquisition unit 21 acquires a fisheye image taken by a camera 100 including a fisheye lens.

Next, in step S42, the central image extraction unit 22 extracts a central image including the center of the optical axis from the fisheye image (FIG. 3). Next, in step S43, the outer peripheral side image extraction unit 23 extracts the outer peripheral side image existing on the outer peripheral side from the predetermined region including the center of the optical axis from the fisheye image (FIG. 4).

Next, in step S44, the dividing unit 24 divides the outer peripheral side image into a plurality of divided images in the circumferential direction (FIG. 4). Next, in step S45, the conversion unit 25 converts each of the plurality of divided images into a rectangular flat image (FIG. 5A).

Next, in step S46, the compositing unit 26 joins a plurality of plane images to create a rectangular composite plane image (FIG. 5A).

Next, in step S47, the first inference unit 27 infers the object shown in the composite plane image based on the composite plane image (FIG. 5A) and the first inference model.

Next, in step S48, the second inference unit 28 infers the object shown in the central image based on the central image (FIG. 5B) and the second inference model. In addition, step S47 and step S48 may be executed at the same time.

In this way, according to the image processing apparatus 1 of the embodiment, the fisheye image can be used for machine learning with high recognition accuracy in consideration of the distortion in the image. Specifically, the fisheye image is divided into a plurality of divided images in the circumferential direction and then converted into a rectangular plane image, and further, the plurality of plane images are joined to create a rectangular composite plane image. In addition, the vertical and horizontal lengths of the composite plane image should be as close as possible. Then, based on the composite plane image and the first inference model learned by machine learning about the plane image, the object reflected in the composite plane image can be inferred with high accuracy.

In addition, when dividing a fisheye image into a plurality of divided images in the circumferential direction, based on statistical information on moving objects in the past fisheye image, it is possible to prevent the division position from becoming a region with many moving objects. (FIGS. 8 to 10), the possibility that a person is cut off at an edge in a composite plane image (FIG. 5 (a)) can be reduced, and recognition accuracy in machine learning can be improved. For example, when there is an entrance / exit of a building or the like in a fisheye image and a person often passes through that part, it is possible to set the division position of the fisheye image while avoiding the area where the person often passes, which is effective. Is.

More specifically, when there are two concentrated areas of people in the fisheye image, for example, when the angle between the straight lines from the center of the fisheye image to the center of each of the two areas is less than 90 degrees. The fisheye image may be divided in the circumferential direction by a single line passing through the center of the fisheye image so that the two regions are on the same side. When the angle is 90 degrees or more, the fisheye image may be divided in the circumferential direction by a single line passing through the center of the fisheye image so that the two regions are on opposite sides. By doing so, the positions of the plurality of people in the image are well-balanced, and the recognition accuracy of the people in machine learning is improved.

In addition, when statistical information is used to divide a fisheye image, the processing can be simplified by using it as heat map information.

In addition, for the fisheye image, the central image is separately extracted, and the machine learning about the central part of the fisheye image is based on the central image and the second inference model learned by machine learning about the central image. The recognition accuracy of the person in is improved.

The above-mentioned processing by the image processing apparatus 1 is particularly effective for a fisheye image from a camera 100 installed so as to photograph a photographing target area vertically downward, but the processing is not limited thereto. For example, the present invention can be applied to a fisheye image from a camera 100 installed so as to capture an image target area in the horizontal direction. This is because distortion exists in the fisheye image regardless of the orientation of the camera 100.

The disclosed technique is not limited to the above-described embodiment, and can be variously modified and implemented without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as necessary, or may be combined as appropriate.

For example, in the present embodiment, the image processing device 1 has been described as being realized by one computer device, but the present invention is not limited to this, and the image processing device 1 may be realized by a plurality of computer devices.

Further, the machine learning used is not limited to deep learning and may be other than deep learning.

Further, in the present embodiment, the inference scene is particularly described among the learning and inference in machine learning, but the present invention is not limited to this, and the present invention can be applied to the learning scene as well.

Further, in FIGS. 9 and 10, the case where the person concentration area is two has been described, but the present invention is not limited to this, and even if the person concentration area is three or more, the person concentration area is ranked according to the degree of concentration. It can be dealt with in the same way by doing so.

Further, in the present embodiment, the central image is extracted from the fisheye image, but the present invention is not limited to this, and the fisheye image is divided in the circumferential direction and used for machine learning without extracting the central image. It may be.

Further, the division of the fisheye image in the circumferential direction is not limited to two divisions, and may be divided into three or more.

Further, the size of the central circular region not included in the regions R2 and R3 in FIG. 4 may be appropriately changed depending on the resolution of the fisheye image, the type of lens, and the like.

The present invention can be applied to, for example, surveillance cameras, spherical cameras, and the like.

1 ... Image processing device, 2 ... Processing unit, 3 ... Storage unit, 4 ... Input unit, 5 ... Display unit, 6 ... Communication unit, 21 ... Acquisition unit, 22 ... Central image extraction unit, 23 ... Outer peripheral side image extraction unit , 24 ... division part, 25 ... conversion part, 26 ... synthesis part, 27 ... first inference part, 28 ... second inference part, 100 ... camera

Claims (8)

An acquisition unit that acquires a fisheye image taken by a shooting means equipped with a fisheye lens,
A division portion that divides the fisheye image into a plurality of divided images in the circumferential direction,
A conversion unit that converts each of the plurality of divided images into a rectangular flat image,
A compositing unit that creates a rectangular composite plane image by connecting a plurality of the plane images.
An image processing device including the composite plane image, a first inference model learned by machine learning about the plane image, and a first inference unit that infers an object appearing in the composite plane image based on the first inference model. .. When the composite plane image is joined together to create a rectangular composite plane image, the composite unit has a plurality of the above so that the ratio of the length to the width of the composite plane image approaches 1: 1. The image processing apparatus according to claim 1, wherein the composite plane image of a rectangle is created by joining the plane images. When the fisheye image is divided into a plurality of divided images in the circumferential direction, the divided portion does not become a divided position as much as a region having many moving objects based on statistical information about moving objects in the past fisheye image. The image processing apparatus according to claim 1, wherein the fisheye image is divided into a plurality of divided images in the circumferential direction. Based on the statistical information, the divided portion is used to divide the fisheye image into two divided images in the circumferential direction so that the two regions having many moving objects in the fisheye image do not become the divided positions. ,
When the angle between the straight lines from the center of the fisheye image to the center of each of the two regions is less than 90 degrees, the fisheye image is displayed so that the two regions are on the same side. Divide in the circumferential direction by a single line passing through the center of the image
3. When the angle is 90 degrees or more, the fisheye image is divided in the circumferential direction by a single line passing through the center of the fisheye image so that the two regions are on opposite sides. The image processing apparatus according to. Based on the heat map information, which is information obtained by classifying the regions in the fisheye image according to the appearance frequency of moving objects based on the statistical information, the dividing portion is set so that the regions with more moving objects are not divided. The image processing apparatus according to claim 3, wherein the fisheye image is divided into a plurality of divided images in the circumferential direction. The image processing device is
A central image extraction unit that extracts a central image including the center of the optical axis from the fisheye image,
Claim 1 further includes a second inference unit that infers an object appearing in the central image based on the central image, a second inference model learned by machine learning about the central image, and the like. The image processing apparatus according to. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the fisheye image taken by the shooting means installed so as to shoot the image target area vertically downward. Computer,
An acquisition unit that acquires a fisheye image taken by a shooting means equipped with a fisheye lens,
A division portion that divides the fisheye image into a plurality of divided images in the circumferential direction,
A conversion unit that converts each of the plurality of divided images into a rectangular flat image,
A compositing unit that creates a rectangular composite plane image by connecting a plurality of the plane images.
To function as a first inference unit that infers an object appearing in the composite plane image based on the composite plane image and a first inference model learned by machine learning about the plane image. Image processing program.

Images