JP2024059428A - Signal processing device, signal processing method, and storage medium - Google Patents

Abstract

A signal processing device includes an image processing unit that sets a bounding box on a captured image corresponding to an object detected in the captured image and calculates the density of the object according to the degree of overlap between the bounding boxes.

Description

This technology relates to a signal processing device, a signal processing method, and a storage medium, and to a technology for calculating the density of objects based on a captured image.

Processes that detect people within the field of view from images captured by surveillance cameras and calculate their density are known to help avoid crowded conditions and to show popular areas in tourist spots.

The following Patent Document 1 describes an example of calculating crowd density from a captured image. Specifically, it discloses a technology that divides a captured image into multiple blocks and detects a crowd for each block.

JP 2017-068598 A

Block-based crowd detection has problems with localization, and while it can provide a rough grasp of the density of people in a captured image, it may not be usable for more detailed analysis.
It is also described that crowds are detected by applying background subtraction processing, but this method has the problem that it requires multiple captured images.

This technology was developed in light of these circumstances, and aims to estimate the density locally without using multiple captured images.

A signal processing device according to the present technology includes an image processing unit that sets a bounding box on a captured image corresponding to an object detected in the captured image, and calculates a density of the object according to a degree of overlap between the bounding boxes.
This makes it possible to calculate the density through simple processing using one captured image.

FIG. 1 is a diagram illustrating an example of the configuration of a congestion calculation system. FIG. 2 is a block diagram showing an example of the internal configuration of the camera device. This is an example of an image input to the AI image processing unit. 1 is an example of an image with a bounding box superimposed on it for a person. 1 is an example of an image on which key points for a person are superimposed. FIG. 1 is a functional block diagram of an AI image processing unit. 4 is an example of a captured image captured by a camera device. FIG. 13 is an explanatory diagram of linking a box centroid position and a keypoint centroid position; This is an example of a target person and an overlapping person. 1 is a calculation example of IoU. This is an example of a flowchart of processing executed by an AI image processing unit or the like to calculate the density. 13 is an example of a flowchart of a density calculation process. FIG. 2 is a functional block diagram of a management device. 13 is an example of an image for presenting congestion information to a user. This is an example of superimposing density information on a bounding box. This is an example in which information on the density is displayed in a circular, overlapping manner. FIG. 1 is a diagram illustrating an example of a congestion calculation system including a cloud server. This is a diagram for explaining each device that registers and downloads AI models and AI applications via the marketplace function provided in the cloud-side information processing device. 1 is a diagram for explaining a connection mode between a cloud-side information processing device and an edge-side information processing device. FIG. FIG. 2 is a functional block diagram of a cloud-side information processing device. FIG. 2 is a block diagram showing the software configuration of the camera. FIG. 1 is a block diagram showing an operating environment of a container when container technology is used. FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device. FIG. 11 is a diagram for explaining the flow of processing in other explanations. FIG. 13 is a diagram illustrating an example of a login screen for logging in to a marketplace. FIG. 13 is a diagram showing an example of a developer screen presented to each developer who uses the marketplace. FIG. 13 is a diagram showing an example of a user-oriented screen presented to an application user who uses a marketplace.

The embodiments will be described below in the following order.
1. Configuration of the congestion calculation system
2. Configuration of the camera device
<3. Functions of AI image processing unit>
4. About the likelihood of keypoints
5. Person detection based on undetected rate
<6. Processing example>
<7. Display Processing>
8. Modifications
<9. Examples of cloud applications>
<10. Functional overview of cloud-side information processing device>
<11. Deployment of AI models and AI applications>
<12. Hardware configuration of information processing device>
<13. Other>
<14. Marketplace screen example>
<15. Summary>
<16. This Technology>

1. Configuration of the congestion calculation system
The configuration of a congestion calculation system 1 according to this embodiment will be described with reference to the accompanying drawings.

The density calculation system 1 is configured, for example, with a management device 2 and multiple camera devices 3. Each camera device 3 executes a process to calculate the density in an image captured by an imaging operation.

The management device 2 is an information processing device that performs various processes for managing each camera device 3. In addition, the management device 2 performs processes such as identifying congested or uncrowded areas based on the density calculated for each camera device 3 and location information where the camera device 3 is installed.

These pieces of information are notified to users, etc., by the management device 2 as appropriate. For example, in an amusement park, etc., areas with high crowding can be identified as congested areas, and users can be informed of popular areas.

Alternatively, it can capture changes in density and detect an increase in density to notify the user that an event has started.

In addition, by identifying areas of high density, users can be notified of congested areas that should be avoided.

On the other hand, it is also possible to provide information to users by identifying areas with low congestion. For example, by identifying areas with low congestion, it is possible to recommend areas with short waiting times or areas that are easy to get around to users.

The management device 2 performs notification processing to provide such information to the user and display processing to display the information on the screen of a user terminal (not shown) held by the user.

The management device 2 and each camera device 3 are configured to be able to communicate data with each other via the communication network 4.

The management device 2 is configured as an information processing device equipped with a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc.

The management device 2 may be an information processing terminal used by a monitor, or may be a server device such as a fog server.

The camera device 3 is a surveillance camera or the like, and is installed in various locations at positions and angles adjusted so that people positioned within the viewing angle can be captured from above. Specifically, the camera device 3 is set so that the angle between the horizontal direction and the optical axis is about 30 degrees to about 50 degrees.
By installing the camera device 3 so as to obtain an image captured from a bird's-eye view to some extent, it is possible to prevent people P from being captured in an image overlapping each other even though they are far away from each other and are not crowded together, and to prevent an inappropriate density from being calculated.

The configuration of the camera device 3 will be described later.

Various configurations are possible for the communication network 4. For example, the communication network 4 may be the Internet, an intranet, an extranet, a LAN (Local Area Network), a CATV (Community Antenna TeleVision) communication network, a virtual private network, a telephone line network, a mobile communication network, a satellite communication network, or the like.
Various examples of transmission media are also conceivable for constituting all or part of the communication network 4. For example, wired media such as IEEE (Institute of Electrical and Electronics Engineers) 1394, USB (Universal Serial Bus), power line carrier, and telephone line, as well as wireless media such as infrared ray such as IrDA (Infrared Data Association), Bluetooth (registered trademark), 802.11 wireless, mobile phone network, satellite line, and terrestrial digital network can be used.

2. Configuration of the camera device
FIG. 2 is a block diagram showing an example of the internal configuration of the camera device 3.
As shown in the figure, the camera device 3 includes an imaging optical system 31, an optical system driving unit 32, an image sensor IS, a control unit 33, a memory unit 34, and a communication unit 35. The image sensor IS, the control unit 33, the memory unit 34, and the communication unit 35 are connected via a bus 36, and are capable of mutual data communication.

The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. The imaging optical system 31 guides light (incident light) from the subject and focuses it on the light receiving surface of the image sensor IS.

The optical system driving unit 32 collectively refers to the driving units for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 31. Specifically, the optical system driving unit 32 has actuators for driving the zoom lens, focus lens, and aperture mechanism, respectively, and driving circuits for the actuators.

The control unit 33 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the camera device 3 by the CPU executing various processes according to programs stored in the ROM or programs loaded into the RAM.

The control unit 33 also issues drive instructions to the optical system drive unit 32 to drive the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the optical system drive unit 32 moves the focus lens and zoom lens, opens and closes the aperture blades of the aperture mechanism, etc.

The control unit 33 also controls writing and reading of various data to and from the memory unit 34 .
The memory unit 34 is a non-volatile storage device such as a hard disk drive (HDD) or a flash memory device, and is used as a storage destination (recording destination) for image data output from the image sensor IS.

Furthermore, the control unit 33 performs various data communications with external devices via the communication unit 35. In this example, the communication unit 35 is configured to be capable of performing data communications with at least the management device 2 shown in FIG. 1.

The image sensor IS is configured as, for example, a CCD type or CMOS type image sensor, and is capable of capturing, for example, an RGB image. Note that the image sensor IS may also be an IR sensor that is sensitive to the IR (Infrared) light band.

The image sensor IS comprises an imaging unit 41, an image signal processing unit 42, an internal sensor control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I/F 46, each of which is capable of data communication with each other via a bus 47.

The imaging unit 41 includes a pixel array section in which pixels having photoelectric conversion elements such as photodiodes are arranged two-dimensionally, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel in the pixel array section, and is capable of outputting the electrical signals as an imaging image signal.

The readout circuit performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion, and also performs A/D (Analog/Digital) conversion.

The image signal processing unit 42 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc. on the captured image signal as digital data after A/D conversion processing.
In the pre-processing, clamping processing is performed to clamp the R, G, and B black levels of the captured image signal to a predetermined level, and correction processing between the R, G, and B color channels is performed. In the synchronization processing, a color separation processing is performed so that the image data for each pixel has all the R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, a demosaic processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, a resolution conversion processing is performed on the image data that has been subjected to various signal processing.
In the codec process, the image data that has been subjected to the above-mentioned various processes is encoded and a file is generated for recording or communication. In the codec process, a file can be generated in a format such as MPEG-2 (Moving Picture Experts Group) or H.264 as a moving image file format. It is also possible to generate a file in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or GIF (Graphics Interchange Format) as a still image file. In addition, when the image sensor IS is a distance measurement sensor, the image signal processing unit 42 calculates distance information about a subject based on two signals output from the image sensor IS as, for example, iToF (indirect Time of Flight), and outputs a distance image.

In the following explanation, the image data output from the image signal processing unit 42 may be simply referred to as the "captured image."

The sensor control unit 43 issues instructions to the imaging unit 41 and controls the execution of the imaging operation. Similarly, it also controls the execution of processing in the image signal processing unit 42.

The sensor control unit 43 also performs processing to output information such as density calculated by the AI image processing unit 44 as metadata for the captured image data. For example, the sensor control unit 43 generates data that conforms to the MIPI (Mobile Industry Processor Interface) standard, with information such as density stored in the payload area.

The AI image processing unit 44 performs image processing using AI (Artificial Intelligence) on the captured image. In the following description, image processing using AI will be referred to as "AI image processing." AI image processing is, for example, image recognition processing.

The AI image processing unit 44 is realized by a DSP (Digital Signal Processor).

In this embodiment, there are multiple types of AI image processing executed by the AI image processing unit 44. One is a process that takes a captured image as input and outputs the detection result of person P in the captured image. The detection result of person P is output, for example, as information on a bounding box BB.

The information about the bounding box BB may be output as coordinate information on the captured image, or an image in which the bounding box BB is superimposed on the captured image may be output.

An example of the detection results for person P is shown in Figures 3 and 4.

Figure 3 shows an image input to the AI image processing unit 44, which is an image captured by the imaging unit 41. As shown in the figure, a person P, who is a soccer player, is positioned within the angle of view of the camera device 3.

Figure 4 shows an image in which a bounding box BB is superimposed on a person P detected by the AI image processing unit 44. As shown in the figure, a rectangular bounding box BB is superimposed so as to surround the person P, who is a soccer player and is located within the angle of view of the camera device 3.

Another type of AI image processing performed by the AI image processing unit 44 is a process that takes a captured image as input and outputs information about the key points KP of a detected person P.

An example of a key point KP is shown in Figure 5.

Key points KP are set corresponding to the main joints of person P detected in the captured image. In this example, as shown in FIG. 5, a total of 13 locations can be set as key points KP: person P's head, both shoulders, both elbows, both hands, both waists, both knees, and both ankles. However, key points KP are not set in parts that cannot be detected in the captured image. For example, for person P whose head is the only part captured at the edge of the captured image, only key points KP for the head are set.

The key points KP shown in FIG. 5 are merely an example, and various configurations are possible, such as setting one key point KP at the navel area instead of setting one key point KP at each of the left and right waist areas.

The AI image processing unit 44 performs various calculation processes described below based on the information of the bounding box BB and the information of the key points KP obtained by the AI image processing. These calculation processes are processes for calculating information for calculating the density described above. Such calculation processes may be executed by the sensor control unit 43.

The AI image processing unit 44 may use separate AI models to perform the process of superimposing the bounding box BB (or the process of setting the bounding box BB without superimposing it) and the process of setting the key points KP, or may use a single AI model to perform each process.

The AI model is switched based on the processing of the control unit 33 of the camera device 3 or the sensor internal control unit 43. The AI model is also switched from among multiple AI models stored in the memory unit 45, for example.

In the case where the bounding box superimposition process and the key point setting process are performed on the captured image periodically or irregularly at relatively long intervals, such as once every few minutes or once every few hours, the AI model may be switched by receiving and deploying the AI model from the cloud-side information processing device for each process. By receiving the AI model from the cloud-side information processing device each time a switch is made, it is no longer necessary to store the AI model in the memory unit 45, and the capacity of the memory unit 45 can be reduced, resulting in a smaller size, lower power consumption, and reduced costs.

The memory unit 45 can be used as a so-called frame memory in which the captured image data (RAW image data) obtained by the image signal processing unit 42 and image data after synchronization processing are stored. The memory unit 45 can also be used for temporary storage of data used by the AI image processing unit 44 in the AI image processing process.

In addition, the memory unit 45 stores information on AI applications and AI models used by the AI image processing unit 44. Note that an AI application refers to an application that uses an AI model. For example, an application that calculates the density of a person P based on information on a bounding box BB obtained by AI image processing using an AI model and setting information on key points KP is an example of an AI application.

The information on the AI application and the AI model may be deployed in the memory unit 45 as a container or the like using a container technology described later, or may be deployed using a microservice technology. By deploying the AI model used for the AI image processing in the memory unit 45, it is possible to change the function type of the AI image processing, or to change to an AI model whose performance has been improved by re-learning.
As described above, in this embodiment, the explanation is based on examples of AI models and AI applications used for image recognition, but the present invention is not limited to these and may also target programs executed using AI technology.
In addition, if the capacity of the memory unit 45 is small, information on the AI application and AI model may be expanded into a memory outside the image sensor IS, such as the memory unit 34, as a container using container technology, and then only the AI model may be stored in the memory unit 45 within the image sensor IS via the communication I/F 46 described below.

The communication I/F 46 is an interface for communicating with the control unit 33, the memory unit 34, and the like that are external to the image sensor IS. The communication I/F 46 communicates to acquire from the outside the program executed by the image signal processing unit 42, the AI application and the AI model used by the AI image processing unit 44, and the like, and stores them in the memory unit 45 provided in the image sensor IS.
As a result, the AI model is stored in part of the memory unit 45 provided in the image sensor IS, making it available for use by the AI image processing unit 44.

The AI image processing unit 44 uses the AI application or AI model obtained in this way to perform a specified image recognition process, thereby recognizing the subject according to the purpose.

Log information such as density obtained by calculation processing using the bounding box BB and key point KP information obtained by AI image processing is output to the outside of the image sensor IS via the communication I/F 46.

That is, from the communication I/F 46 of the image sensor IS, not only the image data output from the image signal processing unit 42 but also information such as density calculated based on the recognition results of the AI image processing is output as metadata of the image data.
It should be noted that the communication I/F 46 of the image sensor IS can be made to output only either the image data or the density information.

For example, when using the re-learning function of the AI model described above, the captured image data used in the re-learning function is uploaded from the image sensor IS to the cloud-side information processing device via the communication I/F 46 and the communication unit 35.

In addition, when inference is performed using an AI model, congestion information as the inference result is output from the image sensor IS to another information processing device outside the camera device 3 via the communication I/F 46 and the communication unit 35.

<3. Functions of AI image processing unit>
The AI image processing unit 44 has various functions for calculating the density. Specifically, the AI image processing unit 44 has a center of gravity position calculation function F1, an Intersection of Union (IoU) calculation function F2, a non-detection rate calculation function F3, and a density calculation function F4 (see FIG. 6).

At least some of these functions may be provided in the image signal processing unit 42 or the sensor internal control unit 43.

The center of gravity calculation function F1 calculates the box center of gravity position BBP and the key point center of gravity position KBP for the person P detected in the captured image.

The box center of gravity position BBP is the center of gravity position of the bounding box BB. Specifically, the x coordinate of the box center of gravity position BBP is the average value of the x coordinate of the left end and the x coordinate of the right end of the bounding box BB, and the y coordinate of the box center of gravity position BBP is the average value of the y coordinate of the top end and the y coordinate of the bottom end of the bounding box BB.

The keypoint center of gravity KBP is the center of gravity of the keypoints KP detected as belonging to the same person. Specifically, the x coordinate of the keypoint center of gravity KBP is the average value of the x coordinates of each keypoint KP for the same person, and the y coordinate of the keypoint center of gravity KBP is the average value of the y coordinates of each keypoint KP for the same person.

In addition, undetected keypoints KP may be taken into consideration when determining the keypoint center of gravity position KBP. For example, when detecting keypoints KP using an AI model, the AI image processing unit 44 performs processing to estimate the positions of undetected keypoints KP for the person P. The center of gravity position calculation function F1 may calculate the x and y coordinates for the keypoint center of gravity position KBP, including the estimated undetected keypoints KP.

The center of gravity position calculation function F1 performs processing to associate a bounding box BB and key points KP for the same person P.
7, a single captured image may contain multiple persons P. In such a case, a process is performed to associate a bounding box BB with a key point KP for each person P detected by image recognition.

Specifically, for each bounding box BB set by AI image processing, the key point center of gravity position KBP that is closest in coordinates to the box center of gravity position BBP is identified and a matching process is performed (see Figure 8, for example).
In addition, when associating, by excluding key point centroid positions KBP that have already been associated, it is possible to prevent multiple bounding boxes BB from being linked to one key point centroid position KBP.

Even if the keypoint center of gravity position KBP is closest to the box center of gravity position BBP, if the keypoint center of gravity position KBP is located outside the area of the bounding box BB, the bounding box BB will not be associated with the keypoint KP.

When matching is performed in this manner, there is a possibility that an independent bounding box BB that is not matched with a key point KP may occur. Density calculation is not performed for the independent bounding box BB. This reduces the processing load.

Note that if both the bounding box BB and information about key points KP for a detected person P can be obtained through AI image processing using a single AI model, matching processing is not necessary.

The center of gravity position calculation function F1 associates a bounding box BB with a key point KP for each person P detected in the captured image. In addition, for a person P with only one of a bounding box BB and a key point KP associated with it, the density is not calculated.

The IoU calculation function F2 performs processing to quantify the degree of overlap between the bounding box BB associated with the person P being processed and the bounding boxes BB associated with other people P.

Here, the person P to be processed is defined as the target person Pt, and the bounding box BB for the target person Pt is defined as the target bounding box BBt.

Furthermore, a bounding box BB that partially overlaps with the target bounding box BBt is referred to as an overlapping bounding box BBd, and a person P to which the overlapping bounding box BBd is linked is referred to as an overlapping person Pd.

Figure 9 is an example showing a target person Pt and two overlapping people Pd1 and Pd2. Note that in Figure 9, the captured image is omitted, and only the bounding boxes BB and key points KP for the target person Pt and overlapping people Pd1 and Pd2 are shown. Specifically, only the target bounding box BBt for the target person Pt, the overlapping bounding box BBd1 for the overlapping person Pd1, and the overlapping bounding box BBd2 for the overlapping person Pd2, and their respective key points KP are shown.

The target bounding box BBt of the target person Pt is shown with a solid line, and the overlapping bounding boxes BBd1 and BBd2 of the overlapping people Pd1 and Pd2 are shown with dashed lines of different line widths.

The IoU calculation function F2 calculates a basic IoU for each pair of the target bounding box BBt and the overlapping bounding boxes BBd1 and BBd2, and calculates the IoU for the target person Pt from the basic IoU.

Specifically, the basic IoU(BBt, BBd1) for the pair of the target bounding box BBt and the overlapping bounding box BBd1 is explained with reference to FIG. 10.

The basic IoU(BBt, BBd1) is calculated as the common area CA1 of the target bounding box BBt and the overlapping bounding box BBd1 divided by the logical sum area DA1 of the target bounding box BBt and the overlapping bounding box BBd1. The larger the basic IoU value, the greater the degree of overlap between the two bounding boxes BB.

The basic IoU is calculated for each pair of a target bounding box BBt and an overlapping bounding box BBd. That is, in the state shown in FIG. 9, basic IoU(BBt, BBd1) and basic IoU(BBt, BBd2) are calculated.

The IoU calculation function F2 calculates the IoU for the target person Pt from the basic IoU. The IoU for the target person Pt is taken as the average value of the basic IoU for the target person Pt. That is, the IoU for the target person Pt in the example shown in FIG. 9 is taken as the average value of the basic IoU (BBt, BBd1) and the basic IoU (BBt, BBd2).

The undetected rate calculation function F3 calculates the undetected rate of key points KP for person P. It is estimated that the undetected rate is higher the more densely people P are located.

Specifically, the undetected rate for the target person Pt shown in FIG. 9 is calculated as the number of undetected key points KP for the target person Pt and overlapping persons Pd1 and Pd2 divided by the total number of key points KP for the target person Pt and overlapping persons Pd1 and Pd2.

The total number of key points KP per person P is 13, so in the example shown in Figure 9, the total number of key points KP for the target person Pt and overlapping persons Pd1 and Pd2 is 39.

In addition, the number of undetected key points KP of the target person Pt is set to 0, the number of undetected key points KP of the overlapping person Pd1 is set to 3, and the number of undetected key points KP of the overlapping person Pd2 is set to 4, so the total number of undetected key points KP is 7. In addition, in Fig. 9, the undetected key points KP are indicated by white circles.
Therefore, the non-detection rate of the key points KP of the target person Pt is calculated by dividing "7" by "39".

The undetected rate calculation function F3 may treat a person P whose bounding box BB is detected as overlapping the edge of the captured image, or a person P whose undetected key point KP is estimated to be located outside the frame of the captured image, as an out-of-view person, and calculate the undetected rate of the key points KP of the target person Pt by excluding the key points KP for that person P.

For example, if the overlapping person Pd2 of the target person Pt and overlapping persons Pd1 and Pd2 shown in Figure 9 is an incomplete person, the total number of key points KP for the target person Pt and overlapping person Pd1 is 26, the number of undetected key points KP is 0 and 3, respectively, and the undetection rate is calculated as the value obtained by dividing "3" by "26".

The density calculation function F4 calculates the density of the target person Pt based on the IoU for the target person Pt calculated by the IoU calculation function F2 and the undetected rate for the target person Pt calculated by the undetected rate calculation function F3.

The density calculation function F4 also calculates the density for each person P by sequentially selecting the people P detected in the captured image as target people Pt. At this time, it is possible not to select people who are out of view as target people Pt.

There are several possible methods for calculating the density. For example, the density may be calculated by averaging the IoU and the undetected rate, or the density may be calculated by multiplying the IoU and the undetected rate.

Also, taking into consideration the difference in importance between the IoU and the undetected rate, the density may be calculated by multiplying each of them by a coefficient and using the result as a correction value. Specifically, when the IoU value is more important for the density, the IoU value may be multiplied by a coefficient higher than the undetected rate so as to have a greater impact on the density.

4. About the likelihood of keypoints
The AI image processing unit 44 may set key points KP of the person P by AI image processing and output the likelihood for each key point KP.

In that case, the non-detection rate for the keypoint KP may be calculated by taking into account the likelihood of each keypoint KP. The higher the likelihood value of a keypoint KP, the more likely the keypoint KP is. In other words, a keypoint KP with a low likelihood is likely to be a keypoint KP that was detected by misidentifying an object other than person P as a joint of person P, etc.

If the likelihood of a keypoint KP is lower than a predetermined likelihood threshold Th1, the keypoint KP is treated as undetected.
This makes it possible to prevent the non-detection rate from being calculated low and the density from being calculated low due to uncertain key points KP.

When the likelihood is output as a numerical value from "0" to "100", the likelihood threshold Th1 is set to, for example, "20" or "30".

5. Person detection based on undetected rate
If the undetected rate is extremely high, the person P may be treated as not having been detected.
For example, when only the right hand of person P is detected, the non-detection rate is 12 out of 13. When the non-detection rate is high like this, it is difficult to say that person P has been appropriately detected.

Therefore, if the undetected rate of the key point KP is equal to or greater than the undetected rate threshold Th2, the person P may be treated as an undetected person. In other words, if the undetected rate of the key point KP is less than the undetected rate threshold Th2, it may be determined that the person P has been detected. In other words, if the undetected rate is equal to or greater than the undetected rate threshold Th2, the key point KP may be excluded, i.e., the density may be calculated excluding the person P.

If the undetected rate of a key point KP is equal to or greater than the undetected rate threshold Th2, the person P will not be selected as the target person Pt or overlapping person Pd described above.

The non-detection rate is, for example, "0.1" or "0.2."

<6. Processing example>
An example of the flow of processing executed by the AI image processing unit 44 of the image sensor IS is shown in Figures 11 and 12. Note that each process is executed by the AI image processing unit 44, but as described above, at least a part of the process may be executed by the image signal processing unit 42 or the sensor internal control unit 43. Also, a part of the process may be executed by the control unit 33 outside the image sensor IS.

For example, the process of setting the bounding box BB and the process of setting the key points KP may be performed within the image sensor IS of the camera device 3, and the process of calculating the density based on this information may be performed outside the image sensor IS.

In addition, at least a portion of each process may be executed outside the camera device 3, for example, by the management device 2.

The AI image processing unit 44 executes pre-processing in step S101 shown in FIG. 11. Pre-processing is, for example, processing for normalizing and resizing the captured image output from the imaging unit 41. This processing can also be said to be processing for preparing the captured image so that it is suitable as input data to be input to an AI model used in AI image processing.

In step S102, the AI image processing unit 44 performs inference processing as AI image processing. Through this processing, a person P is detected from the preprocessed captured image input to the AI model, and a bounding box BB and key points KP for the person P are set.

As mentioned above, the inference process in step S102 may use multiple AI models or only one AI model.

In step S103, the AI image processing unit 44 performs post-processing. The post-processing is, for example, a process of linking the bounding box BB and key points KP for each person P.

In step S104, the AI image processing unit 44 performs a process of selecting one of the detected persons P as a target person Pt. Note that, as described above, only persons P whose undetected rate of key points KP is less than the undetected rate threshold Th2 may be treated as detected persons P.

Next, in step S105, the AI image processing unit 44 executes a density calculation process for the target person Pt.

An example of the density calculation process is shown in FIG.
In step S201, the AI image processing unit 44 identifies an overlapping person Pd for the target person Pt and obtains an overlapping bounding box BBd of the overlapping person Pd.

Next, in step S202, the AI image processing unit 44 calculates the basic IoU for each pair of the target bounding box BBt and the overlapping bounding box BBd, and calculates the IoU for the target person Pt based on the basic IoU.

In step S203, the AI image processing unit 44 acquires key points KP for each overlapping person Pd. At this time, the key points KP may be acquired based on the likelihood output for each key point KP as described above. Specifically, only key points KP whose likelihood is equal to or greater than the likelihood threshold value Th1 may be acquired.

In step S204, the AI image processing unit 44 performs a process of excluding out-of-view persons. In the following step S205, the AI image processing unit 44 calculates a non-detection rate based on the key points KP detected for the target person Pt and the overlapping person Pd with the out-of-view persons excluded.

In step S206, the AI image processing unit 44 calculates the density based on the IoU and non-detection rate for the target person Pt.

Returning to the explanation of FIG.
After calculating the density of the target person Pt, in step S106, the AI image processing unit 44 outputs the density as a log via the communication I/F 46. Note that the output log may include, in addition to the density information, coordinate information of the detected person P, information on the undetected rate, information on IoU, the number of overlapping bounding boxes BBd, and the like.

Then, in step S107, the AI image processing unit 44 determines whether or not there is a person whose density has not been calculated. If it is determined that there is a person P whose density has not been calculated, the AI image processing unit 44 selects a new person P as a target person Pt in step S104, and executes the density calculation process in step S105.

On the other hand, if it is determined that there is no person P for which the density has not been calculated, that is, if it is determined that the calculation of the density for all people P has been completed, the AI image processing unit 44 terminates the series of processes shown in Figure 11.

<7. Display Processing>
Information on the density of each person P captured in the captured image is output to a management device 2, which is an information processing device other than the camera device 3, and a notification process is executed in the management device 2.
Moreover, the management device 2 may perform display processing in an appropriate manner to present information about the congestion level on the user terminal.

In addition, a display process may be performed to display the congestion information using a display unit such as a monitor provided in the camera device 3, instead of the management device 2. In this case, the display process is performed by the control unit 33 of the camera device 3.

Here, we will explain an example in which the display process for density is performed by the management device 2 shown in Figure 1.

When the CPU of the management device 2 executes a specific program, the management device 2 has a communication function F11, a display control function F12, and the like, as shown in FIG. 13.

The communication function F11 receives captured images and density information from each camera device 3. The density information can be acquired simultaneously with the captured image data, for example, by storing it in an extension area when receiving captured image data in a data structure conforming to the MIPI standard.

In addition to the density information, information on the non-detection rate of key points KP and IoU information may also be obtained from the camera device 3.

The display control function F12 performs display processing to present information about the density to the user. An example is shown in FIG. 14.

Figure 14 shows an image received from the camera device 3 with density information superimposed on it. By superimposing an image corresponding to the density of each person P appearing in the captured image, an image such as that shown in Figure 14 is presented to the user. Such an image may be displayed on a user terminal such as a smartphone carried by the user by executing the display control function F12 by the management device 2.

Figure 15 shows density information superimposed on a bounding box BB for one person P displayed in the captured image shown in Figure 14. Note that the bounding box BB in Figure 15 is shown for explanatory purposes, and does not need to be superimposed on the captured image when presenting density information to the user.

The density information is visualized by defining a two-dimensional Gaussian distribution within the bounding box BB, generating a heat map image according to the Gaussian distribution, and superimposing it on the captured image.

The heat map image has a maximum value at the box center of gravity BBP of the bounding box BB, and the values decrease the further away (in pixels) the position is from the box center of gravity BBP, until it reaches "0" at the outer edge of the bounding box BB.

The maximum value of the Gaussian distribution at the box center of gravity position BBP corresponds to the density of the person P. In other words, the higher the density of the person P, the higher the maximum value of the Gaussian distribution at the bounding box BB of the person P.

A heat map image based on a Gaussian distribution is an image in which the higher the numerical value, the darker the specified color (e.g., yellow) becomes.

In the example shown in FIG. 15, the Gaussian distribution is set so that the numerical value is "0" at the outer edge of the bounding box BB, but the numerical value may be uniquely determined by the distance from the box center of gravity position BBP. That is, if the bounding box BB is rectangular, the numerical value may be "0" at the long side of the bounding box BB, and may be greater than "0" at the short side.

In the example of Figure 15, the box center of gravity position BBP of the bounding box BB is set to the maximum value of the Gaussian distribution, but the keypoint center of gravity position KBP of person P may also be set to the maximum value of the Gaussian distribution.

Furthermore, in the example of Figure 15, an example using a Gaussian distribution is described, but it may also be set so that the box center of gravity position BBP is set as the maximum value and the value decreases linearly as the distance from there increases.

In the example of Figure 15, a heat map was generated to visualize the density of person P using a bounding box BB, but a circular image rather than a rectangular image may be generated as the heat map image by setting the box center of gravity position BBP of person P to a maximum value and setting the numerical value to decrease depending on the distance from there (see Figure 16).

8. Modifications
In the above example, the density is calculated using both the bounding boxes BB and the key points KP, but the density may be calculated using only the degree of overlap between the bounding boxes BB.

Specifically, first, one person P is selected as the target person Pt. Next, a target bounding box BBt of the target person Pt is identified, and an overlapping bounding box BBd, which is a region that partially overlaps with the target bounding box BBt, is identified.

Then, a basic IoU is calculated for each pair of the target bounding box BBt and the overlapping bounding box BBd. This basic IoU is used to calculate the density for the target person Pt by taking the average value, for example. Therefore, the basic IoU can be regarded as the basic density.

This allows the density to be calculated using a simpler method than the above-mentioned method for calculating the non-detection rate of key points KP, thereby reducing the processing burden on the camera device 3 and the image sensor IS.

In the above example, a person is given as an example of the object, but the object may be something other than a person. For example, the density may be calculated for a living thing such as an animal, or for an inorganic object such as a vehicle.

<9. Examples of cloud applications>
Presentation of information on the density by the density calculation system 1 may be realized by a function of a cloud application by a cloud server 5 provided other than the management device 2. However, the management device 2 may have the function of the cloud server 5.

An example of a configuration in which the density calculation system 1 has a cloud server 5 is shown in Figure 17.

The information processing system shown in FIG. 17 functions as a system that calculates and presents the density of a person P as a subject based on a captured image.

The density calculation system 1 is configured such that a cloud server 5, a user terminal 6, and a management device 2 are connected to each other via a communication network 7 so that they can communicate with each other.

The management device 2 functions as a fog server.
It is assumed that the management device 2 as a fog server is arranged for each monitored object, for example, in an amusement park or a store to be monitored together with each camera device 3. By providing a management device 2 for each monitored location in this manner, the cloud server 5 does not need to directly receive transmission data from the multiple camera devices 3 at the monitored object, thereby reducing the processing load of the cloud server 5.

Note that when there are multiple stores to be monitored and all of these stores belong to the same chain, it is possible to provide one management device 2 for each of these multiple stores, rather than providing one for each store. In other words, the management device 2 is not limited to being provided for each monitored object, and it is also possible to provide one management device 2 for multiple monitored objects.

Note that if the cloud server 5 or each camera device 3 has the processing capacity to perform the functions of the management device 2, the management device 2 may be omitted from the density calculation system 1, and each camera device 3 may be directly connected to the communication network 7 so that the cloud server 5 directly receives the data transmitted from the multiple camera devices 3.

The management device 2 may be an on-premise server.

The cloud server 5 is configured as an information processing device equipped with a microcomputer having a CPU, ROM, RAM, etc., and provides advanced application functions using the result information of the AI image processing executed in the camera device 3 as an edge-side information processing device (e.g., information such as density information transmitted in conjunction with the captured image and the rate of non-detection of key points KP).

The user terminal 6 is an information processing device that is expected to be used by a user who is the recipient of various services provided by the cloud server 5.

In the following description, the above various devices can be broadly classified into cloud-side information processing devices and edge-side information processing devices.
The cloud-side information processing device corresponds to a server device such as the cloud server 5, and is a group of devices that provide cloud services that are expected to be used by multiple users.

The edge-side information processing device corresponds to the camera device 3 and the management device 2, and can be regarded as a group of devices arranged in an environment prepared by a user who uses a cloud service.
The management device 2 may be a cloud-side information processing device.

Here, various methods are conceivable for registering each application function having a function for presenting congestion information in cloud server 5, which is an information processing device on the cloud side.
An example of this will be described with reference to FIG.
2, the management device 2 may be included in the configuration. In this case, the management device 2 may take on part of the functions of the edge device.

The above-mentioned cloud server 5 is an information processing device that constitutes the cloud environment.
The camera device 3 is an information processing device that constitutes the edge side environment.

In addition to the camera device 3, the image sensor IS can also be considered as an information processing device that constitutes the edge-side environment. In other words, the image sensor IS, which is another edge-side information processing device, can be considered to be mounted inside the camera device 3, which is an edge-side information processing device.

In addition, user terminals 6 used by users who utilize various services provided by the cloud-side information processing device include an application developer terminal 6A used by a user who develops applications used in AI image processing, an application user terminal 6B used by a user who uses applications, and an AI model developer terminal 6C used by a user who develops AI models used in AI image processing.
Of course, the application developer terminal 6A may also be used by a user developing applications that do not use AI image processing.

The cloud-side information processing device is provided with a learning dataset for AI learning and an AI model that serves as the basis for development. A user developing an AI model communicates with the cloud-side information processing device using an AI model developer terminal 6C and downloads these learning datasets and AI models. At this time, the learning dataset may be provided for a fee. For example, the AI model developer may purchase the learning dataset in a state in which it is possible to purchase various functions and materials registered in a marketplace (electronic marketplace) by registering personal information in the marketplace provided as a function on the cloud side.

After developing an AI model using the learning dataset, the AI model developer uses the AI model developer terminal 6C to register the developed AI model in the marketplace. In this way, an incentive may be paid to the AI model developer when the AI model is downloaded.

In addition, a user who develops an application downloads an AI model from the marketplace using the application developer terminal 6A and develops an AI application using the AI model. At this time, as described above, an incentive may be paid to the AI model developer.

The application development user uses the application developer terminal 6A to register the developed AI application in the marketplace. This may allow an incentive to be paid to the user who developed the AI application when the AI application is downloaded.

A user who uses an AI application uses the application user terminal 6B to perform an operation for deploying the AI application and the AI model from the marketplace to the camera device 3 as an edge-side information processing device that the user manages. At this time, an incentive may be paid to the AI model developer.
This makes it possible to perform AI image processing using AI applications and AI models in the camera device 3, making it possible not only to capture images but also to perform processes such as detecting customers or vehicles through AI image processing, or calculating and presenting the density of people P within the angle of view of each camera device 3 as described above.

Here, deployment of an AI application and an AI model refers to installing the AI application and the AI model in a target (device) acting as an execution subject so that the target can use the AI application and the AI model, in other words, so that at least a portion of the program as the AI application can be executed.

In addition, the camera device 3 may be capable of using AI image processing to not only calculate the density from the image captured by the camera device 3 but also extract attribute information of customers visiting the store.
These pieces of attribute information are transmitted from the camera device 3 to the information processing device on the cloud side via the communication network 7 .

Cloud applications are deployed on the information processing device on the cloud side, and each user can use the cloud applications via the communication network 7. The cloud applications include an application for visually displaying the degree of congestion, and an application for analyzing the movement of customers using attribute information and captured images of customers. Such cloud applications are uploaded by application development users, etc.

By using an application for visually displaying the density using the application user terminal 6B, an image such as that shown in FIG. 14 is presented to the user. In addition, by using a cloud application for flow analysis, it is possible to perform an analysis of the flow of customers visiting one's own store and view the analysis results. Viewing the analysis results is performed by displaying the flow of customers visiting the store graphically on a map of the store.

In the cloud-side marketplace, AI models optimized for each user may be registered. For example, images captured by a camera device 3 installed in a store managed by a user are appropriately uploaded to an information processing device on the cloud side and stored.

In the cloud information processing device, a re-learning process of the AI model is performed each time a certain number of uploaded captured images are accumulated, and a process of updating the AI model and re-registering it in the marketplace is executed. In the learning process, learning is performed for each installation angle of the camera device 3 that captured the image, and an AI model optimized for each installation angle of the camera device 3 is generated. This makes it possible to perform AI image processing using an AI model optimized for each camera device 3.
In addition, the re-learning process of the AI model may be made available to users as an option on the marketplace, for example.

As an example other than the installation angle of the camera device 3, for example, an AI model retrained using a dark image from a camera device 3 placed inside a store can be deployed to the camera device 3 to improve the recognition rate of image processing for images captured in dark places. Also, an AI model retrained using a bright image from a camera device 3 placed outside the store can be deployed to the camera device 3 to improve the recognition rate of image processing for images captured in bright places.
In other words, the application user can always obtain optimized processing result information by redeploying the updated AI model to the camera device 3 again.

In addition, in the cloud marketplace, AI models optimized for each camera may be registered. For example, an AI model that is applied to a camera device 3 capable of acquiring RGB images, an AI model that is applied to a camera device 3 equipped with a distance measuring sensor that generates a distance image, etc. may be considered.
In addition, an AI model trained using images of a vehicle or captured in a bright environment as an AI model to be used by the camera device 3 during daylight hours, and an AI model trained using images captured in a dark environment as an AI model to be used by the camera device 3 during dark hours may each be registered in the marketplace.
It is desirable to appropriately update these AI models to AI models with improved recognition rates through re-learning processes.

In addition, if personal information is included in the information (such as captured images) uploaded from the camera device 3 to the information processing device on the cloud side, the data may be uploaded with the privacy information deleted from the standpoint of privacy protection, or the data with the privacy information deleted may be made available to AI model development users and application development users.

<10. Functional overview of cloud-side information processing device>
In this embodiment, the service provided by the server device is assumed to be a service in which a user as a customer can select a function type for the AI image processing of each camera device 3. The selection of the function type can be said to be the setting of the above-mentioned purpose. In addition, for example, an image recognition function and an image detection function may be selected, or a more detailed type may be selected to perform an image recognition function or an image detection function for a specific subject.
For example, as a business model, the service provider sells the camera device 3 and the management device 2 having an AI-based image recognition function to the user, and has the user install the camera device 3 and the management device 2 in the location to be monitored. Then, the service provider develops a service that provides the above-mentioned analysis information to the user.

At this time, since each customer has a different purpose (purpose) for the system, such as calculating density, monitoring stores, or monitoring traffic, it is possible to selectively set the AI image processing function of the camera device 3 so that analytical information corresponding to the purpose desired by the customer can be obtained.

It is also conceivable that the information to be obtained using the camera device 3 will change when a disaster such as an earthquake occurs. Specifically, during normal times, the AI image processing function for detecting customers, calculating crowding levels, and identifying attributes will be activated in order to function as a store surveillance camera, and when a disaster occurs, the AI image processing function will be switched to identify products remaining on the shelves. When making this switch, it is conceivable to change the AI model so that appropriate recognition results can be obtained.

In this example, the cloud server 5 has the function of selectively setting the AI image processing function of such a camera device 3.

The functions of the cloud server 5 may also be provided by the management device 2 or another server device.

Here, the connection between the cloud server 5, which is an information processing device on the cloud side, and the camera device 3, which is an information processing device on the edge side, will be described with reference to FIG. 19.

The cloud-side information processing device is equipped with a re-learning function, device management function, and marketplace function that are available via the hub.

The Hub performs secure and highly reliable communication with the edge-side information processing device. This allows it to provide various functions to the edge-side information processing device.

The re-learning function is a function that performs re-learning and provides a newly optimized AI model, thereby providing an appropriate AI model based on new learning materials.

The device management function is a function for managing the camera device 3 and other edge information processing devices, and can provide functions such as management and monitoring of the AI model deployed in the camera device 3, as well as problem detection and troubleshooting.

The device management function also manages information about the camera device 3 and the management device 2. The information about the camera device 3 and the management device 2 includes information about the chips used as the arithmetic processing units, memory capacity and storage capacity, CPU and memory usage rates, and software information such as the OS (Operating System) installed in each device.

Additionally, device management features ensure secure access by authorized users.

The marketplace function provides functions such as the registration of AI models developed by the above-mentioned AI model developers and AI applications developed by application developers, and the deployment of these developments to authorized edge-side information processing devices. The marketplace function also provides a function related to the payment of incentives in accordance with the deployment of the developments.

The camera device 3, which serves as an edge-side information processing device, is equipped with an edge runtime, an AI application, an AI model, and an image sensor IS.

The edge runtime functions as embedded software for managing applications deployed on the camera device 3 and communicating with the cloud-side information processing device.

As mentioned above, the AI model is an expansion of an AI model registered in the marketplace in the cloud-side information processing device, which enables the camera device 3 to use captured images to obtain AI image processing result information according to the purpose.

An overview of the functions of the cloud-side information processing device will be described with reference to Fig. 20. Note that the cloud-side information processing device is a collective name for devices such as the cloud server 5.
As shown in the figure, the cloud-side information processing device has a license authorization function F21, an account service function F22, a device monitoring function F23, a marketplace function F24, and a camera service function F25.

The license authorization function F21 is a function that performs various authentication-related processes. Specifically, the license authorization function F21 performs processes related to device authentication of each camera device 3, and processes related to authentication of each of the AI models, software, and firmware used in the camera device 3.

Here, the above software refers to software necessary to properly realize AI image processing in the camera device 3.
In order to properly perform AI image processing based on the captured image and to transmit the results of the AI image processing to the management device 2 and the cloud server 5 in an appropriate format, it is required to control the input data to the AI model and properly process the output data of the AI model. The above software is software that includes peripheral processing required to properly realize the AI image processing. Such software is software for realizing a desired function using the AI model, and corresponds to the above-mentioned AI application.

In addition, the AI application is not limited to one that uses only one AI model, but may use two or more AI models. For example, there may be an AI application having a process flow in which image data as information of the recognition result (such as image data, hereinafter referred to as "recognition result information") obtained by an AI model that performs AI image processing using a captured image as an input tensor is input to another AI model as an input tensor to perform a second AI image processing.
Alternatively, the AI application may perform a predetermined image processing as a second AI image processing on an input tensor for the first AI image processing using coordinate information as recognition result information of the first AI image processing. Note that the input tensor for each AI image processing may be a RAW image, or may be an RGB image obtained by performing synchronization processing on a RAW image.

In the license authorization function F21, when the camera device 3 is authenticated via the communication network 7, a process of issuing a device ID (Identification) for each camera device 3 is performed.
In addition, for authentication of AI models and software, a process is carried out to issue unique IDs (AI model ID, software ID) for each AI model and AI application for which registration has been applied for from the AI model developer terminal 6C.
In addition, the license authorization function F21 performs processing to issue various keys and certificates, etc. to the manufacturer of the camera device 3, the AI model developer, and the software developer to ensure secure communication between the camera device 3, the AI model developer terminal 6C, and the cloud server 5, and also performs processing to update or suspend the validity of the certificate.
Furthermore, in the license authorization function F21, when user registration (registration of account information involving issuance of a user ID) is performed by the account service function F22 described below, the license authorization function F21 also performs a process of linking the camera device 3 (the above-mentioned device ID) purchased by the user with the user ID.

The account service function F22 is a function for generating and managing user account information. The account service function F22 accepts input of user information and generates account information based on the input user information (generating account information including at least a user ID and password information).
In addition, the account service function F22 also performs registration processing (registration of account information) for AI model developers and AI application developers (hereinafter sometimes abbreviated as "software developers").

The device monitoring function F23 is a function that performs processing to monitor the usage status of the camera device 3. For example, it monitors information such as the location where the camera device 3 is used, the output frequency of output data for AI image processing, the free space of the CPU and memory used for AI image processing, and the usage rate of the CPU and memory mentioned above as various elements related to the usage status of the camera device 3.

The marketplace function F24 is a function for selling AI models and AI applications. For example, a user can purchase an AI application and an AI model used by an AI application via a sales website (sales site) provided by the marketplace function F24. In addition, a software developer can purchase an AI model for creating an AI application via the sales site.

The camera service function F25 is a function for providing services relating to the use of the camera device 3 to the user.
One example of this camera service function F25 is the function related to the generation of the above-mentioned analysis information. That is, it is a function that performs processing to generate analytical information of a subject based on the processing result information of the image processing in the camera device 3 and to allow the user to view the information via the user terminal 6. This also includes processing to visualize and present the density.

The camera service function F25 may also include an imaging setting search function. Specifically, the imaging setting search function is a function that acquires recognition result information of the AI image processing from the camera device 3, and searches for imaging setting information of the camera device 3 using AI based on the acquired recognition result information. Here, the imaging setting information broadly means setting information related to imaging operations for obtaining an image. Specifically, it broadly includes optical settings such as focus and aperture, settings related to the readout operation of the imaging image signal such as frame rate, exposure time, gain, and further settings related to image signal processing for the readout imaging image signal such as gamma correction processing, noise reduction processing, super-resolution processing, etc. Also, settings regarding the direction of the optical axis of the camera device 3, i.e., settings of the imaging direction of the camera device 3, etc. may be included.
By properly functioning the imaging setting search function, the imaging settings of the camera device 3 are optimized according to the purpose set by the user, and good inference results can be obtained regarding, for example, the density.

The camera service function F25 also includes an AI model search function. This AI model search function acquires recognition result information of AI image processing from the camera device 3, and searches for an optimal AI model to be used for AI image processing in the camera device 3 using AI based on the acquired recognition result information. The AI model search referred to here means a process of optimizing various processing parameters such as weighting coefficients and setting information related to the neural network structure (including, for example, kernel size information) when the AI image processing is realized by a CNN (Convolutional Neural Network) that includes a convolution operation.

The camera service function F25 may also have a function for determining processing load. In the processing load determination function, when an AI application is deployed to an edge-side information processing device, a process is performed to determine the device to which the application is deployed on a SW component-by-component basis. Some SW components may be determined to be executed on a cloud-side device, in which case the deployment process may not be performed as the SW components have already been deployed to the cloud-side device.

By having the above-mentioned image capture setting search function and AI model search function, it is possible to perform image capture settings that will produce good results in AI image processing, and it is also possible to perform AI image processing using an appropriate AI model according to the actual usage environment.
In addition, by having a processing allocation determination function, it is possible to ensure that AI image processing and its analysis processing are executed in an appropriate device.

The camera service function F25 has an application setting function prior to deploying each SW component. The application setting function is a function that sets an appropriate AI application according to the user's purpose.

For example, an appropriate AI application is selected according to the purpose selected by the user. This automatically determines the SW components that make up the AI application. Note that there may be multiple combinations of SW components to achieve the user's purpose using the AI application, in which case one combination is selected according to the information on the edge-side information processing device and the user's request.

For example, if a user wishes to monitor a store, the combination of SW components may be different depending on whether the user's requirements place a premium on privacy or speed.

The application setting function involves processes such as accepting an operation by the user on the user terminal 6 (corresponding to the application user terminal 6B in FIG. 18) to select a purpose (application) and selecting an appropriate AI application according to the selected application.

Here, in the above, the cloud server 5 alone realizes the license authorization function F21, the account service function F22, the device monitoring function F23, the marketplace function F24, and the camera service function F25, but it is also possible to realize these functions by sharing them among a plurality of information processing devices. For example, it is possible to configure each of the above functions to be performed by one information processing device. Alternatively, it is also possible to share a single function among the above functions among a plurality of information processing devices (for example, the cloud server 5 and other server devices).

<11. Deployment of AI models and AI applications>
As described above, information on the AI application and the AI model may be expanded as a container in the memory unit 34 outside the image sensor IS in the camera device 3 using container technology, and then only the AI model may be stored in the memory unit 45 in the image sensor IS. The container technology will be described with reference to the attached drawings.

In the camera device 3, an operation system 51 is installed on various hardware 50 such as a CPU, a GPU (Graphics Processing Unit), a ROM, and a RAM, which serve as the control unit 33 shown in FIG. 2 (see FIG. 21).

The operation system 51 is basic software that performs overall control of the camera device 3 to realize various functions in the camera device 3.

General-purpose middleware 52 is installed on the operation system 51.

The general-purpose middleware 52 is software for implementing basic operations such as a communication function using the communication unit 35 as the hardware 50 and a display function using a display unit (such as a monitor) as the hardware 50.

In addition to the general-purpose middleware 52, an orchestration tool 53 and a container engine 54 are installed on the operation system 51.

The orchestration tool 53 and the container engine 54 deploy and execute the container 55 by constructing a cluster 56 as an operating environment for the container 55 .
The edge runtime shown in FIG. 19 corresponds to the orchestration tool 53 and the container engine 54 shown in FIG.

The orchestration tool 53 has a function for appropriately allocating the resources of the above-mentioned hardware 50 and operation system 51 to the container engine 54. The orchestration tool 53 groups each container 55 into a predetermined unit (a pod, described later), and each pod is deployed to a worker node (described later) that is a logically different area.

The container engine 54 is a piece of middleware installed in the operation system 51, and is an engine that runs the container 55. Specifically, the container engine 54 has a function of allocating the resources (memory, computing power, etc.) of the hardware 50 and the operation system 51 to the container 55 based on a configuration file or the like provided by the middleware in the container 55.

In addition, the resources allocated in this embodiment include not only resources such as the control unit 33 of the camera device 3, but also resources such as the sensor internal control unit 43, memory unit 45, and communication I/F 46 of the image sensor IS.

The container 55 includes middleware such as applications and libraries for implementing predetermined functions.
The container 55 operates to realize a specified function using the resources of the hardware 50 and the operation system 51 allocated by the container engine 54.

In this embodiment, the AI application and AI model shown in FIG. 19 correspond to one of the containers 55. That is, one of the various containers 55 deployed in the camera device 3 realizes a predetermined AI image processing function using the AI application and AI model.

A specific configuration example of a cluster 56 constructed by the container engine 54 and the orchestration tool 53 will be described with reference to FIG. 22. Note that the cluster 56 may be constructed across multiple devices so that functions are realized using not only the hardware 50 of one camera device 3 but also the resources of other hardware of other devices.

The orchestration tool 53 manages the execution environment of the container 55 on a worker node 57 basis. The orchestration tool 53 also constructs a master node 58 that manages all of the worker nodes 57.

In the worker node 57, multiple pods 59 are deployed. The pod 59 is configured to include one or more containers 55 and realizes a specified function. The pod 59 is treated as a management unit for managing the containers 55 by the orchestration tool 53.

The operation of the pod 59 on the worker node 57 is controlled by the pod management library 60.

The pod management library 60 is composed of a container runtime that allows the pods 59 to utilize the resources of the logically allocated hardware 50, an agent that accepts control from the master node 58, and a network proxy that communicates between the pods 59 and with the master node 58.
That is, each pod 59 is enabled to realize a predetermined function using each resource by the pod management library 60 .

The master node 58 includes an application server 61 that deploys the pod 59, a manager 62 that manages the deployment status of the container 55 by the application server 61, a scheduler 63 that determines the worker node 57 on which the container 55 is placed, and a data sharing unit 64 that shares data.

By utilizing the configurations shown in Figures 21 and 22, it is possible to deploy the aforementioned AI applications and AI models to the image sensor IS of the camera device 3 using container technology.
As mentioned above, the AI model may be stored in the memory unit 45 in the image sensor IS via the communication I/F 46 in Figure 2, and AI image processing may be executed within the image sensor IS, or the configurations shown in Figures 21 and 22 may be deployed in the memory unit 45 and sensor control unit 43 in the image sensor IS, and the aforementioned AI application and AI model may be executed within the image sensor IS using container technology.
In addition, as described below, container technology can also be used when deploying AI applications and/or AI models to a management device 2 or a cloud-side information processing device.
In this case, information on the AI application or AI model is deployed as a container or the like in memory such as the non-volatile memory unit 74, storage unit 79, or RAM 73 in Figure 23 described below, and executed.

<12. Hardware configuration of information processing device>
The hardware configuration of information processing devices such as the cloud server 5, the user terminal 6, and the management device 2 included in the density calculation system 1 will be described with reference to FIG.

The information processing device includes a CPU 71. The CPU 71 functions as an arithmetic processing unit that performs the various processes described above, and executes the various processes according to a program stored in a ROM 72 or a non-volatile memory unit 74, such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute the various processes, as appropriate.

The CPU 71 provided in the information processing device serving as the cloud server 5 functions as a license authorization unit, an account service providing unit, a device monitoring unit, a marketplace function providing unit, and a camera service providing unit to realize the above-mentioned functions.

The CPU 71, ROM 72, RAM 73, and non-volatile memory unit 74 are interconnected via a bus 83. An input/output interface (I/F) 75 is also connected to this bus 83.

The input/output interface 75 is connected to an input unit 76 including an operator and an operating device.
For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller.
An operation by the user is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

Further, a display unit 77 such as an LCD or an organic EL panel, and an audio output unit 78 such as a speaker are connected to the input/output interface 75 either integrally or separately.
The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the computer device, or a separate display device connected to the computer device.

The display unit 77 displays images for various types of image processing, videos to be processed, etc., on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71.

The input/output interface 75 may also be connected to a storage unit 79 such as a hard disk or solid-state memory, or a communication unit 80 such as a modem.

The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired/wireless communication, bus communication, etc.

A drive 81 is also connected to the input/output interface 75 as necessary, and a removable storage medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately attached.

The drive 81 can read data files such as programs used for each process from the removable storage medium 82. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. In addition, computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as necessary.

In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.
In addition, images captured by the camera device 3 and the results of AI image processing may be received and stored in a removable storage medium 82 via the storage unit 79 or drive 81.

The CPU 71 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing of the cloud server 5, user terminal 6, and management device 2, which are information processing devices equipped with the above-mentioned arithmetic processing unit.
Note that the cloud server 5, the user terminal 6, and the management device 2 are not limited to being configured as a single computer device as shown in Fig. 23, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

<13. Other>
As described above, after the SW components and AI model of the AI application are deployed, the process flow when the AI model is re-learned and the AI model (hereinafter referred to as the "edge-side AI model") and the AI application deployed in each camera device 3 are updated using the operation of the service provider or user as a trigger will be specifically described with reference to FIG. 24. Note that FIG. 24 is described with a focus on one camera device 3 among the multiple camera devices 3. In addition, the edge-side AI model to be updated in the following description is, as an example, deployed in the image sensor IS provided in the camera device 3, but of course, the edge-side AI model may be deployed outside the image sensor IS in the camera device 3.

First, in processing step PS1, a service provider or user issues an instruction to retrain the AI model. This instruction is issued using an Application Programming Interface (API) function provided by an API module provided in the cloud-side information processing device. The instruction also specifies the amount of images (e.g., number) to be used for learning. Hereinafter, the amount of images to be used for learning will also be referred to as the "predetermined number."

The API module receives this instruction and in processing step PS2 sends a re-learning request and image volume information to the Hub (similar to that shown in Figure 19).

In processing step PS3, the Hub sends an update notification and image volume information to the camera device 3, which serves as the edge-side information processing device.

The camera device 3 transmits the captured image data obtained by capturing an image to an image database (DB) in the storage group in processing step PS4. This capturing process and transmission process is repeated until a predetermined number of images required for re-learning is reached.

When the camera device 3 obtains an inference result by performing inference processing on the captured image data, the camera device 3 may store the inference result, etc. in the image DB as metadata for the captured image data in processing step PS4.

By storing the inference results from the camera device 3 as metadata in the image DB, it is possible to carefully select the data necessary for re-learning the AI model executed on the cloud side. Specifically, re-learning can be performed using only image data where the inference results from the camera device 3 differ from the results of inference executed using abundant computer resources in the cloud-side information processing device. This makes it possible to shorten the time required for re-learning.

After capturing and transmitting the specified number of images, the camera device 3 notifies the Hub in processing step PS5 that transmission of the specified number of captured image data has been completed.

Upon receiving this notification, in processing step PS6, the Hub notifies the orchestration tool that the re-learning data has been prepared.

In processing step PS7, the orchestration tool sends an instruction to execute the labeling process to the labeling module.

The labeling module retrieves the image data to be subjected to the labeling process from the image DB (processing step PS8) and performs the labeling process.

The labeling process referred to here may be the process of classifying images as described above, or it may be a process of estimating the gender and age of the subject of an image and assigning a label to it, or it may be a process of estimating the pose of the subject and assigning a label to it, or it may be a process of estimating the behavior of the subject and assigning a label to it.

The labeling process may be performed manually or automatically. The labeling process may be completed by an information processing device on the cloud side, or may be realized by using a service provided by another server device.

After completing the labeling process, the labeling module stores the labeling result information in the dataset DB in processing step PS9. Here, the information stored in the dataset DB may be a combination of label information and image data, or may be image ID (Identification) information for identifying the image data instead of the image data itself.

When the storage management unit detects that the labeling result information has been stored, it notifies the orchestration tool in processing step PS10.

The orchestration tool that receives this notification confirms that the labeling process for the specified number of image data sheets has been completed, and in processing step PS11, sends a re-learning instruction to the re-learning module.

Upon receiving the re-learning instruction, the re-learning module obtains the dataset to be used for learning from the dataset DB in processing step PS12, and obtains the AI model to be updated from the trained AI model DB in processing step PS13.

The re-learning module re-learns the AI model using the acquired dataset and AI model. The updated AI model obtained in this way is stored again in the trained AI model DB in processing step PS14.

When the storage management unit detects that an updated AI model has been stored, it notifies the orchestration tool in processing step PS15.

Upon receiving the notification, the orchestration tool sends an instruction to convert the AI model to the conversion module in processing step PS16.

The conversion module that receives the conversion instruction obtains the updated AI model from the trained AI model DB in processing step PS17, and performs conversion processing of the AI model.
In the conversion process, the data is converted to match the specification information of the camera device 3, which is the device to which the data is to be deployed. In this process, downsizing is performed to minimize the loss of performance of the AI model, and file format conversion is performed so that the AI model can run on the camera device 3.

The AI model converted by the conversion module is the edge-side AI model described above. This converted AI model is stored in the converted AI model DB in processing step PS18.

When the storage management unit detects that the converted AI model has been stored, it notifies the orchestration tool in processing step PS19.

In process step PS20, the orchestration tool that receives the notification sends a notification to the Hub to execute an update of the AI model. This notification includes information for identifying the location where the AI model to be used for the update is stored.

The Hub, which receives the notification, sends an instruction to update the AI model to the camera device 3. The update instruction also includes information for identifying the location where the AI model is stored.

In processing step PS22, the camera device 3 performs a process of retrieving the target converted AI model from the converted AI model DB and expanding it. This updates the AI model used by the image sensor IS of the camera device 3.

After the camera device 3 has completed updating the AI model by deploying the AI model, it sends an update completion notification to the Hub in processing step PS23.
Upon receiving the notification, the Hub notifies the orchestration tool in processing step PS24 that the AI model update processing of the camera device 3 has been completed.

Note that, although an example has been described here in which an AI model is deployed and used within the image sensor IS of the camera device 3 (e.g., the memory unit 45 shown in Figure 2), the AI model can be updated in the same manner even if the AI model is deployed and used outside the image sensor in the camera device 3 (e.g., the memory unit 34 in Figure 2) or in a memory unit within the management device 2.
In this case, when the AI model is deployed, the device (location) on which the AI model is deployed is stored in a storage management unit on the cloud side, and the Hub reads out the device (location) on which the AI model is deployed from the storage management unit and sends an instruction to update the AI model to the device on which the AI model is deployed.
In processing step PS22, the device that has received the update instruction performs a process of acquiring the target converted AI model from the converted AI model DB and expanding it. This updates the AI model of the device that has received the update instruction.

If you are only updating the AI model, the process is completed up to this point.
When updating an AI application that utilizes an AI model in addition to the AI model, the processing described below is further executed.

Specifically, in processing step PS25, the orchestration tool sends a download instruction for the AI application, such as updated firmware, to the deployment control module.

In process step PS26, the deployment control module sends an instruction to the Hub to deploy the AI application. This instruction includes information to identify the location where the updated AI application is stored.

In processing step PS27, the Hub sends the deployment instruction to the camera device 3.

In processing step PS28, the camera device 3 downloads the updated AI application from the container DB of the deployment control module and deploys it.

In the above explanation, an example was described in which the update of the AI model running on the image sensor IS of the camera device 3 and the update of the AI application running outside the image sensor IS in the camera device 3 are performed sequentially.
For simplicity, an AI application has been described here, but as described above, an AI application is defined by a plurality of SW components such as SW components B1, B2, B3, ..., Bn, and when an AI application is deployed, where each SW component is deployed is stored in a storage management unit on the cloud side, and when processing step PS27, the Hub reads out the device (location) where each SW component is deployed from the storage management unit and transmits a deployment instruction to the deployed device. In processing step PS28, the device that has received the deployment instruction downloads the updated SW component from the container DB of the deployment control module and deploys it.
Note that the AI application referred to here is a SW component other than the AI model.

In addition, if both the AI model and the AI application are to be run on a single device, both the AI model and the AI application may be updated together as a single container. In that case, the update of the AI model and the update of the AI application may be performed simultaneously, not sequentially. This can be achieved by executing the processes of the processing steps PS25, PS26, PS27, and PS28.

For example, if it is possible to deploy containers of both an AI model and an AI application to the image sensor IS of the camera device 3, the AI model and the AI application can be updated by executing the processing steps PS25, PS26, PS27, and PS28 as described above.

By performing the above-mentioned processing, the AI model is retrained using image data captured in the user's usage environment. Therefore, it is possible to generate an edge-side AI model that can output highly accurate recognition results in the user's usage environment.

Furthermore, even if the imaging environment of the camera device 3 changes, such as when a vehicle equipped with the onboard camera 3 is traveling in a different area than before, or when the amount of incident light entering the imaging device changes due to changes in weather or time, the AI model can be appropriately re-learned each time, making it possible to maintain the recognition accuracy of the AI model without degradation.
In addition, each of the above-mentioned processes may be performed not only when re-learning the AI model, but also when the system is operated for the first time in the user's environment.

<14. Marketplace screen example>
An example of a screen presented to a user regarding the marketplace will be described with reference to each drawing.

FIG. 25 shows an example of the login screen G1.
The login screen G1 has an ID input field 91 for inputting a user ID and a password input field 92 for inputting a password.

Below the password input field 92 are a login button 93 for logging in and a cancel button 94 for cancelling the login.

Further below that, operators for transitioning to a page for users who have forgotten their password, and operators for transitioning to a page for new user registration are appropriately placed.

When the user presses the login button 93 after entering the appropriate user ID and password, a process to transition to a user-specific page is executed on both the cloud server 5 and the user terminal 6.

Figure 26 is an example of a screen presented to, for example, an AI application developer using the application developer terminal 6A or an AI model developer using the AI model developer terminal 6C.

Through the marketplace, developers can purchase training datasets, AI models, and AI applications for development. They can also register AI applications and AI models that they have developed themselves on the marketplace.

The developer screen G2 shown in FIG. 26 displays purchasable learning data sets, AI models, AI applications, etc. (hereinafter collectively referred to as "data") on the left side.
Although not shown in the figure, when purchasing a training data set, the user can prepare for training by simply displaying an image of the training data set on a display, using an input device such as a mouse to frame only the desired portion of the image, and entering a name.
For example, if you want to use an image of a cat for AI training, you can prepare an image with a cat annotation for AI training by surrounding only the cat part of the image with a frame and entering "cat" as the text input.
In addition, in order to make it easier to find desired data, it may be possible to select an objective such as "traffic monitoring,""traffic line analysis," or "customer count." That is, a display process is executed in each of the cloud server 5 and the user terminal 6 so that only data that matches the selected objective is displayed.

The purchase price of each piece of data may also be displayed on the developer screen G2.

In addition, on the right side of the developer screen G2, there is an input field 95 for registering learning datasets collected or created by the developer, as well as AI models and AI applications developed by the developer.

For each piece of data, an input field 95 is provided for inputting the name and where the data is saved. In addition, for AI models, a check box 96 is provided for setting whether or not retraining is required.

In addition, a price setting field (shown as input field 95 in the figure) may be provided in which the price required to purchase the data to be registered can be set.

In addition, the upper part of the developer screen G2 displays the user name, last login date, and so on as part of the user information. In addition to this, the amount of currency and points that the user can use when purchasing data may also be displayed.

Figure 27 shows an example of a user screen G3 that is presented to a user (the application user described above) who performs various analyses, for example, by deploying an AI application or an AI model to a camera device 3 as an edge-side information processing device managed by the user.

Users can purchase camera devices 3 to be placed in the space to be monitored via the marketplace. Therefore, on the left side of the user screen G3, radio buttons 97 are provided that allow users to select the type and performance of the image sensor IS to be installed in the camera device 3, as well as the performance of the camera device 3.

Furthermore, a user can purchase an information processing device as the management device 2 via the marketplace. Therefore, radio buttons 97 for selecting each performance of the management device 2 are arranged on the left side of the user-oriented screen G3.
Furthermore, a user who already has a management device 2 can register the performance of the management device 2 by inputting performance information of the management device 2 here.

A user can achieve the desired functions by installing the purchased camera device 3 (or a camera device 3 purchased without going through the marketplace) in a location of their choice, such as a store that they manage, and the marketplace allows users to register information about the installation location of each camera device 3 in order to maximize the functionality of each camera device 3.

A radio button 98 is arranged on the right side of the user screen G3, which allows the user to select environmental information about the environment in which the camera device 3 is installed. By appropriately selecting the environmental information about the environment in which the camera device 3 is installed, the above-mentioned optimal imaging settings are set for the target camera device 3.

When purchasing a camera device 3 and the installation location of the camera device 3 to be purchased has been decided, the user can purchase a camera device 3 with the optimal imaging settings pre-set according to the planned installation location by selecting each item on the left side and each item on the right side of the user screen G3.

The user screen G3 has an execute button 99. Pressing the execute button 99 transitions to a confirmation screen for confirming the purchase or a confirmation screen for confirming the setting of environmental information. This allows the user to purchase the desired camera device 3 or management device 2, or to set environmental information for the camera device 3.

In the marketplace, it is possible to change the environmental information of each camera device 3 in case the installation location of the camera device 3 is changed. By re-inputting the environmental information about the installation location of the camera device 3 on a change screen (not shown), it is possible to re-set the optimal imaging settings for the camera device 3.

<15. Summary>
As described above, a signal processing device such as an image sensor IS or a camera device 3 equipped with an image sensor IS is equipped with an image processing unit (AI image processing unit 44) that sets a bounding box BB corresponding to a person P (subject) detected in the captured image on the captured image and calculates the density of the person P based on the degree of overlap (IoU) between the bounding boxes BB.
This makes it possible to locally calculate the density through simple processing using one captured image.
Therefore, it will be possible to determine and recommend popular areas based on their density, as well as suggest routes that avoid crowds.
Furthermore, by properly calculating the level of crowding, it will be possible to make announcements to avoid crowded conditions, thereby enabling measures to prevent the spread of infection.
In addition, by using the above-mentioned analysis of the movement of person P in combination with the calculation of the density, it becomes possible to determine whether a person infected with an infectious disease visited a highly crowded area before and after infection, and if so, which area. Therefore, it becomes possible to identify other people P who were in the same space as the positive person or in close proximity thereto, and to confirm the movement of such people P, making it possible to clarify the infection route and identify other people P who are suspected of infection.

As described with reference to Figures 6 and 11, an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate the density for each person P.
By calculating the density for each person P, the density can be calculated more precisely than in a method of calculating the density for each area, and the resolution of the density information can be increased.

As described in the modified example, an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate the density using the average value of values indicating the degree of density for each pair of a target bounding box BBt, which is a bounding box BB corresponding to the target person Pt being processed, and an overlapping bounding box BBd, which is a bounding box BB having a common area CA1 with the target bounding box BBt.
This makes it possible to appropriately calculate the density based on the degree of overlap of a plurality of bounding boxes BB without identifying key points KP for the person P. This makes it possible to reduce the processing load on the image sensor IS.

As described in Figure 10 and the modified example, an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate a value indicating the degree of congestion for each group based on the proportion of the common area CA1 to the logical sum area DA1 of the target bounding box BBt and the overlapping bounding box BBd.
This makes it possible to calculate the basic density (basic IoU) for the two bounding boxes BB through simple processing.

As explained with reference to Figures 6 and 12, an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may detect specific parts of a person P in a captured image as key points KP, and calculate the density based on the degree of overlap between bounding boxes BB and the rate at which key points KP go undetected.
By considering not only the degree of overlap of the bounding boxes BB but also whether or not occlusion of the joints of the person P by the body of another person has occurred, it is possible to more appropriately quantify the closeness of the persons P to each other. Therefore, it becomes possible to calculate an appropriate density.

As explained with reference to Figure 8, etc., an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS calculates the center of gravity position of the bounding box BB as the box center of gravity position BBP, calculates the center of gravity positions of multiple key points KP that are estimated to belong to the same person as the key point center of gravity position KBP, and may link the box center of gravity position BBP and the key point center of gravity position KBP that have close coordinates on the captured image as belonging to the same person.
As a result, when the process of setting the bounding box BB and the process of detecting the key points KP are realized using different AI models, it is possible to appropriately link the bounding box BB and the key points KP for each person P. Therefore, it is possible to appropriately calculate the density for each person P.

As described above, an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate the likelihood for each key point KP.
By calculating the likelihood for each key point KP, it is possible to prevent the non-detection rate from being calculated low due to an uncertain key point KP being detected. Therefore, it is possible to calculate the density with high accuracy.

As described with reference to Figure 12, etc., an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may treat a key point KP whose likelihood is less than a predetermined threshold (likelihood threshold Th1) in calculating the density as an undetected key point KP.
This makes it possible to calculate the key point center of gravity position KBP and the non-detection rate using only the key points KP that are likely to be specific parts of the person P, thereby making it possible to calculate the density with high accuracy.

As explained with reference to Figure 11, etc., an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate the density by excluding a key point KP when the undetection rate of a key point KP that is estimated to be the same person is equal to or greater than a predetermined threshold (undetection rate threshold Th2).
In other words, when almost no key points KP can be detected or when many key points KP with low likelihood are detected, it is possible to prevent an inappropriate density from being calculated by determining that the person P has not been detected.

As explained with reference to Figure 9, etc., an image processing unit (AI image processing unit 44) in a signal processing device such as an image sensor IS may calculate the density by excluding a person P that is located on the outer edge of the captured image and is partially cut off.
This makes it possible to prevent a situation in which a non-detection rate of key points KP is calculated to be unreasonably high and the density is calculated to be high due to a part such as a joint being located outside the angle of view of the camera device 3. In other words, it is possible to improve the accuracy of the density.

As described with reference to FIG. 2 etc., a signal processing device such as an image sensor IS may include an output unit (communication I/F 46) that outputs information on the density as a log.
This makes it easier to carry out downstream processing according to the degree of crowding. For example, it is easy to realize a process of determining whether an area is crowded or quiet, or a process of presenting areas with few people as recommended areas, based on multiple captured images obtained from multiple cameras and log information on the degree of crowding.

As described with reference to FIG. 2 etc., a signal processing device such as an image sensor IS may include an output unit (communication I/F 46) that outputs information on the density and non-detection rate as a log.
By outputting information on the density and the rate of non-detection of key points KP as a log, it becomes easier to use them in subsequent processing.

Furthermore, an output section (communication I/F 46) in a signal processing device such as an image sensor IS may output a log as metadata of a captured image.
This makes it possible to output data on the density and the non-detection rate as general data conforming to the MIPI standard, for example, and therefore to output the data to various devices capable of receiving standardized data without unnecessary data processing.

As described with reference to FIG. 2 etc., in a signal processing device such as an image sensor IS, a captured image may be an RGB image.
This makes it possible to use existing programs that detect key points KP based on RGB images, thereby improving processing efficiency.

A signal processing device such as the management device 2 includes a display control section (display control function F12) that executes image display based on the density.
Such display control by the display control unit allows a predetermined display to be realized on an information processing device such as a user terminal, thereby enabling a user to visually grasp the degree of congestion.

In addition, a display control unit (display control function F12) in a signal processing device such as the management device 2 may generate a heat map using a two-dimensional Gaussian distribution weighted according to the density and superimpose it on the captured image.
This allows the user to visually grasp the density based on the image that reflects the density value.

The signal processing method executed by a signal processing device such as an image sensor IS includes a process of setting a bounding box BB corresponding to a person P detected in a captured image on the captured image, and a process of calculating the density of the person P according to the degree of overlap between the bounding boxes BB.

The program in the present technology causes a computing device to execute the functions of setting a bounding box BB on the captured image corresponding to a person P detected in the captured image, and calculating the density of the person P based on the degree of overlap between the bounding boxes BB, and the storage medium of the present technology is readable by a computer device in which such a program is stored.
By using such a program, the image sensor IS serving as the above-mentioned signal processing device can locally estimate the density without using a plurality of captured images.

These programs can be pre-recorded in a HDD (Hard Disk Drive) as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Alternatively, the programs can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called package software.
Such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

Furthermore, the above-mentioned examples may be combined in any manner, and even when various combinations are used, the above-mentioned various operational effects can be obtained.

<16. This Technology>
The present technology can also be configured as follows.
(1)
and an image processing unit that sets a bounding box on a captured image corresponding to an object detected in the captured image, and calculates a density of the object according to a degree of overlap between the bounding boxes.
(2)
The signal processing device according to (1), wherein the image processing unit calculates the density for each of the objects.
(3)
The image processing unit includes:
The signal processing device according to the above (2), which calculates the density using an average value of values indicating the density for each pair of a target bounding box, which is the bounding box corresponding to the object to be processed, and an overlapping bounding box, which is the bounding box having a common area with the target bounding box.
(4)
The signal processing device according to the above (3), wherein the image processing unit calculates a value indicating a degree of congestion for each of the pairs according to a proportion of the common area to a logical sum area of the target bounding box and the overlapping bounding box.
(5)
The image processing unit includes:
Detecting a predetermined portion of the object in the captured image as a key point;
The signal processing device according to any one of (1) to (4), wherein the density is calculated according to a degree of overlap between the bounding boxes and a rate of non-detection of the key points.
(6)
The image processing unit includes:
Calculating the center of gravity of the bounding box as a box center of gravity position;
Calculating the center of gravity of the plurality of key points estimated to be the same object as a key point center of gravity position;
The signal processing device according to (5) above, wherein the box centroid position and the keypoint centroid position that are close in coordinates on the captured image are linked as being related to the same object.
(7)
The signal processing device according to any one of (5) to (6), wherein the image processing unit calculates a likelihood for each of the key points.
(8)
The signal processing device according to (7) above, wherein the image processing unit treats the keypoint whose likelihood is less than a predetermined threshold in the calculation of the density as an undetected keypoint.
(9)
The signal processing device according to (8) above, wherein when a non-detection rate of the keypoints estimated to be the same object is equal to or higher than a predetermined threshold, the image processing unit calculates the density while excluding the keypoints.
(10)
The signal processing device according to any one of (5) to (9) above, wherein the image processing unit calculates the density by excluding objects that are located on the outer periphery of the captured image and are partially cut off.
(11)
The signal processing device according to any one of (1) to (10), further comprising an output unit that outputs information about the congestion degree as a log.
(12)
The signal processing device according to any one of (5) to (10), further comprising an output unit that outputs information on the density and the non-detection rate as a log.
(13)
The signal processing device according to any one of (11) to (12), wherein the output unit outputs the log as metadata of the captured image.
(14)
The signal processing device according to any one of (1) to (13) above, wherein the captured image is an RGB image.
(15)
A process of setting a bounding box on a captured image corresponding to an object detected in the captured image;
and calculating a density of the object according to a degree of overlap between the bounding boxes.
(16)
A function of setting a bounding box on a captured image corresponding to an object detected in the captured image;
A storage medium readable by a computer device, storing a program that causes a processor to execute a function of calculating a density of the object according to the degree of overlap between the bounding boxes.

3. Camera device (signal processing device)
46 Communication I/F (output section)
IS Image sensor (signal processing device)
BB Bounding box BBt Object bounding box BBd1, BBd2 Overlapping bounding boxes KP Key point BBP Box center of gravity position KBP Key point center of gravity position P Person (object)
Pt Target person (target object)
CA1 common area DA1 logical sum area Th1 likelihood threshold (predetermined threshold)
Th2 Undetected rate threshold (predetermined threshold)

Claims (16)

and an image processing unit that sets a bounding box on a captured image corresponding to an object detected in the captured image, and calculates a density of the object according to a degree of overlap between the bounding boxes. The signal processing device according to claim 1 , wherein the image processing unit calculates the density for each of the objects. The image processing unit includes:
The signal processing device according to claim 2 , wherein the density is calculated using an average value of values indicating a density degree for each pair of a target bounding box, which is the bounding box corresponding to the object to be processed, and an overlapping bounding box, which is the bounding box having a common area with the target bounding box. The signal processing device according to claim 3 , wherein the image processing unit calculates a value indicating a degree of congestion for each of the pairs in accordance with a proportion of the common area to a logical sum area of the target bounding box and the overlapping bounding box. The image processing unit includes:
Detecting a predetermined portion of the object in the captured image as a key point;
The signal processing device according to claim 1 , wherein the density is calculated according to a degree of overlap between the bounding boxes and a rate of non-detection of the key points. The image processing unit includes:
Calculating the center of gravity of the bounding box as a box center of gravity position;
Calculating the center of gravity of the plurality of key points estimated to be the same object as a key point center of gravity position;
The signal processing device according to claim 5 , wherein the box centroid position and the keypoint centroid position that are close to each other in coordinates on the captured image are linked as being related to the same object. The signal processing device according to claim 5 , wherein the image processing unit calculates a likelihood for each of the key points. The signal processing device according to claim 7 , wherein the image processing unit treats the keypoints whose likelihoods are less than a predetermined threshold in the calculation of the density as undetected keypoints. The signal processing device according to claim 8 , wherein, when a non-detection rate of the keypoints estimated to be the same object is equal to or higher than a predetermined threshold, the image processing unit calculates the density while excluding the keypoints. The signal processing device according to claim 5 , wherein the image processing unit calculates the density by excluding objects that are located on an outer periphery of the captured image and are partially cut off. The signal processing device according to claim 1 , further comprising an output unit that outputs the congestion information as a log. The signal processing device according to claim 5 , further comprising an output unit that outputs information on the density and the non-detection rate as a log. The signal processing device according to claim 11 , wherein the output section outputs the log as metadata of the captured image. The signal processing device according to claim 1 , wherein the captured image is an RGB image. A process of setting a bounding box on a captured image corresponding to an object detected in the captured image;
and calculating a density of the object according to a degree of overlap between the bounding boxes. A function of setting a bounding box on a captured image corresponding to an object detected in the captured image;
A storage medium readable by a computer device, storing a program for causing a processor to execute a function of calculating a density of the object according to a degree of overlap between the bounding boxes.

Images