WO2020213284A1 - Image processing device, image processing method, and program - Google Patents

Abstract

[Problem] To provide an image processing device which executes an image process at a low load. [Solution] This image processing device comprises a determination unit and an image processing unit. The determination unit determines whether to execute an image process on an image process subject on the basis of the probability of success of executing the image process on the image process subject per image capture situation, which is the situation under which the image process subject is captured. The image processing unit executes the image process on the image process subject when it is determined that an image process is to be performed on the image process subject.

Description

Image processing equipment, image processing methods and programs

The present invention relates to an image processing apparatus, an image processing method and a program.

Image data (video data) captured by the camera is collected, and image processing (information processing) related to the collected image data is performed. Specifically, a computer (computer) performs image processing on a person or an object included in the image data. For example, a computer extracts a photographed face image of a person and uses the face image for a person authentication process. Alternatively, a computer may use a facial image to determine the age and gender of the person.

In these processes (face recognition process, age / gender determination process), a step of specifying an area in which a face image is captured from image data, a step of cutting out the specified face image, and a step of cutting out the cut out face image are sequentially predetermined. Includes a process of processing (face recognition, age / gender determination) based on the content.

For example, Patent Document 1 discloses a face recognition process. In the face recognition process, the feature amount that characterizes the face image, such as the shape and size of the part (extracted part; for example, eyes, nose, mouth, and the entire face) cut out from the face image, is calculated. After that, in the face recognition process, the calculated face feature amount (feature vector composed of a plurality of feature amounts) is collated with the face feature amount (feature vector) registered in the database. Usually, such image processing (processing typified by face recognition and age / gender determination) requires a large amount of calculation and a heavy load on the computer.

If the image quality of the image data to be processed (for example, the image quality of the image captured by the camera) is low, the face recognition process or the like may fail. For example, in the face recognition process, the feature amount is extracted from the face image. At that time, if the size and orientation of the captured face image are not accurate, the feature amount such as the shape and size of the part (eyes, nose, mouth, entire face) to be calculated for the feature amount cannot be extracted. Sometimes.

In this way, as a result of executing the feature amount extraction process from an image in which the size and orientation of the face image are not appropriate, a situation occurs in which the feature amount cannot be extracted or an inaccurate feature amount is extracted. It could be. In order to avoid such inconvenience, the same person is subjected to a plurality of recognition processes, and a large number of obtained recognition results are used.

Normally, image processing such as face recognition is performed by a server on the network (server in the cloud environment) or a server near the sensor (server in the edge environment). Whether the face recognition process is performed in the cloud environment or the edge environment is determined in consideration of cost and communication delay.

When face recognition processing is performed in an edge environment, a computer is placed near the sensor (for example, a camera). The sensor sends data to an application on a nearby computer. Normally, in an edge environment, the computational resources are placed near the sensor, so that the computational resources are expensive. That is, if a plurality of sensors are included in the system, each of the plurality of sensors requires a computational resource (computer), and the cost is high.

When performing face recognition processing etc. in an edge environment, it is difficult to allocate a large amount of computational resources due to restrictions on the location of computers. Therefore, the available computational resources are limited, and it is difficult to increase or decrease the computational resources according to the application load.

On the other hand, if face recognition processing is performed in the edge environment, the processing will be completed in the computer located near the sensor. As a result, communication costs are reduced because communication between computers (servers), which is required in a cloud environment, becomes unnecessary. Specifically, only communication will occur when aggregating the calculated results.

Whether the application (for example, face recognition processing) is executed in the cloud environment or the edge environment is determined by the requirements required for the system. For example, when performing camera image analysis in a small store, it is difficult to transmit data to the cloud environment from the viewpoint of privacy and communication cost. Therefore, when performing image analysis of a camera in a small store, it is suitable to process the face image in an edge environment.

However, since it is difficult to deploy a large amount of computational resources (computers) in a small store, it is necessary for a small-scale computer to execute the above application such as face recognition. As described above, the load of face recognition processing and age / gender determination processing is heavy. Therefore, it is difficult to analyze all the face images included in the image captured by the camera in real time, and there is a possibility that only a part of the face images can be processed.

Patent Document 2 discloses that the image quality is adjusted by a camera in order to reduce the network load. In this document, when the same person is tracked and the face image detected more recently than the face image detected in the past is an image suitable for face image analysis, the image is detected as the best shot image.

Japanese Unexamined Patent Publication No. 11-161790 JP-A-2017-163228

In Patent Document 2, face image analysis is performed using only the best shot image in order to accurately perform image processing related to a person's face image by performing face image recognition only once. At that time, in Patent Document 2, as a judgment index of whether or not the face image is an image suitable for face recognition, it is determined that the person is facing the front, that the person is not out of focus, and that the person's eyes are open. I am using it.

In Patent Document 2, a high-load process utilizing machine learning such as face orientation determination is executed in order to determine whether the image is suitable for authentication processing or the like (process for determining the best shot image). Therefore, even in the technique of Patent Document 2, it is still difficult to process the entire face image in the image in real time.

A main object of the present invention is to provide an image processing apparatus, an image processing method, and a program that contribute to executing image processing with a low load.

According to the first viewpoint of the present invention, the image processing is based on the success probability when the image processing target is executed for the image processing target for each imaging situation, which is the situation when the image processing target is imaged. A determination unit that determines whether or not to execute the image processing on the target, and the image processing on the image processing target when it is determined that the image processing is executed on the image processing target. An image processing apparatus comprising an image processing unit for executing the above is provided.

According to the second viewpoint of the present invention, in the image processing apparatus, the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Based on this, it is determined whether or not to execute the image processing on the image processing target, and when it is determined that the image processing is executed on the image processing target, the image is subjected to the image processing target. Image processing methods are provided that include performing the processing.

According to the third viewpoint of the present invention, image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged by the computer mounted on the image processing device. Based on the success probability of the case, the process of determining whether or not to execute the image processing on the image processing target, and the image when it is determined to execute the image processing on the image processing target. A process for executing the image processing on the processing target and a program for executing the image processing are provided.

According to each viewpoint of the present invention, an image processing apparatus, an image processing method, and a program that contribute to executing image processing with a low load are provided. In addition, according to the present invention, other effects may be produced in place of or in combination with the effect.

FIG. 1 is a diagram for explaining an outline of one embodiment. FIG. 2 is a diagram showing an example of a schematic configuration of an image processing system according to the first embodiment. FIG. 3 is a diagram showing an example of the internal configuration of the image processing apparatus according to the first embodiment. FIG. 4 is a diagram for explaining the operation of the acquisition unit according to the first embodiment. FIG. 5 is a diagram for explaining division of still image data. FIG. 6 is a diagram showing an example of information held by the storage unit according to the first embodiment. FIG. 7 is a flowchart showing an example of the operation of the image processing apparatus according to the first embodiment. FIG. 8 is a diagram showing an example of information held by the storage unit according to the second embodiment. FIG. 9 is a diagram showing an example of information held by the storage unit according to the third embodiment. FIG. 10 is a diagram showing an example of the hardware configuration of the image processing device. FIG. 11 is a diagram showing an example of the internal configuration of the image processing apparatus according to the modified example.

First, the outline of one embodiment will be explained. It should be noted that the drawing reference reference numerals added to this outline are added to each element for convenience as an example to aid understanding, and the description of this outline is not intended to limit anything. In the present specification and the drawings, elements that can be similarly described may be designated by the same reference numerals so that duplicate description may be omitted.

The image processing device 100 according to the embodiment includes a determination unit 101 and an image processing unit 102 (see FIG. 1). The determination unit 101 executes image processing on the image processing target based on the success probability when the image processing target is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Determine whether or not to do so. The image processing unit 102 executes image processing on the image processing target when it is determined that the image processing is executed on the image processing target.

The image processing device 100 executes processing related to the image processing target according to the success probability of image processing according to the situation (for example, the position of the person) when the image processing target (for example, a person) is imaged. Judge whether or not. For example, with respect to an image of a person far away from the camera device 10, it is unlikely that the face recognition process for the image will succeed. Therefore, the image processing apparatus 100 calculates the success probability of the image processing in advance for each situation when the image is acquired, and performs the image processing only when the processing is expected to be completed normally. As a result, the situation where the image processing is tried but fails is avoided, and the resource waste of the limited computational resources (particularly the computational resources arranged in the edge environment) is prevented. That is, image processing can be realized with a low load, and a large number of objects are processed with limited computational resources.

The specific embodiment will be described in more detail below with reference to the drawings.

[First Embodiment]
The first embodiment will be described in more detail with reference to the drawings.

<System configuration>
FIG. 2 is a diagram showing an example of a schematic configuration of an image processing system according to the first embodiment. Referring to FIG. 2, the image processing system includes a plurality of camera devices 10-1 to 10-n (n is a positive integer, the same applies hereinafter), an image processing device 20, and a result storage device 30. Will be done. In the following description, unless there is a particular reason for distinguishing the camera devices 10-1 to 10-n, the term "camera device 10" is simply used.

In the image processing system, each of the plurality of camera devices 10 is connected to the image processing device 20. Further, the image processing device 20 and the result storage device 30 are connected. The system configuration shown in FIG. 2 is an example, and is not intended to limit the number of camera devices 10 and the like. The image processing system may include at least one or more camera devices 10.

The image processing device 20 acquires video data from each camera device 10. The image processing device 20 performs image processing (data analysis) on the acquired video data, and stores the result in the result storage device 30.

The result storage device 30 stores the processing result of the image processing device 20.

<Outline of system operation>
The image processing device 20 separates the video data (moving image) MD acquired from each camera device 10 into m (m is a positive integer, the same applies hereinafter) still image data SDj (j = 0 to m). The image processing device 20 performs image processing (for example, a face) on the face image FPk (k = 0 to p; p is a positive integer, the same applies hereinafter) of the kth person included in the separated still image data SDj. Authentication process) is tried.

At that time, the image processing device 20 determines whether or not to actually execute the image processing on the person included in the still image data SDj based on the probability that the image processing is normally completed (hereinafter referred to as the success probability). to decide.

Here, the success probability is calculated for each situation (hereinafter referred to as an imaging situation) when an image processing target (a person appearing in an image in the above example) is imaged, and the success probability is calculated inside the image processing device 20. Results are accumulated. For example, the imaging situation is the position of a person imaged by the camera device 10 (coordinate position of the person in the image).

The image processing device 20 counts the results (normal end, abnormal end of processing) when image processing is performed on the face image FPk for each imaging status. The image processing device 20 calculates the success probability in each imaging situation based on the number of trials of the image processing (hereinafter, the number of trials).

When the data on the success probability is sufficiently accumulated, the image processing device 20 utilizes the accumulated success probability to determine whether or not to perform image processing. Specifically, the image processing device 20 performs threshold processing on the acquired success probability, and does not execute image processing (for example, face recognition processing) if the success probability is low.

With such a response, the processing is performed under a meaningful situation in which the image processing is executed, so that the load on the computer that performs the face recognition processing and the like can be reduced. In other words, in a situation where image processing is unlikely to succeed, image processing (for example, authentication processing) is not performed on the image processing target (for example, the person appearing in the image), and computer resources are wasted. There is no. As a result, the number of people that can be processed increases as a whole.

For example, in FIG. 2, if the success probability of image processing relating to a person appearing in the still image data acquired from the camera device 10-1 is low, such processing is postponed. As a result, the resources of the image processing device 20 can be allocated to the still image data acquired from the camera device 10 other than the camera device 10-1, and as a result, a large number of face image FPk can be processed.

<Configuration of image processing device>
FIG. 3 is a diagram showing an example of the internal configuration of the image processing device 20 according to the first embodiment. Referring to FIG. 3, the image processing device 20 includes an acquisition unit 201, a determination unit 202, a storage unit 203, an image cutting unit 204, and an image processing unit 205.

The acquisition unit 201 extracts the still image data SDj from the video data DM acquired from the camera device 10 at a predetermined timing (predetermined sampling). For example, the acquisition unit 201 extracts (captures) the still image data SDj shown in FIG.

Next, the acquisition unit 201 attempts to extract a person from the extracted still image data SDj. For example, in the example of FIG. 4, the acquisition unit 201 extracts the person 301. If the acquisition unit 201 cannot extract a person from the still image data SDj, the acquisition unit 201 targets the next still image data SDj + 1 and the video data DM for processing.

Next, the acquisition unit 201 calculates the position of the extracted person in the still image data SDj. For example, the acquisition unit 201 sets the lower left of the still image data SDj as the origin, and calculates the center of gravity of the extracted person and the center of the face as the position of the person. More specifically, the acquisition unit 201 converts the number of pixels from the origin to the position of the center of gravity into XY coordinates and calculates it as the position of a person.

The acquisition unit 201 delivers the still image data SDj from which the person has been extracted and the calculated person position PPk to the determination unit 202. For example, in the example of FIG. 4, the acquisition unit 201 provides the still image data SDj and the person position PPk of the person 301 included in the still image data SDj to the determination unit 202.

In the acquisition unit 201, various methods can be used as a method for extracting a person included in the still image data SDj and a method for calculating the person position PPk. For example, the acquisition unit 201 uses a learning model learned by a CNN (Convolutional Neural Network) to detect a target object (in this case, a person is detected) from the still image data SDj. Alternatively, the acquisition unit 201 may extract a person by using a method such as template matching.

Return the explanation to Fig. 3. The determination unit 202 determines whether or not to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation. More specifically, when the determination unit 202 acquires the still image data SDj and the person position PPk, the determination unit 202 acquires the success probability in the person position PPk from the storage unit 203.

The storage unit 203 stores at least the success probability for each imaging status. More specifically, the storage unit 203 stores information about the imaging status, the success probability, and the number of trials, which is the number of attempts to execute the image processing in the imaging status, in association with each other.

In the first embodiment, the information regarding the imaging status is information regarding the position of a person in an image in which a person to be image-processed is captured. Specifically, the information regarding the imaging status is information (for example, a coordinate range) that identifies each small area in which the still image data SDj is divided into predetermined areas.

For example, the still image data SDj is divided into a plurality of small areas as shown in FIG. Note that FIG. 5 is an example, and does not mean that the division of the still image data SDj is limited to “9”. Further, the still image data SDj may be divided only in the row direction (horizontal direction) or may be divided only in the column direction (vertical direction). Further, the area of each subregion may be equal or different.

FIG. 6 is a diagram showing an example of information held by the storage unit 203 according to the first embodiment. Referring to FIG. 6, the success probability Pk and the number of trials Tk are stored for each small area (divided area) of the still image data SDj. In this way, the storage unit 203 stores the small area of the still image data SDj, the success probability of the image processing in the small area, and the number of attempts (trial number) of the image processing in the small area in association with each other. ..

The number of trials Tk is the number of times that the image processing unit 205 tried image processing (for example, face recognition processing) in each small area of the still image data SDj. The information (database) stored in the storage unit 203 is updated (added) by the image processing unit 205.

Return the explanation to Fig. 3. The determination unit 202 accesses the storage unit 203 to acquire the success probability Pk and the number of trials Tk of the small area corresponding to the person position PPk. Specifically, the determination unit 202 identifies a small area including the coordinates of the person position PPk, and acquires the success probability Pk and the number of trials Tk from the entry of the specified small area. For example, when the person position PPk of the person 301 is included in the area A shown in FIG. 5 in FIG. 4, the determination unit 202 acquires the success probability P01 and the number of trials R01 (see the first line of FIG. 6).

The determination unit 202 executes threshold processing for the acquired number of trials Tk, and executes image processing (for example, face recognition processing) for an image processing target (for example, a person appearing in the image) according to the result. Judge whether or not. Specifically, when the number of trials Tk is smaller than the trial threshold value, the determination unit 202 determines that the process is executed.

As described above, the information stored in the storage unit 203 is updated with the operation of the system. Therefore, at the start of system operation or the like, a sufficient number of image processes may not be executed for a small area corresponding to the desired person position PPk. The index indicating whether or not a sufficient number of processes have been executed is the "number of trials". If the number of trials is small, the corresponding success probability is judged to be unreliable, and it is judged that the reliability of the success probability needs to be increased.

In order to increase the reliability of the success probability, the determination unit 202 determines that the subsequent processing (image cropping processing, image processing) is executed. That is, it is necessary for the image processing unit 205 to accumulate the results of image processing related to the person position PPk and to make the success probability Pk in the small area of the corresponding still image data SDj highly reliable information. Therefore, when the number of trials is smaller than the trial threshold value, the determination unit 202 determines that the subsequent processing is "processing execution".

If the number of trials Tk is equal to or greater than the trial threshold value, the determination unit 202 determines that the success probability Pk stored in the storage unit 203 is highly reliable data. In this case, the determination unit 202 executes the threshold value processing on the acquired success probability Pk, and determines whether or not to execute the subsequent processing (image cropping processing, image processing) according to the result.

Specifically, if the acquired success probability Pk is equal to or greater than the execution threshold value, the determination unit 202 determines that the process is executed. This is because if the success probability Pk is equal to or greater than the threshold value, there is a high probability that the processing will be completed normally if the image processing is executed at the person position PPk.

On the other hand, when the acquired success probability Pk is smaller than the execution threshold value, the determination unit 202 determines that "processing is not executed". The fact that the success probability Pk is smaller than the threshold value indicates that even if the image processing is executed at the person position PPk, the probability that the processing is normally completed is low.

When the determination unit 202 determines that "processing is not executed", the determination unit 202 does not execute any special processing. When it is determined that "processing is not executed", the determination unit 202 notifies the acquisition unit 201 to that effect. Upon receiving the notification, the acquisition unit 201 shifts the processing target to the next data (next still image data, next person).

When the determination unit 202 determines that "processing is executed", the determination unit 202 notifies the image cutout unit 204 of the image cutout request together with the still image data SDj and the person position PPk.

When the image cutout unit 204 acquires the image cutout request or the like, the image cutout unit 204 cuts out the face image (face image area) of the person existing at the person position PPk in the still image data SDj (extracts the face image). The image cutting unit 204 delivers the person position PPk corresponding to the cut out face image FPk to the image processing unit 205.

As a method of specifying the position of the face existing in the person position PPk in the image cutting unit 204, for example, a method of extracting a face image using CNN can be used as in the acquisition unit 201.

The image processing unit 205 executes image processing on the image processing target when it is determined that the image processing is executed on the image processing target. Specifically, the image processing unit 205 performs a predetermined process (for example, face recognition process) using the cut-out face image FPk and the person position PPk. Since existing techniques can be applied to the calculation of the feature amount (feature vector) required for the face recognition process and the similarity (distance between the feature vectors) required for the collation process, detailed description thereof will be omitted.

The image processing unit 205 updates and adds the information stored in the storage unit 203 according to the processing result (normal end, abnormal end). Specifically, the image processing unit 205 updates the fields of the success probability Pk and the number of trials Tk of the entry corresponding to the image-processed person position PPk (small area of the still image data SDj).

The image processing unit 205 calculates, for example, a case where a predetermined number of feature points (for example, feature points such as eyes and nose) cannot be extracted in the calculation of the feature vector, or a large number of unreliable feature quantities are calculated. If so, the authentication process is determined to be "abnormal termination".

The image processing unit 205 calculates the success probability Pk and the number of trials Tk of the corresponding small area according to the following equations (1) and (2), and updates the information held by the storage unit 203.

 

 

 

Res in the above equation (1) indicates the processing result, and "1" is assigned if the processing ends normally, and "0" is assigned if the processing ends abnormally (error end).

By such processing, the success probability regarding the image processing for each small area of the still image data SDj is accumulated in the storage unit 203.

The operation of the image processing device 20 according to the first embodiment is summarized in the flowchart shown in FIG.

The image processing device 20 extracts the person position PPk from the still image data SDj (step S101).

The image processing device 20 reads out the success probability Pk and the number of trials Tk corresponding to the person position PPk from the storage unit 203 (step S102).

The image processing device 20 determines whether or not the acquired number of trials Tk is equal to or greater than the trial threshold value (step S103).

If the number of trials Tk is smaller than the trial threshold value (step S103, No branch), the image processing apparatus 20 executes the processes after step S106.

If the number of trials Tk is equal to or greater than the trial threshold value (step S103, Yes branch), the image processing apparatus 20 determines whether or not the acquired success probability Pk is equal to or greater than the execution threshold value (step S104).

If the success probability Pk is equal to or greater than the execution threshold value (step S104, Yes branch), the image processing device 20 executes the processes after step S106.

If the success probability Pk is smaller than the execution threshold value (step S104, No branch), the image processing device 20 determines that the image processing such as face recognition processing is "not executed" (step S105).

In step S106, the image processing device 20 sets image processing such as face recognition processing to "execution".

The image processing device 20 executes predetermined image processing (for example, face recognition processing and age / gender determination processing) (step S107).

The image processing device 20 reflects the processing result in step S107 in the database constructed in the storage unit 203 (step S108).

Here, as a result of diligent studies by the inventors, the success probability (accuracy) of image processing depends on the size of the face, the orientation of the face, and the amount of light shining on the face in the still image data SDj. It is known. Factors (parameters) that affect the success probability often depend on the position in the image. For example, in a store, since the display layout is fixed, many people look in a specific direction (for example, the direction in which the product is located) when passing through a specific place, and there is a certain tendency toward the face. Therefore, there is a correlation between the success or failure of the image processing and the position of the person appearing in the image (for example, the position according to the product display layout), and the success or failure of the image processing is biased according to the position of the person. In the first embodiment, as a result of considering the position of the person, if the possibility that the image processing is successful is low, the processing related to the person at the position is not performed, thereby preventing the waste of resources.

As described above, the image processing device 20 according to the first embodiment performs image processing by the determination unit 202 according to the success probability Pk for each imaging situation (person position PPk in the still image data SDj) stored in the storage unit 203. Determine whether to execute. Specifically, the success probability of image processing for the processing target is estimated by a light load of threshold processing, and it is determined whether or not to actually execute the image processing. Further, when sufficient data on the success or failure of the image processing is not accumulated, the image processing is actually executed and the success or failure is reflected in the success probability Pk. As a result, the number of image processing trials per person can be reduced while maintaining the success probability of image processing, so that the number of people that can be processed by the entire system can be increased.

[Second Embodiment]
Subsequently, the second embodiment will be described in detail with reference to the drawings.

In the first embodiment, the person position PPk in the still image data SDj was used as the "imaging situation". In the second embodiment, a case where the time when the image processing target is photographed, that is, the time when the still image data SDj is acquired (current time) is used as the imaging situation will be described.

Since the processing configuration of the image processing apparatus 20 according to the second embodiment can be the same as that of the first embodiment, the description corresponding to FIG. 3 will be omitted.

FIG. 8 is a diagram showing an example of information held by the storage unit 203 according to the second embodiment. As shown in FIG. 8, the storage unit 203 stores the success probability Pk and the number of trials Tk for each time zone.

The acquisition unit 201 delivers the current time CT from which the still image data SDj has been acquired to the determination unit 202 together with the still image data SDj and the person position PPk.

The determination unit 202 acquires the success probability Pk and the number of trials Tk according to the current time CT from the storage unit 203. The determination unit 202 performs processing related to the success probability Pk and the number of trials Tk in the same manner as the contents described in the first embodiment.

Specifically, when the number of trials Tk is smaller than the trial threshold value, the determination unit 202 determines that "processing is executed". If the number of trials Tk is equal to or greater than the trial threshold value and the acquired success probability Pk is equal to or greater than the execution threshold value, the determination unit 202 determines that the process is executed.

Even if the number of trials Tk is equal to or greater than the trial threshold value, if the acquired success probability Pk is smaller than the execution threshold value, the determination unit 202 determines that the process is not executed.

Since the operation of the image processing device 20 after the execution or non-execution of the processing is determined can be the same as that of the first embodiment, detailed description thereof will be omitted. When the face recognition process or the like is performed, the image processing device 20 according to the second embodiment updates the corresponding entry (success probability Pk for each time zone, number of trials Tk) of the storage unit 203.

As described above, in the second embodiment, it is determined whether or not to execute the image processing based on the success probability according to the time when the still image data SDj is acquired. As mentioned above, the success rate of image processing may depend on the amount of light hitting the face. Since the amount of light changes according to the time zone, there may be a phenomenon that the face is easily exposed to light in a specific time zone. In the second embodiment, the success probability of the image processing that changes according to the time zone is taken into consideration, and the image processing is not performed in a situation where the image processing is likely to fail, such as at night. This prevents wasting resources.

[Third Embodiment]
Subsequently, the third embodiment will be described in detail with reference to the drawings.

In the third embodiment, a case where the first and second embodiments are combined will be described.

Since the processing configuration of the image processing apparatus 20 according to the third embodiment can be the same as that of the first and second embodiments, the description corresponding to FIG. 3 will be omitted.

In the third embodiment, a case where the position of a person (person position PPk) reflected in the still image data SDj and the time when the still image data SDj is acquired (current time CT) is used as the “imaging situation” will be described. .. That is, the information regarding the imaging status according to the third embodiment includes the position of the person in the image in which the person to be image-processed is captured, and the time when the person was photographed.

FIG. 9 is a diagram showing an example of information held by the storage unit 203 according to the third embodiment. As shown in FIG. 9, the storage unit 203 stores the success probability Pk and the number of trials Tk in association with each other for each time zone of the small area of the still image data SDj.

The acquisition unit 201 delivers the still image data SDj, the current time CT for acquiring the still image data SDj, and the person position PPk to the determination unit 202.

The determination unit 202 acquires the success probability Pk and the number of trials Tk according to the person position PPk and the current time CT from the storage unit 203. The determination unit 202 performs processing related to the success probability Pk and the number of trials Tk in the same manner as the contents described in the first embodiment.

Specifically, when the number of trials Tk is smaller than the trial threshold value, the determination unit 202 determines that "processing is executed". If the number of trials Tk is equal to or greater than the trial threshold value and the acquired success probability Pk is equal to or greater than the execution threshold value, the determination unit 202 determines that the process is executed.

Even if the number of trials Tk is equal to or greater than the trial threshold value, if the acquired success probability Pk is smaller than the execution threshold value, the determination unit 202 determines that the process is not executed.

Since the operation of the image processing device 20 after the execution or non-execution of the processing is determined can be the same as that of the first and second embodiments, detailed description thereof will be omitted. The image processing device 20 according to the third embodiment updates the corresponding entry (small area, success probability Pk for each time zone, number of trials Tk) of the storage unit 203 when face recognition processing or the like is performed. ..

As described above, the image processing device 20 according to the third embodiment determines whether or not to perform image processing according to the position of the person and the success probability for each time. Therefore, as compared with the first and second embodiments, in the third embodiment, the situation in which the person is photographed can be defined in more detail and the success probability can be set. Therefore, it is possible to determine whether or not to perform accurate image processing. Can be done.

<Hardware configuration>
Subsequently, the hardware of each device constituting the image processing system will be described. FIG. 10 is a diagram showing an example of the hardware configuration of the image processing device 20.

The image processing device 20 can be configured by an image processing device (so-called computer), and includes the configuration illustrated in FIG. For example, the image processing device 20 includes a processor 311, a memory 312, an input / output interface 313, a communication interface 314, and the like. The components such as the processor 311 are connected by an internal bus or the like so that they can communicate with each other.

However, the configuration shown in FIG. 10 does not mean to limit the hardware configuration of the image processing device 20. The image processing device 20 may include hardware (not shown), or may not include an input / output interface 313 if necessary. Further, the number of processors 311 and the like included in the image processing device 20 is not limited to the example of FIG. 10, and for example, a plurality of processors 311 may be included in the image processing device 20.

The processor 311 is a programmable device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor). Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs including an operating system (OS; Operating System).

The memory 312 is a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), an HDD (HardDiskDrive), an SSD (SolidStateDrive), or the like. The memory 312 stores an OS program, an application program, and various data.

The input / output interface 313 is an interface of a display device or an input device (not shown). The display device is, for example, a liquid crystal display or the like. The input device is, for example, a device that accepts user operations such as a keyboard and a mouse.

The communication interface 314 is a circuit, module, or the like that communicates with another device. For example, the communication interface 314 includes a NIC (Network Interface Card) and the like.

The function of the image processing device 20 is realized by various processing modules. The processing module is realized, for example, by the processor 311 executing a program stored in the memory 312. The program can also be recorded on a computer-readable storage medium. The storage medium may be a non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. That is, the present invention can also be embodied as a computer program product. In addition, the program can be downloaded via a network or updated using a storage medium in which the program is stored. Further, the processing module may be realized by a semiconductor chip.

Since the hardware configurations of the camera device 10 and the result storage device 30 are obvious to those skilled in the art, the description thereof will be omitted.

[Modification example]
The configuration, operation, and the like of the image processing system described in the above embodiment are examples, and are not intended to limit the configuration and the like of the system. For example, the processing result of the image processing device 20 may not be stored in the result storage device 30, but may be directly transmitted to a device that uses the result of the image processing. For example, if the image processing is a face recognition processing, the processing result may be transmitted to a device that controls the opening and closing of the gate according to the result of the face recognition.

The image processing system according to the above embodiment may be realized in a cloud environment or an edge environment. When the image processing system is realized in a cloud environment, the image processing device 20 and the result storage device 30 operate as servers on the network. When the image processing system is realized in an edge environment, the image processing device 20 operates as a server on the edge side, and the result storage device 30 operates as a server on the cloud side.

In the above embodiment, the case where the image processing device 20 captures the still image data from the moving image data acquired from the camera device 10 has been described, but the camera device 10 periodically (for example, at 1-second intervals) captures the still image data. It may be transmitted to the processing device 20.

In the above embodiment, the camera device 10 is assumed to be a fixed camera device such as a surveillance camera, but the image (moving image) data input to the image processing device 20 is data acquired from the mobile camera device. It may be. That is, the camera device 10 includes a surveillance camera, a digital camera, a mobile phone, a smartphone, and the like. That is, the "camera device" disclosed in the present application can be any electronic device having a photographing function.

In the above embodiment, a person is set as the target of image processing, but the target of image processing can be arbitrary. For example, an animal may be the target of image processing, or a device (object) such as a robot may be the target of image processing.

In the above embodiment, the authentication process using the face image has been described as a predetermined image process, but the process performed by the image processing unit 205 is not limited to the authentication process. For example, the image processing unit 205 may execute the age / gender determination process. Further, the portion to be processed by the image processing unit 205 is not limited to the “face”. For example, the image processing unit 205 may process a portion such as a “hand” or a “foot”. In this case, the image cutting unit 204 cuts out a portion required for processing by the image processing unit 205.

In the above embodiment, it is assumed that each of the plurality of camera devices 10 included in the image processing system executes the same image processing (for example, face recognition processing), but the implementation is performed according to the camera device 10 for acquiring moving image data. You may change the processing to be performed. For example, face recognition processing is performed on the moving image data (still image data) transmitted by the camera device 10-1, and age / gender determination processing is performed on the moving image data transmitted by the camera device 10-2. May be good. In this case, each camera device 10 transmits an identifier that identifies itself to the image processing device 20 together with the moving image data. In this way, the image processing device 20 may change the content of the image processing according to the attribute (for example, the installation position) of the camera device 10.

In the above embodiment, the position of the person appearing in the image and the time (current time) at the time of image acquisition are set as the "imaging status", but other information may be set as the imaging status. For example, information such as "weather" and "brightness" at the time of image acquisition may be used as the imaging status. For example, in this case, the storage unit 203 stores the success probability Pk for each weather (sunny, rain, cloudy) and the success probability Pk for each brightness. The weather at the time of image acquisition may be acquired from an external server, or the weather may be estimated using a brightness sensor or the like.

In the above embodiment, the result of image processing for each imaging status is accumulated from the time of system operation to improve the reliability of the success probability, but the success probability and the number of trials for each imaging status are acquired before the system is operated. It may be stored in the storage unit 203 in advance. By storing the success probability and the number of trials in the storage unit 203 in advance in this way, it is possible to realize execution determination of image processing based on highly reliable data (success probability) from the start of system operation. That is, by setting the initial value of the storage unit 203 to a value measured in advance, the success probability can be quickly converged to an appropriate value. For example, the image processing device 20 may be provided with a test mode, and a person may stand at various places at various times to perform image processing and use the result as an initial value to be stored in the storage unit 203.

In the above embodiment, the image processing unit 205 calculates the success probability and the number of trials, and updates the contents of the storage unit 203. However, the image processing unit 205 may store only the result of the image processing in the storage unit 203, and the determination unit 202 may calculate the success probability based on the stored result.

In the above embodiment, if the success probability of the imaging status required for the storage unit 203 is not registered, the image processing is determined to be "not performed". However, even in such a case, the required success probability may be predicted from the information stored in the storage unit 203. For example, as shown in FIG. 11, the image processing device 20 may further include a prediction unit 206. For example, the prediction unit 206 predicts the success probability according to the imaging situation to be processed, based on the success probability of the imaging situation different from the imaging situation to be processed. For example, in FIG. 5, the success probabilities of the small area A and the small area C are stored in the storage unit 203 as highly reliable data (the number of trials in each area is equal to or greater than the trial threshold value), but the success probabilities of the small area B Consider the case where is not stored (success probability is 0). In this case, the prediction unit 206 calculates the average value of the success probabilities of the small area A and the small area B adjacent to the small area B, and provides the success probability of the small area B to the determination unit 202.

By installing an image processing program in the storage unit of the computer, the computer can function as an image processing device. Further, by causing the computer to execute the image processing program, the image processing method can be executed by the computer.

Further, in the flowchart used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the order of description. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents, for example, each process is executed in parallel.

Some or all of the above embodiments may also be described, but not limited to:
[Appendix 1]
Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Judgment units (101, 202) that determine whether or not,
When it is determined that the image processing is to be executed on the image processing target, the image processing unit (102, 205) that executes the image processing on the image processing target and
An image processing apparatus (20, 100).
[Appendix 2]
The image processing apparatus according to Appendix 1, wherein the determination unit (101, 202) determines whether or not to execute the image processing on the image processing target based on the result of the threshold value processing for the success probability. 20, 100).
[Appendix 3]
The image processing apparatus (20, 100) according to Appendix 1 or 2, further comprising a storage unit (203) for storing the success probability for each imaging situation.
[Appendix 4]
The storage unit (203) stores information about the imaging status, the success probability, and the number of trials, which is the number of attempts to execute the image processing in the imaging status, in association with each other, as described in Appendix 3. Image processing device (20, 100).
[Appendix 5]
The image processing apparatus (20, 20) according to Appendix 4, wherein the determination unit (101, 202) determines that the image processing is executed on the image processing target when the number of trials is smaller than the trial threshold value. 100).
[Appendix 6]
The description in Appendix 4 or 5, wherein the image processing unit (102, 205) updates the information stored in the storage unit (203) according to the result of attempting the image processing on the image processing target. Image processing device (20, 100).
[Appendix 7]
The image processing apparatus (20, 100) according to any one of Appendix 4 to 6, wherein the information regarding the imaging status includes the position of the image processing target in the image in which the image processing target is captured.
[Appendix 8]
The image processing apparatus (20, 100) according to any one of Supplementary note 4 to 7, wherein the information regarding the imaging status includes the time when the image processing target was photographed.
[Appendix 9]
The image processing apparatus (20, 100) according to any one of Appendix 1 to 8, wherein the image processing target is a human face.
[Appendix 10]
The image processing apparatus (20, 100) according to Appendix 9, further comprising an image cutting section (204) for cutting out the face region to be image processed from the image.
[Appendix 11]
In the image processing apparatus (20, 100)
Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Judge whether or not
An image processing method including executing the image processing on the image processing target when it is determined to execute the image processing on the image processing target.
[Appendix 12]
On the computer (311) mounted on the image processing device (20, 100),
Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. The process of determining whether or not
When it is determined that the image processing is executed on the image processing target, the processing of executing the image processing on the image processing target and the processing of executing the image processing on the image processing target.
A program that executes.
Note that the form of Appendix 11 and the form of Appendix 12 can be expanded to the forms of Appendix 2 to the form of Appendix 10 in the same manner as the form of Appendix 1.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments. It will be appreciated by those skilled in the art that these embodiments are merely exemplary and that various modifications are possible without departing from the scope and spirit of the invention.

This application claims priority based on Japanese application Japanese Patent Application No. 2019-076908 filed on April 15, 2019, and incorporates all of its disclosures herein.

The present invention contributes to executing image processing such as face recognition processing with a low load in an environment where computational resources are limited.

10, 10-1 to 10- n Camera device 20, 100 Image processing device 30 Result storage device 101, 202 Judgment unit 102, 205 Image processing unit 201 Acquisition unit 203 Storage unit 204 Image cutting unit 206 Prediction unit 311 Processor 312 Memory 313 I / O interface 314 communication interface

Claims (12)

1. Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Judgment unit to judge whether or not
   An image processing unit that executes the image processing on the image processing target when it is determined to execute the image processing on the image processing target.
   An image processing device.
2. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not to execute the image processing on the image processing target based on the result of the threshold value processing for the success probability.
3. The image processing device according to claim 1 or 2, further comprising a storage unit that stores the success probability for each imaging situation.
4. The image according to claim 3, wherein the storage unit stores information about the imaging status, the success probability, and the number of trials, which is the number of attempts to execute the image processing in the imaging status, in association with each other. Processing equipment.
5. The image processing apparatus according to claim 4, wherein the determination unit determines that the image processing is executed on the image processing target when the number of trials is smaller than the trial threshold value.
6. The image processing device according to claim 4 or 5, wherein the image processing unit updates the information stored in the storage unit according to the result of attempting the image processing on the image processing target.
7. The image processing apparatus according to any one of claims 4 to 6, wherein the information regarding the imaging status includes the position of the image processing target in the image in which the image processing target is captured.
8. The image processing apparatus according to any one of claims 4 to 7, wherein the information regarding the imaging status includes the time when the image processing target was photographed.
9. The image processing device according to any one of claims 1 to 8, wherein the image processing target is a human face.
10. The image processing apparatus according to claim 9, further comprising an image cutting portion that cuts out the face region to be image processed from the image.
11. In the image processing device
    Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. Judge whether or not
    An image processing method including executing the image processing on the image processing target when it is determined to execute the image processing on the image processing target.
12. For computers installed in image processing equipment
    Whether to execute the image processing on the image processing target based on the success probability when the image processing is executed on the image processing target for each imaging situation, which is the situation when the image processing target is imaged. The process of determining whether or not
    When it is determined that the image processing is executed on the image processing target, the processing of executing the image processing on the image processing target and the processing of executing the image processing on the image processing target.
    A program that executes.
    
    

Images