JP2015139165A - Image processing system, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently and appropriately perform image recognition processing in accordance with an environment.SOLUTION: Image recognition processing for recognizing an object from image data is performed in accordance with an input of image data from an imaging apparatus. Stop and resumption of the image recognition processing are controlled in accordance with a switchover of an imaging mode for switching over an imaging mode of the imaging apparatus.

Description

  The present invention relates to an image processing technique for processing image data obtained by photographing with a photographing apparatus.

  In recent years, image distribution systems that use IP (Internet Protocol) networks such as the Internet have been increasing, and they have been adopted for Internet sites that distribute the status of facilities such as ski resorts and zoos, as well as monitoring operations such as store monitoring and building monitoring. Has been.

  For example, in store monitoring and building monitoring, a photographed image is analyzed / recognized to detect a specific event, and when the event occurs, an image is accumulated, or an administrator views / confirms it. Yes.

  Specific events for image recognition include, for example, object detection for detecting whether or not an object has appeared on a captured image, detection of detection of a suspicious object being placed, detection of removal of a monitoring target, etc. For example, there is a tampering detection for detecting whether or not a network camera is tampered with from a captured image.

  In addition, the image recognition method includes a background difference that creates a background image based on a photographed image at a certain time in the past or a photographed image from the past to the present, and performs object recognition based on a difference from the current photographed image. Exists.

  At this time, the background image to be used has a large difference from the background if the captured image changes suddenly due to changes in the external environment such as a fluorescent lamp, camera day / night function, infrared illumination, focus, white balance, aperture, etc. It will lead to false detection. For this reason, when the captured image changes abruptly, it is necessary to create the background image again.

  For example, Patent Document 1 discloses a method of updating a background image when switching on / off of infrared illumination. Patent Document 2 discloses a method of updating a photographing mode and a reference image by comparing luminance components of a reference image and a current image.

JP 2010-50637 A JP 2013-13123 A

  However, in the prior art disclosed in the above-mentioned patent document, the time taken for switching the shooting parameters and the switching order are not considered, and if only the luminance on the image is stable, the background image is recreated. End up.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing technique capable of executing an image recognition process efficiently and appropriately according to an environment.

In order to achieve the above object, an image processing apparatus according to the present invention comprises the following arrangement. That is,
An image processing device that processes image data obtained by photographing with a photographing device,
Recognizing means for performing image recognition processing for recognizing an object from the image data in response to input of image data from the imaging device;
Switching means for switching the photographing mode of the photographing device;
Control means for controlling stop and restart of the image recognition process in response to switching of the photographing mode by the switching means.

  According to the present invention, it is possible to execute image recognition processing efficiently and appropriately according to the environment.

It is a system configuration diagram. It is a hardware block diagram of a camera server. It is a software block diagram of a camera server. FIG. 3 is a sequence diagram in the first embodiment. 3 is a flowchart of an image recognition program in the first and second embodiments. FIG. 10 is a sequence diagram in the second embodiment. FIG. 10 is a sequence diagram in the third embodiment. 10 is a flowchart of an image recognition program in Embodiment 3.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the embodiment.

  A camera server 200 and a client 220 are connected to each other via a network 230. The camera server 200 includes a camera with a variable shooting angle of view or a fixed shooting angle of view, and can distribute image data shot by the camera via the network 230. The client 220 can access the camera server 200 and acquire image data. Details of the operations of the camera server 200 and the client 220 will be described later.

  In FIG. 1, for simplicity of explanation, one camera server 200 is used, but two or more camera servers 200 may be used. In addition to the client 220, there may be other clients that access the camera server 200 to receive and store image data.

  The network 230 includes a plurality of network devices such as routers, switches, cables, and lines that satisfy communication standards such as Ethernet (registered trademark). In the present embodiment, the communication standard, scale, and configuration are not limited as long as communication between each server and client can be performed without any problem. That is, the network 230 can be applied from the Internet to a LAN (Local Area Network).

  FIG. 2 is a hardware configuration diagram of the camera server 200.

  The camera server 200 includes a CPU 300, a primary storage device 310, a secondary storage device 320, an image capture I / F 330, and a network I / F 360, and these components are connected to each other via an internal bus 301.

  Here, the primary storage device 310 is a writable high-speed storage device represented by a RAM, and is loaded with an OS, various programs, and various data, and is also used as a work area for the OS and various programs. . The secondary storage device 320 is a nonvolatile storage device represented by FDD, HDD, flash memory, CD-ROM drive, etc. In addition to being used as a permanent storage area for the OS, various programs, and various data, It is also used as a storage area for various short-term data. The details of data such as various programs stored in the primary storage device 310 and the secondary storage device 320 of the camera server 200 will be described later.

  The image capture I / F 330 is an I / F (interface) for connecting an image input device. Here, a camera 370 that is a photographing device is connected. The image capture I / F 330 inputs image data captured by the camera 370, converts and compresses the input image data into a predetermined format (for example, JPEG, MPEG, etc.), and transfers the image data to the primary storage device 310.

  The network I / F 360 is an I / F for connecting to the network 230, and is responsible for communication with other terminals such as the client 220 via the network 230.

  FIG. 3 is a software configuration diagram of the camera server 200.

  On the primary storage device 310, an OS 400, a shooting processing program 410, a shooting mode management program 411, an image recognition program 412, a distribution processing program 414, and a temporary storage unit 413 are loaded.

  The OS 400 is a basic program that controls the entire camera server 200. The imaging processing program 410 acquires and encodes the image data generated by the camera 370 via the image capture I / F 330 and stores it in the temporary storage unit 413. Here, the image data stored in the temporary storage unit 413 is image data after encoding, but it may be image data before encoding or both before and after encoding. .

  The shooting mode management program 411 refers to the image data stored in the temporary storage unit 413, and instructs to change a plurality of types of shooting parameters such as day / night, focus, white balance, and infrared illumination automatically or from the user / client 220. Is controlled (changed) continuously or simultaneously in parallel with the value designated by the above, and the state change is notified to the image recognition program 412 as necessary.

  Note that the camera server 200 may be provided with a console (not shown) for inputting (changing instructions) shooting parameters. This shooting parameter can be input from the client 220 via the network 230. In addition, the photographing mode includes a plurality of types of modes such as a day mode for photographing under a visible light environment and a night mode for photographing under an infrared light environment.

  The image recognition program 412 creates background image data from one or more image data stored in the temporary storage unit 413. The image recognition program 412 performs the image recognition process by comparing the background image data with the latest image data while updating the background image data with the latest image data even after the creation of the background image data.

  The image recognition process by the image recognition program 412 divides the background image data and the latest image data into small regions (regions of a predetermined size) each having a constant resolution, compares the corresponding small regions, and compares the small regions with differences. To recognize objects. Further, the image recognition program 412 detects specific events such as moving object detection, leaving detection, removal detection, and tampering detection by measuring feature quantities such as a movement trajectory of the recognized object over time. Then, the image recognition program 412 stores the object recognition information and the specific event information generated by the image recognition processing in the temporary storage unit 413 as image recognition information.

  In response to a request from the client 220, the distribution processing program 414 controls the network I / F 360 and the image data stored in the temporary storage unit 413 and, if necessary, the image recognition information, and the client 220 via the network 230. Send to.

  Cooperation between the programs is realized using a function provided by the OS 400 as necessary.

  Various programs operating on the OS 400 are realized by the CPU 300 reading and executing on the primary storage device 310.

  The client 220 has the same hardware configuration as that of the camera server 200. That is, the client 220 that is an information processing apparatus has standard components (for example, a CPU, a RAM, a ROM, a hard disk, an external storage device, a network interface, a display, a keyboard, and a mouse) mounted on a general-purpose computer. ing.

  In each of the following embodiments, an imaging mode including a change in a plurality of types of imaging conditions such as infrared illumination ON / OFF, imaging parameter (day / night, focus, white balance) change (parameter state change), timeout for a predetermined time, etc. A configuration for controlling the stop and restart of the image recognition processing in accordance with the switching will be described. More specifically, control is performed so that the image recognition process is started and then restarted based on at least one of the start time and the end time of the change of the shooting conditions accompanying the switching of the shooting mode. Details of an example of the start and restart control will be described in the following embodiments.

<Embodiment 1>
The image processing apparatus according to the first embodiment will be described below.

  FIG. 4 is a diagram illustrating a processing sequence example of the camera server 200 according to the first embodiment.

  In the first embodiment, during the image recognition process, the shooting mode management program 411 detects that the shooting environment is dark, so that infrared light is emitted from the day mode for shooting in a visible light environment to emit infrared light. A case will be described in which the white balance is changed to an appropriate state by changing to the night mode for shooting in an environment.

  Note that the detection of the dark shooting environment is, for example, dark when the light detection sensor built in the camera 370 detects that the shooting environment of the camera server 200 is below a certain illuminance (brightness / luminance). It is determined that this is an illuminance environment.

  First, in step S500, the imaging processing program 410 saves the image data in the temporary storage unit 413, and transmits an image data generation notification indicating that the image data has been generated to the image recognition program 412. The image data generation notification is also transmitted to the shooting mode management program 411, but is omitted in this figure.

  At this point, it is assumed that the image recognition program 412 already has background image data. Therefore, in step S520, the image recognition program 412 performs image recognition processing using the received image data, and stores the image recognition information obtained by the image recognition processing in the temporary storage unit 413.

  Here, steps S501 to S511 indicate that the photographing processing program 410 generates image data that is temporally continuous at a constant interval and transmits a corresponding image data generation notification.

  Assume that the shooting mode management program 411 initially determines that the shooting area is sufficiently bright from the image data stored in the temporary storage unit 413 (step S530). In other words, it is assumed that the irradiation with infrared illumination is not executed, and the day mode is for photographing a visible light environment.

  In step S531, if the shooting mode management program 411 determines that the brightness of the shooting area is darker than a predetermined value from the latest image data, the shooting mode change notification for changing the shooting mode from the day mode to the night mode is displayed in the image recognition program 412. Send to. In the first embodiment, the imaging mode change notification includes a message indicating the start of infrared illumination irradiation (illumination ON start). In this embodiment, the brightness of the shooting environment is determined based on the brightness of the shooting area from the image data. However, the brightness of the shooting environment is determined by another measurement method such as an illuminance sensor built in the camera 370. The thickness may be measured / calculated.

  When the image recognition program 412 receives the message indicating the illumination ON start by the shooting mode change notification and determines that the shooting mode has been changed, in step S521, the image recognition program 412 stops the subsequent image recognition processing. Transition to the state to do. In the present embodiment, when the shooting condition is changed, it is determined that the shooting mode is changed. As a result, the image recognition process is stopped during steps S502 to S507, and therefore the image recognition program 412 discards the image data generation notification received from the photographing process program 410.

  In step S532, the shooting mode management program 411 starts the night mode transition start notification indicating that the transition to the night mode for shooting in the infrared light environment has started after starting the irradiation of the infrared illumination. 412 is transmitted. In step S533, the shooting mode management program 411 provides the image recognition program 412 with an illumination end notification indicating that the infrared illumination has been terminated when the intensity of the infrared illumination reaches the target level. In step S534, the shooting mode management program 411 transmits to the image recognition program 412 a notice of the end of night mode transition indicating that the transition to night mode has ended.

  Thereafter, in step S535, the shooting mode management program 411 transmits to the image recognition program 412 a white balance operation start notification indicating that the white balance operation for achieving an appropriate white balance (WB) has started. In step S536, the shooting mode management program 411 transmits a white balance operation end notification indicating that the white balance operation is ended by reaching an appropriate white balance to the image recognition program 412.

  Note that the notification transmitted in each of steps S531 to S536 is transmitted to the image recognition program 412 by the shooting mode management program 411 as one of the messages included in the shooting mode change notification.

  The image recognition program 412 receives the notification of the end of the white balance operation, thereby satisfying the end condition for ending the operation of canceling the image data generation notification from the shooting processing program 410 (the image recognition processing stop state). Check. In other words, the image recognition program 412 confirms that the change of the shooting mode has been completed. As described above, the image recognition program 412 receives the notification of the end of the white balance operation, thereby completing the irradiation of the infrared illumination, the transition to the night mode, and the end of the white balance operation. Confirm that the completion of changing the shooting conditions is satisfied as an end condition. Thereby, the change of the photographing mode including the change of the photographing condition is completed, and the image recognition program 412 shifts the state during the initialization of the background image data in step S522.

  In the first embodiment, the three end conditions are infrared illumination, transition to the night mode, and white balance change. However, the number of end conditions is not limited to this. The number can be increased / decreased according to the use and purpose, and the end condition can be added with the end of the adjustment of the photographing parameters such as focus, aperture, gain adjustment, and the like as the end condition.

  In step S508, step S509, and step S510, the image recognition program 412 receives the image data, and creates background image data from the plurality of image data. By doing so, in step S523, the image recognition program 412 completes the creation of the background image data. As a result, the image recognition program 412 can transition to a state where image recognition processing can be started.

  In step S511, the image recognition program 412 receives the next image data after completing the creation of the background image data. Accordingly, in step S524, the image recognition program 412 executes image recognition processing using the received image data. That is, the image recognition program 412 performs image recognition processing by comparing the image data input from the photographing processing program with the background image data. The background image data is image data used for comparison with image data input from a photographing processing program in image recognition processing.

  FIG. 5 is a flowchart illustrating processing executed by the image recognition program according to the first embodiment.

  The image recognition program 412 is activated when the power is turned on or when an external terminal such as the client 220 is first requested to start image recognition processing.

  In step S101, the image recognition program 412 sets the program state during initialization of the background image data.

  In step S102, the image recognition program 412 waits for an event in order to receive a notification message from another program. In the first embodiment, the image recognition program 412 waits for image data generation notification from the shooting processing program 410 or shooting mode change notification from the shooting mode management program 411. The shooting mode change notification includes change information of various shooting parameters such as day / night, focus, white balance, and infrared illumination.

  When the notification message is received, the contents of the event are determined in step S103, and each process is executed based on the determination result.

  As a result of the determination in step S103, when the image data generation notification is received from the shooting processing program 410, in step S104, the image recognition program 412 first determines the state.

  If the result of determination in step S104 is that the state is in the process of initializing background image data, in step S105, the image recognition program 412 creates background image data. Since the background image data is generally created from a plurality of image data, the image recognition program 412 generates the background image data by receiving a predetermined number or more of the image data from the imaging processing program 410. Become. That is, the image recognition program 412 can create background image data for the first time by repeating the processing of step S103 to step S106. Therefore, in step S106, the image recognition program 412 determines whether the creation of the background image data has been completed. If completed (YES in step S106), in step S107, the image recognition program 412 sets the status to “operation” and returns to waiting for the event in step S102. On the other hand, if not completed (NO in step S106), the process returns to the event waiting in step S102.

  If the result of determination in step S104 is that the status is in operation, in step S108, the image recognition program 412 executes image recognition processing. In step S108, the image recognition program 412 stores object recognition information and specific event information obtained by the image recognition processing in the temporary storage unit 413 as image recognition information, and transmits a message to the distribution processing program 414 as necessary. To do.

  If the result of determination in step S104 is that the status is stopped, nothing is done and the process returns to waiting for the event in step S102.

  As a result of the determination in step S103, when the shooting mode change notification is received from the shooting mode management program 411, in step S109, the image recognition program 412 first checks the state.

  In step S109, if the state is initializing or operating the background image data, in step S110, the image recognition program 412 determines whether or not the shooting mode switching is started based on the content of the shooting mode change notification. to decide. In the first embodiment, when the shooting mode change notification includes a message indicating the start of infrared illumination ON, it is determined that the shooting mode switching is started. In the first embodiment, it is determined that the infrared illumination ON start is the photographing mode switching start, but the present invention is not limited to this. If it is within the target range, it may be determined that shooting mode switching is started even when a message indicating day / night, focus, white balance, and other shooting parameters are included in the shooting mode change notification.

  If it is determined in step S110 that shooting mode switching has started (YES in step S110), in step S111, the image recognition program 412 changes the state to stop. On the other hand, when it is determined that the shooting mode switching is not started (NO in step S110), nothing is done and the process returns to the event waiting in step S102.

  If the result of determination in step S109 is that the status is stopped, in step S112, the image recognition program 412 determines whether or not shooting mode switching has been completed. In the first embodiment, when the shooting mode change notification includes a message indicating the end of the white balance operation, it is determined that the shooting mode switching has been completed. In the present embodiment, the end of the white balance operation is determined to be the completion of shooting mode switching, but the present invention is not limited to this. If it is within the target range, it may be determined that the shooting mode switching is complete even when a message indicating day / night, focus, white balance, and other shooting parameters are included in the shooting mode change notification.

  As a result of the determination in step S112, if it is determined that shooting mode switching is complete (YES in step S112), in step S113, the image recognition program 412 discards the background image data. In step S114, the image recognition program 412 changes the state during initialization of the background image data, and returns to waiting for an event in step S102. On the other hand, if it is determined that shooting mode switching has not been completed (NO in step S112), nothing is done and the process returns to waiting for an event in step S102.

  In the first embodiment, only the start of shooting mode switching and the completion of shooting mode switching are determined. However, the present invention is not limited to this, and changes in all shooting parameters may be determined.

  As described above, according to the first embodiment, the image recognition process is stopped at the start of shooting mode switching, a plurality of shooting parameters are changed, and recreation of background image data is started after the shooting mode switching is completed. As a result, even when multiple shooting parameters of the camera are changed due to changes in the shot image data, erroneous detection of the image recognition process is suppressed, and background image data is recreated at an appropriate timing and the image recognition process is continued. can do.

<Embodiment 2>
The operation of the image processing apparatus according to the second embodiment will be described.

  FIG. 6 is a diagram illustrating a processing sequence example of the camera server 200 according to the second embodiment.

  Also in the second embodiment, in the same way as the first embodiment, during the image recognition process, the shooting mode management program 411 detects that the shooting environment is dark, so that the infrared illumination is changed from the day mode for shooting in the visible light environment. Will be described, and the white balance is changed to an appropriate state by changing to the night mode for photographing in an infrared light environment.

  In FIG. 6, the image data generation notification by the photographing processing program 410 in steps S600 to S611 corresponds to steps S500 to S511 of the first embodiment, and thus details thereof are omitted.

  Assume that the image recognition program 412 already has background image data when it receives the image data generation notification from the imaging processing program 410 in step S600. Therefore, in step S620, the image recognition program 412 performs image recognition processing using the received image data, and stores image recognition information obtained by the image recognition processing in the temporary storage unit 413.

  Assume that the shooting mode management program 411 initially determines that the shooting area is sufficiently bright from the image data stored in the temporary storage unit 413 (step S630). In other words, it is assumed that the irradiation with infrared illumination is not executed, and the day mode is for photographing a visible light environment.

  If the shooting mode management program 411 determines in step S631 that the brightness of the shooting area is darker than a predetermined value from the latest image data, the shooting mode change notification for changing the shooting mode from the day mode to the night mode is displayed in the image recognition program 412. Send to. In the second embodiment, the imaging mode change notification includes a message indicating completion of infrared illumination (illumination ON). That is, in the second embodiment, the shooting mode management program 411 notifies the image recognition program 412 of the state in which the irradiation of infrared illumination is completed. This indicates that no erroneous detection occurs in the image recognition process even if the time from the start to completion of irradiation of infrared illumination is not taken into consideration when executing the image recognition process.

  In the second embodiment, the brightness of the shooting environment is determined based on the brightness of the shooting area from the image data. However, the brightness of the shooting environment is determined by another measurement method such as an illuminance sensor built in the camera 370. The thickness may be measured / calculated.

  When the image recognition program 412 receives the message indicating the illumination ON completion by the shooting mode change notification and determines the change of the shooting mode, in step S621, the image recognition program 412 stops the subsequent image recognition processing. Transition to the state to do. As a result, the image recognition process is in a stopped state between step S602 and step S607, and therefore the image recognition program 412 discards the image data generation notification received from the photographing process program 410.

  In step S632, the photographing mode management program 411 recognizes the completion of the transition to the night mode indicating that the transition to the night mode for photographing in the infrared light environment is completed after the irradiation of the infrared illumination is completed. Send to program 412. Thereafter, in step S633, the shooting mode management program 411 transmits to the image recognition program 412 a notification of the end of the white balance operation indicating that the white balance operation has ended by reaching an appropriate white balance.

  Note that the notifications transmitted in steps S631 to S633 are all transmitted as one of the messages included in the shooting mode change notification by the shooting mode management program 411 to the image recognition program 412.

  In the second embodiment, when executing the image recognition process, an error may occur in the image recognition process without considering the time from the start to the completion of the transition to the night mode and the time from the start to the completion of the white balance change. An example in which no detection occurs is shown.

  The image recognition program 412 receives the notification of the end of the white balance operation, thereby confirming that the end condition for ending the operation of discarding the image data generation notification from the photographing processing program 410 is satisfied. As described above, the image recognition program 412 receives the notification of the end of the white balance operation, thereby completing the irradiation of the infrared illumination, the transition to the night mode, and the end of the white balance operation. Confirm that the termination condition is satisfied. Thereby, in step S622, the image recognition program 412 shifts the state during the initialization of the background image data.

  In the second embodiment, the three end conditions are infrared illumination, transition to the night mode, and white balance change. However, the number of end conditions is not limited to this. The number can be increased / decreased depending on the application and purpose, and the contents of the conditions can be added as completion conditions for completion of adjustment of imaging parameters such as focus, aperture, and gain adjustment.

  In step S608, step S609, and step S610, the image recognition program 412 receives the image data, and creates background image data from the plurality of image data. By doing so, in step S623, the image recognition program 412 completes the creation of the background image data. As a result, the image recognition program 412 can transition to a state where image recognition processing can be started.

  In step S611, the image recognition program 412 receives the next image data after completing the creation of the background image data. Accordingly, in step S624, the image recognition program 412 executes image recognition processing using the received image data.

  The processing executed by the image recognition program 412 in the second embodiment is in accordance with the flowchart of FIG. Therefore, here, only processing unique to the second embodiment will be described with reference to FIG. 5, and operations and processing common to those in FIG. 5 will be omitted as appropriate.

  The second embodiment is the same as the first embodiment except that the determination content in step S109 is different.

  In step S109, if the state is initializing or operating the background image data, in step S110, the image recognition program determines whether or not shooting mode switching is started based on the content of the shooting mode change notification. To do. In the second embodiment, when a message indicating completion of infrared illumination ON is included in the shooting mode change notification, it is determined that the shooting mode switching is started. In the second embodiment, it is determined that the infrared illumination ON completion is the photographing mode switching start, but the present invention is not limited to this. If it is within the target range, it may be determined that the shooting mode switching is completed also when a message indicating day / night, focus, white balance, and other shooting parameters is included in the shooting mode change notification.

  In the second embodiment, the start / end of shooting mode switching is determined at the end of changing the shooting parameter (white balance). However, the present invention is not limited to this. For example, whether it is necessary to consider both the start and end of the change of the shooting parameter or only the end of the change may be determined individually depending on the characteristics of the shooting parameter. That is, an embodiment in which the first embodiment and the second embodiment are combined may be realized.

  As described above, according to the second embodiment, when there is no problem in ignoring the transition time of the shooting parameter in the image recognition process, the image recognition process is performed after detecting that the change of the shooting parameter is completed. Stop and start recreating the background image data after completing a plurality of shooting parameter changes. Even in such a configuration, it is possible to suppress erroneous detection of the image recognition processing due to the change in the captured image data, and to re-create the background image data at an appropriate timing and continue the image recognition processing.

<Embodiment 3>
The operation of the image processing apparatus according to the third embodiment will be described.

  FIG. 7 is a diagram illustrating a processing sequence example of the camera server 200 according to the third embodiment.

  Also in the third embodiment, in the same way as the first embodiment, during the image recognition process, the shooting mode management program 411 detects that the shooting environment is dark, so that the infrared illumination is changed from the day mode for shooting in the visible light environment. Will be described, and the white balance is changed to an appropriate state by changing to the night mode for photographing in an infrared light environment.

  Note that the operation in FIG. 7 of the third embodiment is basically the same as the operation in FIG. 6 of the second embodiment. That is, step S700 to step S711, step S720 to step S724, and step S730 to step S732 in FIG. 7 correspond to step S600 to step S611, step S620 to step S624, and step S630 to step S632 in FIG. Therefore, in the following description, differences from the second embodiment will be described.

  In the third embodiment, in step S721, the image recognition program 412 shifts to a state in which the subsequent image recognition processing is stopped, and starts a timer for detecting the completion of shooting mode switching.

  Thereafter, the image recognition program 412 receives a notification of the end of the white balance operation from the shooting mode management program 411. In the third embodiment, unlike the second embodiment, no processing is executed at this point. When a timer for detecting completion of shooting mode switching occurs, in step S722, the image recognition program 412 shifts the state during initialization of the background image data. This timeout value is set in advance according to the application and purpose.

  In step S710, step S711, and step S712, the image recognition program 412 receives the image data and creates background image data from the plurality of image data. By doing so, in step S723, the image recognition program 412 completes the creation of the background image data. As a result, the image recognition program 412 can transition to a state where image recognition processing can be started.

  In step S713, the image recognition program 412 receives the next image data after completing the creation of the background image data. Thereby, in step S724, the image recognition program 412 executes an image recognition process using the received image data.

  FIG. 8 is a flowchart illustrating processing executed by the image recognition program according to the third embodiment.

  In FIG. 8, the detailed description of the same processing as that in the flowchart of FIG. In particular, steps S801 to S807 in FIG. 8 correspond to steps S101 to S807 in FIG.

  After step S801 and step S802, in step S803, the image recognition program 412 waits for an event in order to receive a notification message from another program. In the third embodiment, the image recognition program 412 waits for image data generation notification from the shooting processing program 410, shooting mode change notification from the shooting mode management program 411, or timeout notification from the image recognition program 412 itself.

  As a result of the determination in step S803, when the shooting mode change notification is received from the shooting mode management program 411, in step S809, the image recognition program 412 first checks the state.

  If it is determined in step S809 that the background image data is being initialized or is in operation, in step S810, the image recognition program determines whether or not shooting mode switching is started based on the content of the shooting mode change notification. To do. In the third embodiment, when the message indicating the completion of infrared illumination is included in the shooting mode change notification, it is determined that the shooting mode switching is started. In the third embodiment, the completion of infrared illumination ON is determined to be the start of shooting mode switching, but the present invention is not limited to this. If it is within the target range, it may be determined that the shooting mode switching is completed also when a message indicating day / night, focus, white balance, and other shooting parameters is included in the shooting mode change notification.

  If it is determined in step S810 that shooting mode switching has started (YES in step S810), in step S811, the image recognition program 412 starts a timer for measuring the completion of shooting mode switching. In step S812, the image recognition program 412 changes the state to stop. On the other hand, when it is determined that the shooting mode switching is not started (NO in step S810), nothing is done and the process returns to waiting for an event in step S802.

  If the result of determination in step S809 is that the state is stopped, in step S813, the image recognition program 412 determines again whether or not shooting mode switching has been started. If it is determined again that shooting mode switching is started (YES in step S813), the timer time-out time is updated to start from the current time in step S814. On the other hand, if it is determined that the shooting mode switching is not started (NO in step S813), nothing is done and the process returns to waiting for an event in step S802.

  As a result of the determination in step S803, if a time-out notification indicating completion of shooting mode switching from the image recognition program 412 itself is received / detected, in step S815, the image recognition program 412 first checks the state.

  If the state is in the process of initializing or operating the background image data in step S815, nothing is done and the process returns to waiting for the event in step S802. On the other hand, if the state is stopped, the image recognition program 412 discards the background image data in step S816. In step S817, the image recognition program 412 changes the state during initialization of the background image data, and returns to waiting for an event in step S802.

  In the third embodiment, a case is described in which the time-out time in the entire shooting mode is used. However, a time-out time is provided for each shooting parameter, and all time-out times have elapsed (a time after a certain time has elapsed). It may be determined that the shooting mode switching has been completed. In addition to the shooting parameter timeout period, the shooting parameter end condition and the maximum waiting time may be specified. For example, the end condition of the white balance operation can be specified as a first predetermined time (for example, 1 second) or more, and the maximum waiting time can be specified as a second predetermined time (for example, 30 seconds).

  As described above, according to the third embodiment, when the total transition time of all the shooting parameters to be changed at the time of shooting mode switching does not change greatly, after waiting for a certain time from the start of shooting mode switching, the background image data Start rebuilding. Even in such a configuration, it is possible to suppress erroneous detection of the image recognition processing due to the change in the captured image data, and to re-create the background image data at an appropriate timing and continue the image recognition processing.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An image processing device that processes image data obtained by photographing with a photographing device,
Recognizing means for performing image recognition processing for recognizing an object from the image data in response to input of image data from the imaging device;
Switching means for switching the photographing mode of the photographing device;
An image processing apparatus comprising: control means for controlling stop and restart of the image recognition processing in response to switching of the photographing mode by the switching means. The control means controls to stop the image recognition process based on at least one of the start time and the end time of the change of the shooting parameter of the shooting apparatus accompanying the switching of the shooting mode, and then restart the image recognition process. The image processing apparatus according to claim 1, wherein: The switching means changes a plurality of types of shooting parameters for the shooting device continuously or simultaneously in parallel as the shooting mode switching,
The control means controls the image recognition process to be stopped and then restarted with at least one of the change of the shooting parameter at the start and end of the change of the shooting parameter as the shooting mode is switched. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The image processing apparatus according to claim 3, wherein the change timing of the shooting parameter is at least one of a start time and an end time of the change of the value of the shooting parameter. The completion of the change of each of a plurality of types of imaging parameters of the imaging apparatus in accordance with the switching of the imaging mode, as an end condition for ending the stop state of the image recognition processing by the control means, the control means, The image processing apparatus according to claim 1, wherein the image recognition process is resumed. The image recognition process compares background image data created from at least one of a plurality of pieces of image data input continuously in time at regular intervals and the latest image data of the plurality of pieces of image data. Is a process of recognizing an object included in the image data and detecting a specific event,
6. The control unit according to claim 1, wherein when the image recognition process is resumed, the image recognition process is resumed after the creation of the background image data is completed. An image processing apparatus according to 1. The control means stops the image recognition process at the start of switching of the shooting mode, and restarts the image recognition process with a time after a certain time has elapsed from the start as the end of switching of the shooting mode. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The apparatus further comprises an ending condition for ending an operation for changing each of a plurality of types of imaging parameters of the imaging apparatus according to the switching of the imaging mode, and designation means for designating a maximum waiting time for the operation. The image processing apparatus according to any one of 1 to 7. A control method for an image processing apparatus for processing image data obtained by photographing with a photographing apparatus,
A recognition step of performing image recognition processing for recognizing an object from the image data in response to input of image data from the imaging device;
A switching step of switching the photographing mode of the photographing device;
A control step of controlling stop and restart of the image recognition processing in response to switching of the photographing mode by the switching step. A program for causing a computer to control the control of an image processing device that processes image data obtained by photographing with a photographing device,
The computer,
Recognizing means for performing image recognition processing for recognizing an object from the image data in response to input of image data from the imaging device;
Switching means for switching the photographing mode of the photographing device;
A program that functions as control means for controlling stop and restart of the image recognition processing in response to switching of the photographing mode by the switching means.

Images