JP6809114B2 - Information processing equipment, image processing system, program - Google Patents

Description

The present invention relates to an information processing device, an image processing system and a program.

Data and services on the Internet have increased sharply due to improvements in the communication environment and reductions in communication costs. As a result, a service form called a cloud service has become known. A cloud service is a form of service in which a user receives a service via the Internet and pays a service fee as needed. Since services on the Internet such as cloud services can collect data from all over the world, it is practically possible to use a large amount of data called big data, and data processing utilizing this has become active. ing.

In addition, the number of hardware connected to the Internet, such as smartphones, has increased rapidly, and the concept of the Internet of Things (IoT) was born. By connecting things to the Internet, cloud services will be able to provide services that require things, and service providers will be able to easily provide a wider variety of cloud services. ..

One of the cloud services is the analysis of images captured by an imaging device. Development of image recognition processing algorithms that identify people and people in images and identify general names of objects in images is in progress. Once the image is input, intelligent information can be obtained by using these algorithms. This algorithm can be executed by a processing chip built in the image pickup device that captures an image, but can also be executed on the cloud as a cloud service as described above.

In cloud services, WebAPI is known as a mechanism for connecting things and the Internet via communication. A lot of information can be extracted from the image by making the image captured by the image pickup device a target of image recognition by WebAPI. For example, the cloud service can extract various information according to the purpose such as the presence / absence and number of people, the presence / absence and number of vehicles, the presence / absence of obstacles, and the state of traffic lights.

It is said that human vision accounts for 70% of the total sense, but if there is a cloud service WebAPI that provides intelligent visual information, users can easily create advanced applications by appropriately combining other WebAPIs. ..

Such image recognition can also be performed on moving images (see, for example, Patent Document 1). Patent Document 1 discloses a method of performing recognition processing using a frame most suitable for character recognition among frames before and after performing recognition processing.

However, the conventional technique has a problem that the recognition accuracy of moving images may be lowered. That is, it is necessary to transmit a moving image from an imaging device or the like to the cloud side, but the amount of moving image data is very large, a wide band is required, and the communication cost is high. In order to reduce the communication cost, the moving image may be compressed with a high compression rate, but if the compression rate is high, the image may be deteriorated and the accuracy of image recognition may be lowered. Even if the image recognition process is performed only on the moving image stored in the storage device without transmitting the moving image, if the compression rate is low, the capacity is compressed, and if the compression rate is high, the accuracy of image recognition may decrease.

An object of the present invention is to provide an information processing device capable of suppressing a decrease in recognition accuracy of moving images.

The present invention is an information processing device that recognizes an object from an image, and determines whether or not the moving image developing means for expanding the moving image data into a plurality of still images and whether or not the object is to be recognized according to the type of the still image. determined, a recognition processing parameter setting means for setting the parameters of the recognition process, and the recognition processing image recognition means to the type of the still image determined as parameter setting means is a target recognition processing for recognizing processing of the object Have,
The moving image data has an image quality index value for each still image.
The recognition processing parameter setting means provides an information processing apparatus characterized in that the still image whose image quality index value is equal to or higher than a threshold value is determined as a target of recognition processing .

It is possible to provide an information processing device capable of suppressing a decrease in moving image recognition accuracy.

It is an example of the figure explaining the schematic operation of an image processing system. It is a figure which shows the system configuration example of an image processing system. It is a figure which shows an example of the application which can be used by an image processing system. It is a figure which shows an example of cooperation of a plurality of Web services. This is an example of a schematic hardware configuration diagram of the recognition server. This is an example of a functional block diagram showing the functions of the recognition server in a block shape. This is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit determines a frame for performing recognition processing according to the magnitude of change. This is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit determines a frame and a template to perform recognition processing according to the magnitude of a change in an image. This is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit changes the target detection threshold value according to the image quality. This is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit instructs a specific frame to be recognized.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings with reference to examples.

FIG. 1 is an example of a diagram illustrating a schematic operation of the image processing system 100 of the present embodiment. The image pickup device 10 is capturing a moving image, and appropriately transmits the image data (moving image data) of the moving image to the recognition server 30. The recognition server 30 performs image recognition and provides the recognition result to various cloud services via the Web API.
(1) The image pickup apparatus 10 compresses the moving image at a high compression rate and transmits it to the recognition server 30. Communication costs can be reduced by increasing the compression rate.
(2) However, a high compression rate means that compression noise is likely to be included, and the image recognition accuracy may decrease. Here, the compressed moving image has three types of frames (I frame, B frame, P frame, etc.), but the magnitude of compression noise (good image quality) differs depending on the frame (the magnitude of compression noise is not necessarily a frame). Although it does not depend only on the type of frame, we will focus on the type of frame here for convenience of explanation).

Therefore, the recognition server 30 changes the parameters of the recognition process according to the type of the frame. Details will be described later, but for example, the template for target detection prepared for each frame is switched depending on the frame type, and the threshold value for target detection is changed. Recognition processing can be performed using parameters according to the image quality, etc., which differ depending on the type of frame, and highly accurate image recognition is possible even for moving images with a high compression rate.

In addition to changing the parameters, only a predetermined type of frame may be determined as the recognition target. For example, the recognition process is performed only on a specific frame (for example, I frame) having less compression noise. In this case as well, highly accurate image recognition is possible for moving images having a high compression rate.

<Terminology>
The type of still image in the claims refers to a still image included in the moving image data in order to reproduce the moving image data as a moving image. For example, there are I, P, and B frames as types of still images.

The image quality of a still image means that not only the degree of goodness of the still image but also the properties of the still image are different. For example, the difference in image quality can be caused by the difference in the information contained in the I, P, and B frames.

<System configuration example>
FIG. 2 is a diagram showing a system configuration example of the image processing system 100. The imaging device 10 and the recognition server 30 are connected via a network. The image pickup apparatus 10 may be called an IOT (Internet of Things) in the sense that it is a thing connected to the Internet. The image pickup apparatus 10 includes a photoelectric conversion element such as a CCD or CMOS, and converts the light incident through the lens into image data having luminance information. Specifically, it is called a digital still camera or a digital video camera, but it does not have to be called a camera as long as it has an imaging function. For example, some smartphones, tablet terminals, PCs (Personal Computers), game machines, and the like have a camera. Further, for example, a multifunction device, a projector, a video conference terminal, an electronic blackboard, a drone, or the like may be used. Further, the camera may be externally attached, and a device to which the camera is externally attached may operate as a camera.

Further, the image pickup device 10 may perform the image recognition process of the present embodiment, and in this case, the image pickup device 10 transmits the recognition result to the recognition server 30 or the Web service 301.

The recognition server 30 is an information processing device whose substance is a PC, a server device, or the like. The recognition server 30 is connected to various Web services 301, Web DB services 302, or an application 303 created by a user via a Web API. The recognition result is sent to these to realize an advanced application. In addition, WebAPI refers to an agreement (method of requesting processing, method of specifying data, etc.) when a device and a device perform processing via a network. Since the Web service 301 and the Web DB service 302 expose the Web API, the recognition server 30 and the Web service 301 or the Web DB service 302 can be linked with each other. Since the Web API of the application 303 created by the user is known to the user, the recognition server 30 can use it as well.

The application of the present embodiment means an application example in which useful information is obtained by using the recognition result of the recognition server 30 or equipment is controlled, or a meaning equivalent thereto. It does not necessarily refer to one piece of software.

According to the configuration of FIG. 2, the recognition server 30 recognizes the image captured by the imaging device 10 to detect a predetermined target, and at least the Web service 301, the Web DB service 302, or the application 303 created by the user. Information is transmitted to the user or the controlled object 304 via one. The application is what kind of information the user obtains or what kind of control target is controlled and how.

There is a Web service for linking the Web services 301 shown in FIG. 2, and the recognition server 30 and another Web service 301 can be linked by the user using such a linked Web service 301a. Can be done. IFTTT, zapier, myThings, etc. are examples of the linked Web service 301a. These linked Web services 301a serve as an interface for linking the recognition server 30 and another Web service 301.

The linked Web service 301a is a Web service that improves work efficiency by automating the actions of other Web services triggered by the work of an arbitrary Web service (or a terminal device such as a smartphone). By using the linked Web service 301a, the user can easily construct various applications by using the recognition result of the recognition server 30.

The Web DB service 302 is a Web service that mainly stores data. The Web DB service 302 also has a linked Web service 302a that serves as an interface. For example, a linked Web service 302a called fluentd is known. In the case of the application 303 created by the user, since the Web API is also created by the user, the linked Web services 301a and 302a are unnecessary, but the linked Web services 301a and 302a may be used. The application 303 created by the user may be any kind and may be various. This user includes both a company and an individual, and can arbitrarily create an application 303 specific to a certain company and an application 303 specific to an individual. In the case of a company, an application 303 that uses the recognition result for security, equipment control, FA (factory automation), etc. can be considered, and in the case of an individual, an application 303 that organizes image data of still images and moving images based on the recognition result, etc. Conceivable.

The recognition server 30 sends a request to send an image to the image pickup apparatus 10. The request is triggered when the user's operation of the image processing system 100, the Web service for recognition of the painter is directly or indirectly requested, or when a certain period of time set in the recognition server 30 has elapsed. Is. A request from the user, a request from the Web service 301, or a request triggered by the passage of a certain period of time is made by designating the imaging device 10. The image pickup device 10 is designated by the image pickup device ID. Alternatively, when the user requests, a user ID associated with the imaging device ID is specified. ID is an abbreviation for Identification and means an identifier or identification information. An ID refers to a name, a code, a character string, a numerical value, or a combination of one or more of these, which is used to uniquely distinguish a specific object from a plurality of objects.

Further, the request may include, in addition to the imaging device ID or the user ID, imaging conditions (imaging conditions suitable for recognition processing such as color and monochrome), imaging period of moving image data to be transmitted, and the like. Further, when the image pickup apparatus 10 performs the image recognition process, the information of the parameters of the recognition process of the present embodiment is included.

The image pickup apparatus 10 which has acquired the request newly acquires an image and transmits it to the recognition server 30. Alternatively, the latest image among the acquired (captured) images is always returned to the recognition server 30. At that time, the image is encrypted on the image pickup device 10 side before transmission so that eavesdropping by another person does not succeed in the communication path from the image pickup device 10 to the recognition server 30.

The recognition server 30 that has received the image decrypts the code and performs a predetermined or designated recognition process on the image. The recognition result obtained by the recognition process is transmitted to the Web service 301, the Web DB service 302, or the application 303 created by the user using the Web API. After that, information on these processing results is transmitted to the user or the control target 304. Although the image itself is not transmitted as described later, the image may be transmitted together with the processing result.

The recognition server 30 charges the usage fee of the image processing system 100 for each individual user, counts the number each time it responds to one request, and individually based on the count number. Determine the billing amount of the user.

The Web API format may be any format such as XML, JSON format or CSV format as long as the recognition result is stored. The Web service 301 that has acquired the recognition result or the application 303 created by the user transmits the recognition result to the user or the control target 304. As a result, the controlled object 304 determines, for example, that an intruder into the site has been detected, sounds an alarm, or sends an e-mail to the user. In addition, it is determined that a user (for example, a householder) has approached the front door of the home, and the lock mechanism of the home is driven so as to unlock the door. Alternatively, it is determined that a suspicious person has approached the front door of the house, and the door key is locked (including confirmation of the lock) and the security level is increased.

The image transmitted by the image pickup apparatus 10 may be an image as it is (bitmap or RAW data), or may be a compressed image such as a jpg image in consideration of communication cost. Alternatively, the feature amount based on the image such as SIFT, Surf, Haar-like feature, HOG feature, etc. may be used instead of the image itself. However, for example, in the case of human detection, there is no problem with a monochrome image, but color information is required for general object recognition, and an appropriate compression method or information reduction method differs depending on the type of recognition processing.

If the image pickup device 10 transmits an uncompressed (for example, bitmap format) or losslessly compressed (for example, tiff format) image, the information is delivered to the recognition server 30 without being lost, so that any recognition process can be supported. Is. However, since the amount of image data transmitted from the imaging device 10 to the recognition server 30 increases, the latency (delay) for the user or the control target 304 to acquire the recognition result after the request by the recognition server 30 increases. , The amount of data passing through the communication line increases, and the communication cost increases.

Therefore, depending on the type of recognition processing, the image pickup apparatus 10 uses, for example, a jpg image with lossy compression, or in the case of recognition processing in which fine resolution information is not required, the compression rate is increased or the image to be transmitted is transmitted. Reduce the resolution of. Further, in the case of recognition processing in which a monochrome image is sufficient, the image pickup device 10 is informed of what kind of moving image data is transmitted by the recognition server 30 according to the type of recognition process, such as the image pickup device 10 transmitting a monochrome image. It is desirable to instruct.

Further, when the recognition server 30 uses the HOG feature as the feature amount when performing human recognition, the image is converted into the HOG feature on the imaging device 10 side, and the HOG feature data is losslessly compressed and the recognition server is used. Send to 30. That is, if a feature amount suitable for the type of recognition processing is extracted, and only the feature amount is losslessly compressed and transmitted to the recognition server 30, the recognition processing can be performed well and the communication amount can be suppressed.

Further, in the case of a fixed point image pickup device fixed at a place where the image pickup device 10 is located, the background of the image captured by the image pickup device 10 is almost unchanged. Therefore, the image pickup device 10 separates the background and other parts, and the image pickup device 10 is separated. By setting the image or feature amount transmitted from the side to the recognition server 30 to a portion other than the background, the amount of communication data can be further reduced.

When a continuous frame (a plurality of still images or a short moving image) is required, such as when the recognition process performed by the recognition server 30 detects a movement of waving a person, the image pickup device 10 side has a fixed amount. The frame is imaged, and the still image group (or moving image) is transmitted to the recognition server 30. Even in that case, it is desirable to perform appropriate video compression processing before transmission.

When the recognition server 30 generates a request to the imaging device 10 at regular intervals, the user can set the imaging conditions of the imaging device 10 with a simple Web API command. For example, it is possible to set which image pickup device 10, how often, and how to take images. Further, when the image pickup apparatus 10 performs the recognition process, it is possible to set what kind of recognition process is executed and what kind of feature amount is to be extracted. By doing so, the image processing system 100 can easily make an application that monitors whether or not there is a person every second, and if there is a person, sends an alert to a set smartphone or mobile phone. Can be realized.

Further, the image pickup apparatus 10 may acquire a color image or a monochrome image. Further, depending on the application, the imaging device 10 that can acquire a spectroscopic image or a polarized image may be used. Further, it is preferable to use a stereo imaging device capable of acquiring distance information or a monocular imaging device capable of acquiring distance information by TOF (Time Of Flight) or the like. Further, an imaging device capable of acquiring a spherical image may be used.

In the recognition server 30, the type of the image pickup device 10 is registered for each image pickup device 10, and the optimum recognition process is performed according to the type of the image pickup device 10. Further, lighting may be installed around the imaging device 10 or the imaging device 10 so as to be able to image a dark place. In this case, turning on the illumination only when the imaging device 10 receives a request for imaging consumes power. Can be reduced.

For example, in an application for detecting a person, when the recognition server 30 periodically performs a process of detecting a person, when the recognition server 30 detects a person, a predetermined command is sent to the corresponding image pickup device 10 to perform imaging. The device 10 records a moving image for a certain period of time thereafter at a high frame rate. For example, since the recognition server 30 can record images after the appearance of a suspicious person in a fine time series, the recognition server 30 and related persons can analyze the images later over time.

Further, the information transmitted by the recognition server 30 to the user is limited to the recognition result information, and the information that allows the user to restore the original image before the recognition process is removed. That is, the image is not returned, but only the text information of the recognition result such as how many people are shown is sent. This avoids invasion of privacy.

<Application>
FIG. 3 is used to supplement the applications available by the image processing system 100. FIG. 3 shows an example of an application that can be used by the image processing system 100. FIG. 3 shows an example of an application for each category. Categories include, but are not limited to, for example, transportation, security, offices, long-term care, construction, stores, health, factories, agriculture, homes, fisheries, livestock, surveillance, and logistics.

For example, there is an application called traffic survey in the traffic category. It is necessary to detect each passing vehicle in order to investigate the traffic volume. In the past, it was necessary to configure the system using a dedicated and expensive imaging device 10 capable of detecting a vehicle, and it was necessary to move and install the system when the location of the traffic volume survey changed. In addition, since it is movable, there is a risk of theft, and it has not been put into practical use in general.

However, according to the image processing system 100 of the present embodiment, it is sufficient that the image pickup device 10 which is inexpensive and has a communication function is fixed at the measurement point. The user accesses the recognition server 30 with a PC or the like, and specifies the image pickup device ID and the vehicle detection process of the image pickup device 10 at the location where the traffic volume survey is desired. This makes it possible to measure the number of vehicles currently captured by the imaging device 10 from a remote location. If the user sets the image to be imaged every 5 seconds by specifying the time and the DB service 302 of the Web service is specified as the output destination of the recognition result, an application that automatically detects the number of vehicles can be easily realized. It is possible.

<About WebAPI>
Some examples of Web API when the recognition server 30 is provided as a Web server will be described.
1. 1. The number of faces in the image pickup device 10 is returned.
http://rirs.xxxxxxx.com/mysite/cgi-bin/face.cgi
2. 2. The number of people in the image pickup device 10 is returned.
http://rirs.xxxxxxx.com/mysite/cgi-bin/human.cgi
3. 3. The name of what is reflected in the image pickup apparatus 10 is returned.
http://rirs.xxxxxxx.com/mysite/cgi-bin/god.cgi
4.1 The number of faces in the image pickup device 10 is returned every 1 sec.
http://rirs.xxxxxxx.com/mysite/cgi-bin/start_face.cgi
Stop returning the number of faces in the image pickup device 10 every 5.1 seconds.
http://rirs.xxxxxxx.com/mysite/cgi-bin/stop_face.cgi
The user accesses the linked Web service 301a with a smartphone or a PC, and links the URLs 1 to 5 with an arbitrary Web service 301b. This makes it possible to automate the following Web services (construction of applications).

FIG. 4 is a diagram showing an example of cooperation between a plurality of Web services. The first Web service in FIG. 4A is a Web service provided by the recognition server 30 of the present embodiment, and is, for example, "1. Returns the number of faces captured in the image pickup device 10." The Web service 301b is "automatically turn on the electric fan or the like when there is a person in the seat". The user or the control target 304 is an electric fan. The linked Web service 301a links the first and second Web services. As a result, an application for automatically turning on the electric fan when a face is reflected in the image pickup device 10 is realized.

The first Web service in FIG. 4B is a Web service provided by the recognition server 30 of the present embodiment, and is, for example, "2. Returns the number of people in the image pickup device 10." The Web service 301c is "automatically make a call to a designated telephone number". The user or the control target 304 corresponds to a device that makes an incoming call or a telephone call to the user. The linked Web service 301a links the first and second Web services. As a result, when a person intrudes at night, an application that automatically calls a specified telephone number is realized.

<Hardware configuration example of recognition server 30>
FIG. 5 is an example of a schematic hardware configuration diagram of the recognition server 30. The recognition server 30 includes a CPU 201 and a memory 202 that enables high-speed access to data used by the CPU 201. The CPU 201 and the memory 202 are connected to other devices or drivers of the recognition server 30, such as the graphics driver 204 and the network driver (NIC) 205, via the system bus 203.
The graphics driver 204 is connected to the LCD (display device) 206 via a bus to monitor the processing result by the CPU 201. Further, the network driver 205 connects the recognition server 30 to the network N at the transport layer level and the physical layer level to establish a session with the image pickup apparatus 10 and the Web service 301.
An I / O bus bridge 207 is further connected to the system bus 203. On the downstream side of the I / O bus bridge 207, a storage device such as an HDD (hard disk drive) 209 is used by IDE, ATA, ATAPI, serial ATA, SCSI, USB, etc. via an I / O bus 208 such as PCI. Is connected. The HDD 209 stores a program 209p that controls the entire recognition server 30. HDD 209 may be SSD (Solid State Drive).

Further, an input device 210 such as a keyboard and a mouse (called a pointing device) is connected to the I / O bus 208 via a bus such as USB, and receives inputs and commands from an operator such as a system administrator. There is.

The hardware configuration of the recognition server 30 shown in the figure does not have to be housed in one housing or provided as a group of devices, and it is preferable that the recognition server 30 has a hardware element. Is shown. Further, in order to support cloud computing, the physical configuration of the recognition server 30 of this embodiment does not have to be fixed, and hardware resources are dynamically connected / disconnected according to the load. It may be configured.

<About functions>
FIG. 6 is an example of a functional block diagram showing the functions of the recognition server 30 in a block shape. The recognition server 30 has a moving image development unit 31, a recognition processing unit 33, and a recognition processing parameter setting unit 32. These are functions or means realized by an instruction or the like output by the CPU 201 shown in FIG. 5 according to the program 209p.

The moving image development unit 31 acquires a moving image from an IOT such as an imaging device 10, reads a moving image from a storage medium such as an SD card, and develops the moving image. For example, Mpeg1, Mpeg2, Mpeg4, or H.264 decoder. The moving image development unit 31 extracts still images (I, P, B frames described later) included in the moving image.

The recognition processing parameter setting unit 32 sets the parameters when the recognition processing unit 33 performs the image recognition processing in the recognition processing unit 33. Although the details will be described later, parameters for recognition processing are set in the recognition processing unit 33 according to which of the I, P, and B frames. In addition, the recognition processing unit 33 is notified whether or not the recognition processing is performed according to which of the I, P, and B frames.

The recognition processing unit 33 performs image recognition processing according to the parameters set by the recognition processing parameter setting unit 32. For example, the recognition target is detected from the image, or the movement of the target is detected.

<Recognition method for video>
Hereinafter, how the recognition server 30 recognizes the moving image will be described. Generally, when the moving image captured by the imaging device 10 is transmitted to the recognition server 30 via the network or stored in a storage medium such as an SD card, the moving image is compressed. As a video compression method, standardized compression methods such as Mpeg1, Mpeg2, Mpeg4, and H.264 are known, but in each case, the compressed video has the following three frames with different image quality. It is configured. It should be noted that this image quality not only refers to the degree of goodness of the image, but also includes that the properties of the image are different.
I-frame: A frame that holds all the information in that frame (called a key frame)
P frame: A frame that retains the difference from the previous "I frame" in time B frame: A frame that retains the difference between the "I frame" and "P frame" before and after in time (the difference from the B frame is retained) In some cases)
Further, a technique for performing lossy compression by dividing into these I, P, and B frames is called a GOP (Group of Picture). Here, when the compression rate is increased, the image quality deteriorates as the compression rate increases, and the accuracy of image recognition also decreases. However, the compressed image has the above frame structure, and when compressing, for example, the I frame is input as a frame with a large change from the previous frame, such as when the scene changes, or is constant. It may be entered at intervals. For this reason, in a scene with a lot of movement, the image quality of the I frame is less deteriorated than that of the P frame and the B frame which are the difference data. On the contrary, in the case of a scene in which the background is fixed, the image quality of the P frame and the B frame, which can store the difference information, is less deteriorated than that of the I frame.

<< Selection of frame based on image change >>
Since the image quality of the I frame is often the frame in which the scene changes significantly, for example, when the recognition server 30 recognizes the moving image captured by the in-vehicle image pickup device 10 whose background is constantly changing, the P frame having a low image quality or the B The recognition accuracy can be improved by selecting and using only the I frame rather than the frame and performing image recognition processing. That is, in the case of a frame having a large change, the recognition server 30 performs image recognition processing using an I frame considered to have high image quality.

On the contrary, since the image quality of the P frame and the B frame is higher than that of the I frame in the frame with less movement, the recognition server 30 performs the image recognition process in the P and B frames.

Therefore, it is preferable to determine the degree of change and switch the frame for performing the recognition process according to the magnitude of the change. The magnitude of the change can be determined, for example, by taking the difference in pixel values between the frame of interest and the frame immediately before in time, and the sum of the absolute values.

FIG. 7 is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 determines a frame for performing recognition processing according to the magnitude of the change.

First, the moving image development unit 31 develops the moving image into I, P, and B frames (S10).

The recognition processing parameter setting unit 32 calculates the difference between the two frames in the time series (S20). In this case, the two frames are at least one or more of the I frames closest in time, the P frames closest in time, or the B frames closest in time.

Next, the recognition processing parameter setting unit 32 determines whether or not the difference is equal to or greater than the threshold value (S30). If the determination in step S30 is Yes, the change (movement) is large, so the recognition processing parameter setting unit 32 notifies the recognition processing unit 33 to perform the image recognition processing on the I frame considered to have good image quality (S40).

If the determination in step S20 is No, there is little change (movement), so the recognition processing parameter setting unit 32 notifies the recognition processing unit 33 to perform image recognition processing on the P or B frame considered to have good image quality (S50). ).

By doing so, it is possible to select a frame with high image quality and perform image recognition processing, so that the total recognition accuracy can be improved.

<< Change template >>
When an I frame or a P or B frame is selected as in the process of FIG. 7, it is effective to further change the template. The template is data that serves as a reference for comparison (data related to an image in the case of image recognition) that reflects the characteristics of the object recognized from the image. For example, consider the case of detecting a signboard or the like in an image. In the case of a moving image of the in-vehicle imaging device 10 whose background changes, the signboard is relatively clearly shown in the I frame, whereas it is often blurred in the B or P frame. Therefore, the template that easily matches the signboard is also different between the I frame and the B and P frames.

Therefore, the recognition accuracy can be improved by changing the template in the I frame and the template in the B and P frames and performing template matching by the recognition processing unit 33.

FIG. 8 is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 determines a frame and a template for performing recognition processing according to the magnitude of change in the image. Note that FIG. 8 mainly describes the difference from FIG. 7.

Steps S10 to S50 are the same as in FIG. In step S60, the recognition processing parameter setting unit 32 instructs the recognition processing unit 33 of the template for the I frame, and in step S70, the recognition processing parameter setting unit 32 instructs the recognition processing unit 33 of the template for the P and B frames. To do.

As a result, the recognition processing unit 33 can perform recognition processing on the image with a template suitable for the type of frame.

<< Change the threshold value to detect the target according to the image quality or frame type >>
When the image recognition process is executed for a frame whose image quality has deteriorated, there is a disadvantage that the detection rate of the target object is lowered and many false detections occur. It is better that both the decrease in the detection rate and the false detection are small. For example, when the detection rate is lowered, if it is a moving image, it may be detected in another frame, but there may be a situation where a predetermined control is delayed or cannot be controlled. In addition, when the number of false positives increases, unnecessary alarms increase. For example, when the recognition server 30 (or the in-vehicle imaging device 10 may be used) recognizes surrounding obstacles for automatic braking of the vehicle, the vehicle erroneously detects that there are obstacles even though there are no obstacles in the surroundings. May brake. If it is a video, unnecessary alarms will increase further.

In order to suppress such inconvenience, it is effective to change the threshold value of whether or not it is determined that the object has been detected according to the image quality. That is, when the image quality is low, the threshold value is set high so that the recognition server 30 does not detect the object unless the object to be detected becomes high. Therefore, erroneous detection can be reduced. On the contrary, when the image quality is high, by setting the threshold value low, the object is detected even if the object-likeness is not high, so that the recognition rate is improved. Therefore, the detection rate can be improved, erroneous detection can be suppressed, and the total recognition accuracy can be improved.

Information on image quality can be determined based on image quality index values such as PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). The smaller the value of PNSR, the greater the deterioration, and the higher the value, the less the deterioration. SSIM is an image quality index value that more accurately indexes the differences that humans perceive.

The image quality index value can be calculated by the recognition server 30. However, when the image pickup apparatus 10 compresses, it is preferable to compare the frames before and after the compression and store the image quality index values such as PSNR and SSIM for each frame. The recognition processing parameter setting unit 32 can change the threshold value for determining whether or not an object has been detected based on PSNR or SSIM.

FIG. 9A is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 changes the target detection threshold value according to the image quality.

First, the moving image development unit 31 develops the moving image into I, P, and B frames (S10).

The recognition processing parameter setting unit 32 acquires the image quality index value of each frame (S20). Since the image quality can be determined from one frame, the image quality index value is acquired from each of the I, P, and B frames. Alternatively, the image quality index value may be acquired every few frames.

Next, the recognition processing parameter setting unit 32 determines whether or not the image quality index value is equal to or greater than the threshold value (S30). If the determination in step S30 is Yes, the threshold value for determining whether or not an object has been detected is set to the threshold value A because the image quality is high (S40). The threshold value A is a relatively low value that can detect the object to be recognized with a certain probability.

If the determination in step S30 is No, the image quality is low, so a threshold value for determining whether or not an object has been detected is set in the threshold value B (S50). The threshold value B is a value at which the certainty of the object to be recognized is considered to be sufficiently high. It should be noted that the threshold value A <threshold value B.

By doing so, it is possible to change the threshold value for determining whether or not the object is detected according to the image quality, so that it is possible to suppress the decrease in the detection rate and reduce the false detection.

In addition, the image quality can be estimated to some extent depending on the type of frame. For example, in the case of a moving image of an in-vehicle imaging device 10 whose background is constantly changing as described above, since the image quality of the I frame (key frame) is high, it is determined that the object is detected depending on whether it is the I frame or not. You can change the threshold for whether or not.

FIG. 9B is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 changes the target detection threshold value according to the type of frame.

First, the moving image development unit 31 develops the moving image into I, P, and B frames (S10).

Next, the recognition processing parameter setting unit 32 determines whether or not the frame of the developed moving image is an I frame (S20).

In the case of the I frame, since it is estimated that the image quality is high, the threshold value for determining whether or not the object is detected is set to the threshold value A (S30).

If it is not an I frame, it is estimated that the image quality is low, so a threshold value for determining whether or not an object has been detected is set in the threshold value B (S40). It should be noted that the threshold value A <threshold value B.

In FIG. 9B, the threshold value for determining whether or not an object was detected for the I frame was reduced, but it is estimated that the image quality is high in the case of a scene in which the background is fixed. The threshold value for determining whether or not an object has been detected may be reduced with respect to the P frame or the B frame.

Not limited to the threshold value for determining whether or not an object has been detected, the following parameters may be set according to the type of frame. For example, in detecting the color information of a traffic light, the traffic light is in color, but the accuracy of the color information differs depending on each frame type. Therefore, the recognition accuracy can be improved by changing, for example, the threshold value for detecting a red signal or the color range depending on the type of frame. When the red, blue, and yellow color information of a traffic light is represented by Y, U, and V, the range in which the color information can be obtained is set by Y_min, Y_max, U_min, U_max, V_min, and V_max depending on the frame type. For example, Y_min, Y_max, U_min, U_max, V_min, V_max, which are judged to be red lights in the I frame, and Y_min, Y_max, U_min, U_max, V_min, V_max, which are judged to be red lights in the P and B frames, are changed. The same applies to green and yellow traffic lights.

FIG. 9C is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 changes the threshold value of the color information according to the type of the frame. Steps S10 and S20 are the same as in FIG. 9B.

In step S30, the recognition processing parameter setting unit 32 sets a threshold value or a color range for detecting the color information for the I frame, and in step S40, sets the threshold value or the color range for detecting the color information for the P and B frames. Set a threshold or color range.

In this way, by setting the threshold value or the color range for detecting the color information according to the type of the frame, the color detection accuracy is improved.

<< Recognition processing using only I frame or P, B frame >>
The recognition process may be performed using only the I frame or the P and B frames. For example, in image processing for recognizing a person from image data captured by a monitoring image pickup device 10 installed in a place where the background changes one after another, detection is performed by recognizing only a high-quality I-frame image. The rate can be increased and false positives can be reduced. Further, in the imaging device 10 for monitoring installed in a place where the background hardly changes, the image quality may be better in the B frame and the P frame having a large amount of integrated information because there is no motion information. For example, the recognition process is performed using only the P frame.

FIG. 10 is an example of a flowchart showing a procedure in which the recognition processing parameter setting unit 32 instructs a recognition target of a specific frame. In FIG. 10, it is assumed that the recognition server 30 performs image processing for recognizing a person in the image data captured by the monitoring image pickup device 10 installed in a place where the background changes one after another.

First, the moving image development unit 31 develops the moving image into I, P, and B frames (S10).

Next, the recognition processing parameter setting unit 32 determines whether or not it is an I frame (S20). In the case of an I-frame, the recognition processing parameter setting unit 32 allows the recognition processing unit 33 to perform recognition processing for the I-frame (S30).

If it is not an I frame, the recognition processing parameter setting unit 32 does not allow the recognition processing unit 33 to perform recognition processing for the I frame (S40).

By doing so, the recognition process can be performed using only the high-quality I-frame image.

In FIG. 10, an image processing for recognizing a person in image data captured by a monitoring image pickup device 10 installed in a place where the background changes one after another is taken as an example, but an image pickup device installed in a place where the background hardly changes. In the image processing for the image data of 10, the recognition process is permitted for the P or B frame, and the recognition process is not permitted for the I frame.

Further, in FIG. 10, the presence / absence of recognition processing is controlled by the type of frame, but as shown in FIGS. 7 and 8, it is determined whether the frame is an I frame or a P or B frame, and the presence / absence of recognition processing is controlled. You may. In this case, the parameters need only be set for the frame to be recognized. Further, as shown in FIG. 9, the quality of the image quality or the type of the frame may be determined from the image quality index value to control the presence or absence of the recognition process. Frames with good image quality are subject to recognition processing. In this case, the parameters need only be set for the frame to be recognized.

<Summary>
As described above, by changing the recognition processing parameters according to the type of frame, highly accurate image recognition is possible even for a moving image having a high compression rate. Further, by performing the recognition process only on a predetermined type of frame, it is possible to recognize an image with high accuracy for a moving image having a high compression rate. Since the compression rate is high, the communication cost or the cost of the storage device can be reduced, and even if the image data has a high compression rate, a decrease in recognition accuracy can be suppressed.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to Examples, but the present invention is not limited to these Examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, in the present embodiment, although moving images have been described as an example, it can be applied to continuous still images and image data having similar frames.

In addition, Mpeg1, Mpeg2, or H.264 was mentioned as the type of video, but AVI (Audio Video Interleave), MOV, ASF (Advanced Systems Format), Ogg (Ogg), Matroska (Matroska), DivX (Davix) ) Etc. can also be applied to videos. Further, any moving image having I, P, and B frames can be applied to moving images other than these standards.

The moving image developing unit 31 is an example of the moving image developing means, the recognition processing parameter setting unit 32 is an example of the recognition processing parameter setting means, and the recognition processing unit 33 is an example of the image recognition means. The I frame is an example of the first frame, the P frame is an example of the second frame, and the B frame is an example of the third frame.

10 Image pickup device 30 Recognition server 31 Video development unit 32 Recognition processing parameter setting unit 33 Recognition processing unit 100 Image processing system

Japanese Unexamined Patent Publication No. 2009-88944

Claims (6)

An information processing device that recognizes an object from an image
A video expansion method that expands video data into multiple still images,
A recognition processing parameter setting means that determines whether or not to be the target of the recognition processing according to the type of the still image and sets the parameters of the recognition processing.
Have a, an image recognition means for performing recognition processing of the object with respect to the type of the still image determined to be a target for the recognition parameter setting means recognition process,
The moving image data has an image quality index value for each still image.
The information processing device is characterized in that the recognition processing parameter setting means determines the still image whose image quality index value is equal to or higher than a threshold value as a target of recognition processing . The type of still image is the first frame that holds all the information of the frame.
A second frame that retains the difference from the previous first frame in time, or
The information processing apparatus according to claim 1 , which is a third frame that retains the difference between the first frame and the second frame that are back and forth in time. The information processing apparatus according to claim 2 , wherein the recognition processing parameter setting means determines the first frame as a target of recognition processing. The recognition processing parameter setting means calculates the magnitude of the change in the still image in time series, and obtains the magnitude of the change.
When the size is equal to or larger than the threshold value, the first frame is determined as the target of recognition processing.
The information processing apparatus according to claim 2 , wherein when the size is not equal to or larger than a threshold value, the second frame or the third frame is determined as the target of recognition processing. An image processing system having an information processing device that detects an object from an image and a Web service that communicates via a network.
A video expansion method that expands video data into multiple still images,
The moving image data has an image quality index value for each still image, and the still image whose image quality index value is equal to or higher than the threshold value is determined as a target of recognition processing, and depends on the type of the determined still image. The recognition processing parameter setting means for setting the recognition processing parameters,
It has an image recognition means that performs the recognition processing of the target on the still image based on the parameters set by the recognition processing parameter setting means.
An image processing system in which the information processing device transmits the recognition result to a Web service via a Web API to which the recognition result is transmitted to provide information created by the Web service, or the Web service controls a device. An information processing device that recognizes an object from an image, a video expansion means that expands video data into multiple still images, and
The moving image data has an image quality index value for each still image, and the still image whose image quality index value is equal to or higher than the threshold value is determined as a target of recognition processing, and depends on the type of the determined still image. The recognition processing parameter setting means for setting the recognition processing parameters,
An image recognition means that performs the target recognition processing on the still image based on the parameters set by the recognition processing parameter setting means.
A program characterized by functioning as

Images