JP2017045423A - Image processing apparatus, image processing method, image processing system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, method, system and program that can more accurately identify objects in which customers are interested.SOLUTION: An image processing apparatus comprises a person's position detector 203 that detects the position of a person in an image picked up with an image pickup device, a head position detector 206 that detects the height of the head of the person, and an information generator 207 that generates information regarding an object attentively watched by the person on the basis of the position of the person and any change in the height of the person's head. The apparatus further has a personal attributes estimator 205 that estimates attributes of the person on the basis of the image, and the information generating means 207 adds information on attributes of the person to the information regarding the object attentively watched by the person.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a video processing apparatus, a video processing method, a video processing system, and a program for analyzing videos taken by a plurality of cameras.

  In recent years, with the improvement of video analysis technology, a system has been devised that analyzes the purchasing behavior of customers who visit a store based on video taken by a camera placed in a store such as a supermarket or a convenience store. For example, Patent Document 1 proposes a system that specifies a product that a customer is interested in, and outputs information related to the product and information about the customer in association with each other. Further, Patent Document 2 proposes an analysis device that detects an action of a person picking up an article even in a situation where the movement of the person's hand or arm cannot be detected.

JP 2009-3701 JP-A-2015-11649

  In an actual store, merchandise is placed on many display shelves, and customers can look in different directions to see the merchandise while standing in the same position. Further, when the position of a desired product is far from the customer's line of sight, the customer may squat down or step on a stepladder and view the product. However, in the above-described conventional technology, in order to grasp information on products that the customer is interested in, changes in the direction of the customer's face and the vertical movement from the customer's ground are not taken into consideration.

  This invention is made | formed in view of the said subject, and aims at capturing the target which a customer is interested more correctly.

  As a means for achieving the above object, the video processing apparatus of the present invention comprises the following arrangement. That is, person position detecting means for detecting the position of the person in the video imaged by the imaging device, head position detecting means for detecting the height of the head of the person, the position of the person, and the head of the person Generating means for generating information relating to an object to be watched by the person based on a change in the height of the section.

  According to the present invention, it is possible to more accurately capture an object that a customer is interested in.

The figure which shows an example of the functional block structure of the video processing system in 1st embodiment. The figure which shows the operating environment of the video processing system in 1st embodiment. The figure which shows an example of the hardware constitutions of the server apparatus in 1st embodiment. The flowchart which shows operation | movement of the video processing system in 1st embodiment. The flowchart which shows the operation | movement of a person position detection in 1st embodiment. The figure explaining the process of the calibration in 1st embodiment. The figure explaining the process of the matching search of a person and the detection of the three-dimensional position coordinate of a person in 1st embodiment. The figure explaining the process of a complement of the detection omission in 1st embodiment. The figure which shows the example of the screen displayed on the display apparatus in 1st embodiment. The figure which shows the example of the screen displayed on the display apparatus in 1st embodiment. The figure which shows the example of the screen displayed on the display apparatus in 1st embodiment.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

<First Embodiment 1>
FIG. 2 is a diagram showing an operating environment of the video processing system in the first embodiment. The camera 100, the server device 200, the storage device 300, and the display device 400 are connected by a LAN (Local Area Network) 500 that is a network line. Being a LAN network is just an example, and other networks may be used.

  The camera 100 is an imaging device that can be connected to a network. The server device 200 collects (for example, receives) video data captured by a plurality of cameras 100 connected via the LAN 500, and performs video analysis processing. That is, the server device 200 functions as a video processing device. The video analysis process includes, for example, processes such as moving object detection, moving object tracking, human body detection, face recognition, and object detection. The server device 200 collects past video data and analysis data recorded in the storage device 300 in addition to the analysis data subjected to the video analysis processing. Then, the server device 200 manages video information throughout the store by using the collected data. Video data captured by the camera 100 and analysis data subjected to video analysis processing by the server device 200 are recorded in the storage device 300 via the LAN 500.

  The display device 400 displays an image that combines the video data recorded in the storage device 300 and the analysis data. The display device 400 can also display information such as video information and time managed by the server device 200. The display device 400 can accept an operation for video search via a user I / F (interface) (not shown). In this case, the server device 200 receives information related to the search target from the display device, searches for a specific event scene to be searched from the data recorded in the storage device 300, and provides the search result to the display device 400. . The display device 400 displays the provided information. Note that the display device 400 may be incorporated in a part of the server device 200. Further, the server device 200 may have a display control function for displaying information on the display device 400.

  As shown in FIG. 2, the display device 400 is assumed to be a combination of a PC and a monitor, for example. However, since the physical connection form of the display device 400 to the LAN 500 is not only wired but may be wireless, the display device 400 may be a wireless terminal such as a tablet terminal. When the server device 200 has a display control function for displaying information on the display device 400, the display device 400 may be a simple monitor. Thus, the form of the display device 400 is not limited. Further, the number of cameras 100 constituting the video processing system is four in FIG. 2, but any number is possible. Further, the number of server devices 200, storage devices 300, and display devices 400 connected to the LAN 500 is not limited to one as shown in FIG. 1, and may be plural as long as they can be identified by an address or the like. It is assumed that the position where each camera 100 is installed is known.

FIG. 1 is a diagram illustrating an example of a functional block configuration of a video processing system according to the present embodiment. The camera 100 includes an imaging sensor unit 101, a video processing unit 102, a video encoding unit 103, and a communication control unit 104.
The imaging sensor unit 101 is an imaging element such as a CMOS, and converts a light image formed on the imaging surface into a digital electric signal by photoelectric conversion.
The video processing unit 102 performs predetermined pixel interpolation and color conversion processing on the digital electric signal obtained by photoelectric conversion from the image sensor unit 101. By performing the above processing, the video processing unit 102 generates a digital video such as RGB or YUV. Further, the video processing unit 102 performs predetermined arithmetic processing on the generated digital video, and performs video processing such as white balance adjustment, sharpness adjustment, contrast adjustment, and color conversion based on the obtained calculation result. It can also be done.

The video encoding unit 103 encodes the digital video signal input from the video processing unit 102. For example, the video encoding unit 103 performs compression processing on the input digital video signal in order to distribute the video. The compression processing method is based on standards such as MPEG4, H.264, MJPEG, or JPEG. Further, the video encoding unit 103 converts the video data into a file according to the mp4 or mov format. Note that the compression process is not necessarily performed.
The communication control unit 104 performs communication control for communicating with the server device 200. For example, the communication control unit 104 performs communication control for communication with the server device 200 in accordance with the 802.11 series. Further, the communication control unit 104 constructs a network file system such as NFS (Network File System) or CIFS (Comon Internet File System) by cooperating with a communication control unit (not shown) of the storage apparatus 300. It is also possible to record video data.

The server device 200 includes a communication control unit 201, a video decoding unit 202, a person position detection unit 203, a face detection unit 204, a person attribute estimation unit 205, a head position detection unit 206, and an information generation unit 207. The communication control unit 201 has a function equivalent to that of the communication control unit 104 of the camera 100 described above.
The video decoding unit 202 decompresses and decodes video data distributed from the camera 100. The video decoding unit 202 can also decompress and decode the video data acquired from the storage device 300.
The person position detection unit 203 performs position detection processing for detecting the position of the person using the triangulation method while performing tracking processing of the person being photographed using the video data from the plurality of cameras 100.
The face detection unit 204 detects the face of the person being photographed from the decoded video data. The face detection unit 204 detects the orientation of the face by recognizing a part of the detected person's face such as eyes, nose, or mouth.
The person attribute estimation unit 205 estimates the age and sex of the person based on the face of the person detected by the face detection unit 204. The head position detection unit 206 detects the position of the head of the person being photographed from the decoded video data, and estimates the height of the person based on the detected position of the head. The height may be estimated by using the distance from the lower end of the frame when passing through the entrance door frame to the position of the head, or by detecting a person's foot by pattern matching. The distance from the head to the head may be used for estimation. Alternatively, the floor surface may be recognized and estimated from the distance from the floor to the head. The method of detecting a person's foot can estimate the height of the person at an arbitrary point.
The information generation unit 207 generates information related to an object that the person gazes at. It is assumed that the information generation unit 207 has information about the position of the product in the store and the position of the product shelf in advance.

  FIG. 3 is a diagram illustrating an example of a hardware configuration of the server device 200 and the display device 400 in the present embodiment. As an example, an example in which the display device 400 is incorporated in the server device 200 is shown. The control unit 31 is a CPU (Central Processing Unit), for example, and controls the operation of each component. A ROM (Read Only Memory) 32 stores control commands, that is, programs. A RAM (Random Access Memory) 33 is used for temporary storage of work memory and data when executing a program. The communication unit 34 performs control for physically communicating with an external device. The display unit 35 performs various displays. The user I / F 36 receives a user operation.

  Next, the operation of the person position detection unit 203 will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the person position detection unit 203. The person position detection unit 203 preliminarily determines the coordinates on the image and the actual coordinates (three-dimensional) on an object (for example, a person) included in the image based on the image data generated by the plurality of cameras 100. Calibration for associating with (position coordinates) is performed (S11).

The process of S11 will be described with reference to FIG. FIG. 6 is a diagram for explaining the calibration process. In FIG. 6, cameras 1 to 3 correspond to the camera 100 in FIGS. 1 and 2. In this process, as shown in FIG. 6, the person position detection unit 203, for example, the coordinates (X ₁ , Y ₁ , Z ₁ ) on the video of the person 602 included in the video data generated by the camera 1, The fact that the three-dimensional actual coordinates (X _a , Y _a , Z _a ) of the person 601 are the same is related. The person position detection unit 203 performs similar processing on the video data captured by the camera 2 and the camera 3. As a result, the coordinates (X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ), (X ₃ , Y ₃ ) of the persons 602 to 604 included in the images generated by the cameras 1 to 3 respectively. , Z ₃ ) and the three-dimensional actual coordinates (X _a , Y _a , Z _a ) of the person 601 are the same, and the following processing is started. Although details will be described later, the three-dimensional actual coordinates (X _a , Y _a , Z _a ) of the person 601 are calculated, for example, every predetermined time. In the calibration stage, the actual coordinates (X _a , Y _a , Z _a ) in three dimensions may be arbitrary initial values.

After the calibration in S11 is completed, the person position detection unit 203 performs a tracking process for tracking a person included in the video generated by each camera 100 (S12). For example, a person can be detected using pattern matching, the image feature amount of the detected person can be recorded, and a tracking process can be performed by detecting a region most similar to the image feature amount in the next frame. . Various other methods can be used as a method for tracking a person.
Next, the person position detecting unit 203 searches the person who is performing the tracking process by searching for the correspondence between the persons using the multi-viewpoint geometry analysis (S13). Thereafter, the person position detection unit 203 calculates the three-dimensional position coordinates of the person who is performing the tracking process (S14).

The processes of S13 and S14 will be described with reference to FIG. FIG. 7 is a diagram for explaining the process of searching for the correspondence between persons and calculating the three-dimensional position coordinates of the person. Cameras 1 and 2 in FIG. 7 correspond to the camera 100 in FIGS. 1 and 2. As shown in FIG. 7, a straight line 702 connecting the person 701 (the head in FIG. 7) and the camera 1 included in the video imaged by the camera 1 is like a straight line 703 on the video imaged by the camera 2. appear. The person position detection unit 203 calculates the position of the straight line 703 on the video captured by the camera 2 using information indicating the position of each camera. The information indicating the position of each camera may be information indicating at least the relative position of each camera, but may be information expressing the space where the camera is installed in three-dimensional coordinates.
The person position detecting unit 203 detects a person 704 that intersects with the straight line 703 on the image captured by the camera 2 (a person whose head and the straight line 703 intersect in FIG. 7) and a person 701 included in the image captured by the camera 1. Identify. That is, the person 701 included in the video generated by the camera 1 and the person 704 included in the video generated by the camera 2 are determined to be the same person 705 and are associated with each other. Next, the person position detection unit 203 detects the three-dimensional position coordinates of the person 705 by triangulation using the known position information of the cameras 1 and 2 (S14). Specifically, the person position detection unit 203 is a straight line connecting the person 701 and the camera 1 included in the video captured by the camera 1 from information indicating the position of each camera and information indicating the shooting direction of the camera. 702 is calculated. Then, a straight line connecting the person 701 and the camera 2 included in the video photographed by the camera 2 is calculated. Then, the intersection of the two straight lines is calculated as the position of the person 701 (the head in FIG. 7). Thereby, the actual position of the person 701 with respect to the installation position of the camera can be calculated.

  Furthermore, the person position detection unit 203 supplements the detection omission with respect to the camera that could not associate the person in S13 (S15). The process of S15 will be described with reference to FIG. FIG. 8 is a schematic diagram for explaining detection omission complementing processing. Cameras 1 to 3 in FIG. 8 correspond to the camera 100 in FIGS. 1 and 2. In FIG. 8, as described with reference to FIG. 7, the camera 1 and the camera 2 are associated with the person 705, but the camera 3 is not associated with the person 705. In this case, the person position detection unit 203 detects the position coordinates on the video when the three-dimensional position coordinates of the person 705 detected in S14 are projected on the video based on the video data generated by the camera 3. . Then, an image capable of recognizing the position of the person 705 calculated based on the detection result of another camera is displayed at that position. As a result, the person 705 appears in the video data generated by the camera 3, and the detection omission in the camera 3 is complemented (S15). The image displayed at this time may be an image obtained by capturing the person 705 with another camera, or may be a figure showing the person.

  The person position detection unit 203 performs the person position detection process by repeating the processes from S12 to S15 described above. Note that the above-described position detection processing method is an example, and the person position detection unit 203 can also detect the position of the specific person using another position sensor or the like. For example, it can be detected by a radar or the like.

  Next, details of the operation of the video processing system in the present embodiment having the above-described configuration will be described. FIG. 4 is a flowchart showing the operation of the video processing system. FIG. 4 shows, as an example, the operation of the system when a person who is a customer visits a store and moves in the store. First, a person who is a customer visits a store. Then, the camera 100 installed in the store shoots a person, generates video data, and distributes it to the server device 200. The distributed video data is decoded by the video decoding unit 202 of the server device 200 and input to the person position detection unit 203. The person position detection unit 203 starts processing for detecting the position of the person in the store while tracking the person using the video data (S1). At that time, the person position detection unit 203 assigns a unique tracking ID to the tracking target person included in the video. The position of the person is associated with the tracking ID and managed in the RAM 33 or the like (S1).

  Then, in a state where the person position detecting unit 203 is performing position detection / tracking on the person, the face detecting unit 204 detects the face of the person from the video data and detects the face direction (S2). . Further, for the face of the person detected by the face detection unit 204, the person attribute estimation unit 205 estimates the age and sex of the person based on the image feature amount of the person's face (S3). The attribute data such as age and gender, which is the estimation result, is associated with the same tracking ID as that given in S1 and managed in the RAM 33 or the like. Next, the head position detection unit 206 detects the position of the person's head from the video data, and estimates the height of the person from the detection result (S4). The position of the head can be detected using pattern matching or the like. The estimated height data is associated with the same tracking ID as that given in S1, and is managed in the RAM 33 or the like. In step S3, when the person attribute estimation unit 205 cannot estimate the age, the head position detection unit 206 roughly classifies adults or children from the estimated height data, and based on the classification. Age and gender may be estimated. Also in this case, the attribute data including the estimated age and sex is associated with the same tracking ID as that given in S1 and managed in the RAM 33 or the like.

  Through the above process, the server device 200 continues to perform the tracking process while the person walks in the store. Further, the video data generated by the camera 100 and the information managed as described above by the server device 200 are displayed on the display device 400. FIG. 9 shows an example screen 900 displayed on the display device 400. The cameras 1 to 4 correspond to the camera 100 in FIGS. Screens 901 to 904 represent video data generated by the cameras 1 to 4, respectively. A screen 905 represents the positional relationship between the cameras 1 to 4 and the person. In the video data generated by the cameras 1 and 2, a human face is detected, and the person attribute estimation unit 205 estimates the age and sex from the video data. As described above, the head position detection unit 206 may estimate the age of the person. Furthermore, as shown in the screens 901 to 904, the head position detection unit 206 detects the person's head from the video data generated by the cameras 1 to 4.

Next, the person position detecting unit 203 determines whether or not the person is stationary in front of a certain product shelf from the position detection result (S5). If it is determined that the camera is not stationary (No in S5), the process returns to S2, and the person position detection unit 203 continues to track the person. When it is determined that it is stationary (Yes in S5), the head position detection unit 206 detects the head of the person, estimates the height of the person, and determines whether the height has changed by more than a threshold value. Determine (S6). A change in height can be detected if at least the position of the head can be detected, but it may also be detected from the positions of the foot and head.
If it is determined that the height has not changed (the height of the head has not changed more than the threshold) (No in S6), the process proceeds to S7. When it is determined that the height has changed (the height of the head has changed) (Yes in S6), the face detection unit 204 detects the orientation of the person's face. This detection result is associated with the tracking ID of the person and passed to the information generation unit 207.

  The information generation unit 207 detects a product placed in the direction of the face of the person detected by the face detection unit 204 or the position of the product shelf, and the detected position is a product that the person is interested in. That is, it is specified as the position in the store where the object to be watched by the person is arranged. Then, the information generation unit 207 generates the information on the specified position as information related to a target to be watched by the person (S8). This is because when a person squats down or rides on a stepladder or the like and views the product, it can be determined that the action is to watch the product. In addition, when the height of the head changes, information may be generated as a target on which a person gazes at a different target from the case where the head does not change. For example, when the height of a person's head becomes low, the product at the lower part of the shelf may be an object to be watched. If the server device 200 has a timer (not shown), the information generation unit 207 acquires information on the time when the face detection unit 204 detects the face orientation from the timer, and the person gazes. It may be included in the information regarding the target to be performed. The information generation unit 207 can acquire the above-described information managed in the RAM 33 or the like based on the tracking ID, and can include the acquired information in information related to a target that a person watches. Thereafter, the information generated by the information generation unit 207 is recorded in a database in the storage apparatus 300 via the LAN 500. The information generated by the information generation unit 207 may be sent to the display device 400 via the LAN 500.

  FIG. 10 shows an example screen 1000 displayed on the display device 400. FIG. 10 shows a situation where a person is looking at a product in front of a product shelf in the store. The cameras 1 to 4 correspond to the camera 100 in FIGS. A screen 1001 represents video data generated by any one of the cameras 1 to 4. A screen 1002 shows the positional relationship between the cameras 1 to 4 and the person.

  Returning to FIG. 4, when it is determined in S6 that the height has not changed (No in S6), the head position detection unit 206 directly determines whether or not the position of the person changes more than a threshold value (the person is determined in advance). The determination of whether or not to move beyond the distance is continued for a predetermined time (No in S7). If the position of the person does not change for a predetermined time (Yes in S7), the head position detection unit 206 notifies the face detection unit 204 that the predetermined time has elapsed. Then, the same process as S8 described above is performed.

  After that, while the person can be tracked by the server apparatus 200, it is determined that the person is still in the store, and the server apparatus 200 continues the above processing (No in S9). If the person can no longer track the server apparatus 200, the server apparatus 200 determines that the person has left the store and ends the process (Yes in S9).

  FIG. 11 shows an example screen 1101 displayed on the display device 400, in which video data and information related to a target to be watched by a person are combined. As an example, the screen 1101 displays the face data, gender, time, position of the product shelf that the person is interested in, and identification information.

  As described above, according to the present embodiment, the place where the person who is the customer is stationary for a predetermined time and the place where the height is changed while standing still are the position in the store of the product or shelf where the person is interested. Identified as As a result, it is possible to more accurately grasp the products that the customer is interested in. In the above-described embodiment, the information generation unit 207 uses the face orientation of the person detected by the face detection unit 204 to generate information about the object that the person is gazing at. As long as it is possible to generate information about an object to be watched by the person from the position, the orientation of the person's face need not be used. In this case, the information generation unit 207 receives the determination that the person's height has been changed by the position of the stationary person and the head position detection unit 206, and generates information on the target to be watched by the person. Can do.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Camera, 200 Server apparatus, 300 Storage apparatus, 400 Display apparatus, 201 Communication control part, 202 Image | video decoding part, 203 Person position detection part, 204 Face detection part, 205 Person attribute estimation part, 206 Head position detection part, 207 Information generator

Claims (12)

Person position detecting means for detecting the position of the person in the video imaged by the imaging device;
A head position detecting means for detecting the height of the head of the person;
Generating means for generating information on an object to be watched by the person based on the position of the person and a change in the height of the head of the person;
A video processing apparatus comprising: Attribute estimation means for estimating the attribute of the person based on the video;
The video processing apparatus according to claim 1, wherein the generation unit includes information relating to the attribute of the person in information relating to an object to be watched by the person. Further comprising face detection means for detecting the face of the person based on the video,
The video processing apparatus according to claim 2, wherein the attribute estimation unit estimates the attribute of the person based on the face detected by the face detection unit. The face detection means detects the orientation of the face based on the detected face;
The generation unit generates information on a target to be watched by the person based on a face orientation when the position of the person does not change and a change in the height of the head of the person. Item 4. The video processing apparatus according to Item 3.   When the position of the person does not change for a predetermined time, the generation unit includes the position of the target arranged in the direction of the face at the position of the person in the information regarding the target to be watched by the person. The video processing apparatus according to claim 4.   When the height of the person's head changes, the generation unit includes the position of the target arranged in the direction of the face at the position of the person in the information related to the target to be watched by the person. The video processing apparatus according to claim 4, wherein:   The head position detecting means, when the attribute of the person is not estimated by the attribute estimating means, estimates the attribute of the person based on the height of the head of the person. The video processing apparatus according to any one of 2 to 6.   The video processing apparatus according to claim 2, wherein the attribute of the person includes at least one of a sex and an age of the person.   The video processing apparatus according to claim 1, further comprising display means for displaying the video and information related to an object to be watched by the person. A person position detecting step of detecting the position of the person in the video imaged by the imaging device;
A head position detecting step for detecting the height of the head of the person;
Based on the position of the person and the change in the height of the head of the person, a generating step for generating information on a target to be watched by the person;
A video processing method characterized by comprising: Imaging means for photographing a person;
Person position detecting means for detecting the position of the person in the video imaged by the imaging means;
A head position detecting means for detecting the height of the head of the person;
Generating means for generating information on an object to be watched by the person based on the position of the person and a change in the height of the head of the person;
Display means for displaying the video and information of a target to be watched by the person;
A video processing system comprising:   The program for functioning a computer as each means of the video processing apparatus of any one of Claim 1 to 9.

Images