JP7348529B2 - Image processing system, image processing system control method, and program - Google Patents

Description

The present invention relates to an image processing system, a method for controlling an image processing system, and a program.

In recent years, systems have been proposed that count the number of people using images from network cameras. For example, Patent Document 1 proposes a mechanism that calculates the number of people according to the number of faces in the captured image by acquiring a captured image output from a camera and performing face recognition on the captured image. ing.

Further, Patent Document 2 proposes a mechanism for counting the number of people passing through a specific location by photographing a specific location and performing image processing on the captured video.

Japanese Patent Application Publication No. 2011-221906 Japanese Patent Application Publication No. 2008-217205

By analyzing network camera images (videos) using the mechanisms in Patent Documents 1 and 2, the number of people in the image and the number of people who have passed a predetermined position can be calculated. I can do it.

However, when using the technology proposed in these patent documents, for example, when you want to count the number of people who participated at an event venue, you end up counting people who are not participants at the event, such as staff members. .

Another problem was that it was impossible to calculate the length of stay for each user because people who stayed for a long time and people who stayed only for a short time were counted in the same way.

There was also the problem that the same users were counted multiple times, making it impossible to calculate the correct number of people when it was desired to count the actual number of people.

In view of these problems, it is an object of the present invention to provide a mechanism by which it is possible to easily confirm which people are in the previous image and who are not in the target image. do.

The image processing system according to the present invention includes an acquisition unit that acquires a first image of a predetermined area, a second image of the area taken at a different time from the first image, and Based on the person identified from each of the first image and the second image, the person who is shown in the second image and who is also shown in the first image, and The present invention is characterized by comprising an output control means for controlling to output information indicating a person who is not shown in the first image as character information to the viewing screen.

In the control method for an image processing system according to the present invention, the acquisition means acquires a first image taken of a predetermined area and a second image taken of the area at a different time from the first image. and the output control means, based on the person identified from each of the first image and the second image, the person appearing in the second image, The present invention is characterized by comprising an output control step for controlling to output information indicating a person who is also shown in the image and a person who is not shown in the first image as text information to the viewing screen.

A program according to the present invention is a program executable by an image processing system, and the program is a program that executes the image processing system by capturing a first image of a predetermined area, and capturing the area at a time different from the first image. an acquisition means for acquiring a photographed second image, and a person appearing in the second image based on the person identified from each of the first image and the second image, A program for functioning as an output control means for controlling information indicating a person who is also shown in the first image and a person who is not shown in the first image to be output as text information to a viewing screen. be.

According to the present invention, it is possible to provide a mechanism in which it is possible to easily confirm which persons are included in the target image and which are not included in the previous image.

1 is a system configuration diagram showing an example of the system configuration of an image processing system according to an embodiment of the present invention. 1 is a diagram illustrating an example of a hardware configuration of an information processing apparatus applicable to a server 100 and a client PC 101 of an image processing system according to an embodiment of the present invention. It is a diagram showing an example of the hardware configuration of a network camera 102 in an embodiment of the present invention. FIG. 1 is a functional configuration diagram showing an example of the functional configuration of an image processing system according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of an image used for counting the number of people in an embodiment of the present invention. FIG. 3 is a diagram showing an example of an image used for counting the number of people passing by in an embodiment of the present invention. 3 is a flowchart illustrating an example of image storage processing of the image processing system according to the embodiment of the present invention. 2 is a flowchart illustrating an example of a process for counting the number of people passing by the image processing system according to an embodiment of the present invention. 3 is a flowchart illustrating an example of a people counting process of the image processing system according to an embodiment of the present invention. It is a flow chart showing an example of face recognition processing of the image processing system in an embodiment of the present invention. 3 is a flowchart illustrating an example of viewing processing of the image processing system according to the embodiment of the present invention. It is a diagram showing an example of a person DB of the image processing system in the embodiment of the present invention. It is a figure showing an example of the viewing screen of the number of people count in an embodiment of the present invention. FIG. 3 is a diagram showing an example of a viewing screen for counting the number of people passing by according to an embodiment of the present invention. 7 is a flowchart illustrating an example of a process for counting the number of people passing by an image processing system according to another embodiment. It is a figure showing an example of person position information DB. It is a diagram showing an example of a boundary crossing information DB. It is a diagram showing an example of a line counter DB. 12 is a flowchart illustrating an example of a process for counting the number of people in an image in an image processing system according to another embodiment. FIG. 3 is a diagram showing an example of a viewing screen for counting the number of people.

Embodiments of the present invention will be described in detail below with reference to the drawings. Note that the embodiment described below shows an example of a case where the present invention is specifically implemented, and is one of the specific embodiments of the configuration described in the claims.

FIG. 1 is a system configuration diagram showing an example of the system configuration of an image processing system according to an embodiment of the present invention.

In an image processing system (information processing system) according to an embodiment of the present invention, a server 100, a client PC 101, and a network camera 102 which is an imaging device are communicably connected via a wireless (or wired) LAN 120. Note that the system configuration shown in FIG. 1 is one example, and there are various configuration examples depending on the usage and purpose.

The server 100 stores and manages image data captured by the network camera 102. Further, the server 100 has a function of analyzing image data stored and managed by the server 100, and counting the number of users included in the image data and counting the number of people who have passed through a predetermined area.

The image management system according to the embodiment of the present invention includes a plurality of network cameras 102 [network cameras 102 (A) and (B)], and communicates using the TCP/IP protocol. A fixed IP address is uniquely assigned, and an external port number is assigned to each network group.

Therefore, the server 100 can identify from which network camera 102 the received image data was transmitted, and can distinguish between the received image data and which network camera 102 it was transmitted from. It is possible to store and manage the information in association with each other.

The network camera 102 is installed, for example, in a meeting space or an event venue on the floor of a corporate building (hereinafter also referred to as a "predetermined area"). For example, the network camera 102 (A) captures still images or A video is taken, and the network camera 102(B) takes a still image or a video to identify the person photographed.

Examples of the predetermined area include an area 106 indicating the entire area, area A103, and area B104. A position 105 provided between the area A103 and the area B104 is a reference position when counting the number of users (USERs) who have passed through the position 105. That is, this position is a boundary that separates area A103 and area B104. This position may be set linearly, circularly, or rectangularly.

The client PC 101 is used by a user to view image data (video from the network camera 102) managed by the server 100. A user can view image data by logging into the monitoring system via the client PC 101.

Next, an example of the hardware configuration of an information processing apparatus applicable to the server 100 and client PC 101 shown in FIG. 1 will be described using FIG. 2.

FIG. 2 is a diagram showing an example of the hardware configuration of an information processing apparatus applicable to the server 100 and client PC 101 of the image processing system according to the embodiment of the present invention.

In FIG. 2, 201 is a CPU that centrally controls each device and controller connected to the system bus 204. In addition, the ROM 202 or external memory 211 contains BIOS (Basic Input/Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and various other programs described below that are necessary to realize the functions executed by the PC. Programs etc. are stored.

203 is a RAM, which functions as the main memory, work area, etc. of the CPU 201; The CPU 201 loads programs and the like necessary for execution of processing from the ROM 202 or the external memory 211 into the RAM 203, and executes the loaded programs to realize various operations.

Further, an input controller 205 controls input from a pointing device such as a keyboard (KB) 209 or the like. A video controller 206 controls display on a display such as a CRT display (CRT) 210. Although the CRT 210 is shown in FIG. 2, the display device is not limited to a CRT, but may be another display device such as a liquid crystal display.

207 is a memory controller that stores boot programs, various applications, font data, user files, editing files, various data, etc. in an external storage device (hard disk (HD)), flexible disk (FD), or PCMCIA card slot. Controls access to external memory 211 such as CF memory connected via an adapter.

A communication I/F controller 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP/IP is possible.

Note that the CPU 201 enables display on the CRT 210 by, for example, executing an outline font development (rasterization) process in a display information area in the RAM 203. Further, the CPU 201 allows the user to give instructions using a mouse cursor (not shown) on the CRT 210 .

Various programs to be described later for realizing the present invention are recorded in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. Further, configuration files and the like used when executing the above program are also stored in the external memory 211, and detailed explanations thereof will be given later. This concludes the explanation of FIG. 2.

Next, an example of the hardware configuration of the network camera 102 shown in FIG. 1 will be described using FIG. 3.

FIG. 3 is a diagram showing an example of the hardware configuration of the network camera 102 in the embodiment of the present invention.

The CPU 301 centrally controls each device and controller connected to the system bus 304. In addition, the ROM 302 or external memory 305 contains BIOS (Basic Input/Output System), which is a control program for the CPU 301, an operating system program (hereinafter referred to as OS), and other programs necessary to implement the functions executed by the image processing server 108. Various programs and the like, which will be described later, are stored. The RAM 303 functions as the main memory, work area, etc. of the CPU 301.

The CPU 301 loads programs and the like necessary for execution of processing into the RAM 303 and executes the programs to realize various operations.

A memory controller (MC) 306 is a CF that is connected via an adapter to a hard disk (HD) or PCMCIA card slot that stores boot programs, various applications, font data, user files, editing files, various data, image data, etc. Controls access to external memory 305 such as memory and SmartMedia (registered trademark).

The camera unit 307 is connected to an image processing unit 308, and after photoelectrically converting the light obtained by passing through a lens directed toward the monitoring target using a light receiving cell such as a CCD or CMOS, the camera unit 307 converts the light into an RGB signal. and complementary color signals to the image processing unit 308.

The image processing unit 308 performs white balance adjustment, gamma processing, and sharpness processing based on the RGB signal and color capture signal, and further performs YC signal processing to generate a luminance signal Y and a chroma signal (hereinafter referred to as YC signal). Then, the YC signal is compressed in a predetermined compression format (for example, JPEG format or Motion JPEG format), and this compressed data is temporarily stored in external memory 305 as image data.

A communication I/F controller (communication I/FC) 309 connects and communicates with external devices via a network, executes communication control processing on the network, and stores images stored in the external memory 305. The data is transmitted to the external device by the communication I/F controller 309. This concludes the explanation of FIG. 3.

FIG. 4 is a functional configuration diagram showing an example of the functional configuration of an image processing system according to an embodiment of the present invention.

In the image processing system according to this embodiment, the server 100 has the following functions: an image acquisition section 411, a people counting section 412, a person recognition section 413, a person exclusion section 414, and a result screen display control section 415. Note that the server 100 is not limited to the functional configuration illustrated in FIG. 4, and includes, for example, a storage unit that stores persons determined to be excluded from counting, and passing direction information indicating the direction in which the persons have passed through a predetermined position. It may also have various other functional configurations, such as a passing direction acquisition unit that acquires information, a stay time calculation unit that calculates the time that a person stays in a predetermined area, and the like.

The image acquisition unit 411 acquires an image photographed by the network camera 102, as explained in FIG. Images include still images and videos. In the embodiment, the network camera 102(A) photographs a predetermined area, and the network camera 102(B) is used to recognize an identified person (for example, facial recognition) and identify the individual.

The people counting unit 412 uses images taken by the network camera 102(A) to count the number of people in a predetermined area as explained in FIG. Count the number of people who passed through the location.

The person recognition unit 413 uses the image taken by the network camera 102(B) to authenticate the person shown in the image. An example of the authentication process will be described with reference to FIG.

The person exclusion unit 414 determines, based on the image acquired by the image acquisition unit 411, whether the person in the image is a count exclusion target to be excluded from the number of people counted by the number of people counting unit 412. The person exclusion unit 414 performs processes such as identifying a person to be excluded from the number of people counted by the number of people counting unit 412, and storing authenticated people in a person table or a passing person table shown in FIG. 12. The person exclusion unit 414 determines whether a person who has passed a predetermined position is to be excluded from counting, based on, for example, passing direction information indicating the direction in which the person in the image has passed the predetermined position. Furthermore, the person exclusion unit 414 determines whether or not the person in the image is to be excluded from counting based on the time the person in the image stays in a predetermined area.

The result screen display control unit 415 performs processing to control display of image processing results on the user's terminal according to the user's request.

FIG. 5 is a diagram showing an example of an image used for counting the number of people in the embodiment of the present invention.

Image A501 is an image taken of a predetermined area at a certain place, and image B502 is an image taken of the same point and same area as image A at a different time. These images are managed as images to be compared.

In Figure 5, image A contains five people from Mr. A to Mr. E, and for example, in image B, which is an image taken 5 minutes later, Mr. D and Mr. E are missing even though they are the same five people. , is an example in which Mr. F and Mr. G are photographed. Here, for example, Mr. A, Mr. B, and Mr. C are users who were also shown in the image B taken five minutes ago, so they are excluded from the number of people count. That is, the person exclusion unit 414 determines Mr. A, Mr. B, and Mr. C, whose stay time in the predetermined area 501 is 5 minutes or more, to be excluded from counting. The number of people counting unit 412 then counts the number of people while excluding the person who has been determined by the person exclusion unit 414 to be excluded from counting.

It is also possible to set, for example, that Mr. A is excluded from the number of people count because he is a staff member. That is, the person exclusion unit 414 may determine that Mr. A is to be excluded from counting based on the fact that he is a predetermined person. Note that the number of people counting section 412 is not limited to the persons determined to be excluded from counting by the person excluding section 414, and may exclude persons set in advance by the administrator or the like when counting the number of people.

This embodiment is configured to be able to display changes in the person photographed on the screen shown in FIG. 13.

FIG. 6 is a diagram showing an example of an image used for counting the number of people passing by in the embodiment of the present invention.

The images are images taken of a predetermined area at a certain place, and time has elapsed in the order of image A601→image B602→image C602.

Here, the number of people who moved from area A to area B (or vice versa, from area B to area A) is counted based on a predetermined position 604, and the number of people who move in the same direction multiple times or the default user is counted. can be excluded from counting. Here, the same direction is not limited to the exact same direction on a straight line from a certain point to a certain point, but refers to, for example, the same direction as the direction from the first area to the second area. For convenience, the case where a person moves from area A to area B will be counted as "OUT", and the case where a person moves from area B to area A will be counted as "IN". In this case, the same direction means that both are "OUT" (or both are "IN").

Image A shows Mr. C moving from area A to area B (moving a predetermined position 604 from area A to area B) (OUT).

Image B shows Mr. C moving from area B to area A, and Mr. D moving from area B to area A (IN).

Image C shows Mr. C moving from area A to area B again, and Mr. A moving from area A to area B (OUT). Here, since Mr. C is moving in the same direction for the second time, he is excluded from the people count. That is, the person exclusion unit 414 determines that Mr. C, who passes through the predetermined position 604 multiple times in the same passing direction, is to be excluded from counting. The number of people counting unit 412 then counts the number of people while excluding the person who has been determined by the person exclusion unit 414 to be excluded from counting.

Furthermore, since Mr. A is set to be excluded in advance, he is excluded even if it is his first time moving. That is, the person exclusion unit 414 may determine that Mr. A is to be excluded from counting based on the fact that he is a predetermined person. Note that the number of people counting section 412 is not limited to the persons determined to be excluded from counting by the person excluding section 414, and may exclude persons set in advance by the administrator or the like when counting the number of people.

FIG. 7 is a flowchart illustrating an example of image storage processing of the image processing system according to the embodiment of the present invention.

This process is a process of transmitting an image taken by the network camera 102 to the server 100 and storing it.

In step S701, the network camera 102 photographs the monitoring area (region 106), and transmits the photographed image to the server in step S702.

In step S703, the server 100 receives the image from the network camera 102, and in step S704, stores the received image as a recording file in the server's external memory 211 or the like, and ends the process.

Using the image stored here, the image processing system counts the number of users appearing in the image or the number of people passing by.

FIG. 8 is a flowchart illustrating an example of a process for counting the number of people passing by the image processing system according to the embodiment of the present invention.

This process is a process of counting how many users have passed through a predetermined position (for example, 604) set in a predetermined area (for example, 601), and is a process that counts the number of users who have passed through a predetermined position (for example, 604) set in a predetermined area (for example, 601). Process as a target.

In step S801, processing is repeated for a predetermined period of time designated by the user or administrator. In this embodiment, explanation will be given assuming that 5 minutes from 9:00 to 9:05 is specified. Furthermore, in reality, a five-minute video consists of a large number of images, but in this embodiment, for convenience, three images will be used for explanation.

Assume that the image 601 is taken at 9:00, the image 602 is taken at 9:03, and the image 603 is taken at 9:05.

In step S802, line passage detection area information is read. Specifically, it is determined whether a person has passed through a predetermined position 604.

In step S803, if the person can be identified (recognized), the process proceeds to step S804, and if the person cannot be identified (recognized), the process proceeds to step S811, where a loop determination (whether a predetermined period of time has elapsed or not) is performed.

In step S804, passing direction information indicating the direction in which the person determined to have passed through the predetermined position in step S802 passes through the predetermined position is acquired by image analysis processing or the like (passing direction acquisition means). The passing direction information is information indicating either a first direction from area A to area B or a second direction from area B to area A. (first direction) is defined as "OUT", and movement from area B to area A (second direction) is defined as "IN" (the reverse is also possible).

In step S805, processing is performed to identify the person recognized in step S803. This process, which will be explained with reference to FIG. 10, is a process for specifying a recognized person (person specifying means). This process makes it possible to exclude unnecessary people from the people count among the identified people.

In step S806, it is determined whether the person recognized in step S803 is to be excluded from counting. If it is determined in step S806 that the target person is to be excluded from counting, the process proceeds to step S808, where the movement of the person is stored as a subject to be excluded from counting (storage means) so that it is not reflected in the number of people count, and the process proceeds to step S809. move on. If it is determined in step S806 that the target person is not subject to count exclusion, the process proceeds to step S807, where the IN/OUT movement count is incremented, and the process proceeds to step S809.

In step S809, it is confirmed whether all the processes for the person recognized in step S803 have been completed. If not, the process returns to step S804 and the process is performed for the next person. If the process for all the people has been completed, If so, the process advances to step S810.

In step S810, the contents processed in this flow are saved in a result table, and the process ends. Note that this process may be saved all at once in this step, or may be saved each time in each step of the flowchart.

Next, the result table (person table) saved in this flowchart will be explained using FIG. 12.

FIG. 12 is a diagram showing an example of the person DB of the image processing system according to the embodiment of the present invention. The person DB 1200 is a result table in which the results of the people counting process described in FIG. 9 are stored. The person DB 1200 stores the user names and types of users recognized in images A and B. Details will be described later.

The passing person DB 1210 is an example of a passing person DB, which is a result table saved in the line passing counting process shown in FIG.

The passing person DB 1210 stores an image 1211, a time 1212 when the image was taken, a passing direction 1213, a user 1214, and a counter 1215 in association with each other.

The image 1211 is an image name, for example, the file name of the image. Images A to C correspond to images A601 to C603 in FIG. 6, respectively, and were photographed at 9:00, 9:03, and 9:05, respectively.

As the passing direction 1213, the direction detected in step S804 is stored, and as the user 1214, the user specified in step S805 is stored.

The counter 1215 is an IN/OUT counter in step S807, and is a counter for the number of IN users and OUT users. For example, it can be seen that 2 people were counted as OUT and 1 person was counted as IN.

Also, Mr. C in image C is excluded. This is due to the setting that "moves by the same user in the same direction within a predetermined period of time are excluded from counting." That is, the person exclusion unit 414 determines that the same user who moves in the same direction within a predetermined period of time is to be excluded from counting, and the determination result is stored in the result table.

Also, Mr. A in image C is excluded. This is due to the setting of "Mr. A is (excluded) by default" as shown in FIG. That is, the person exclusion unit 414 determines that the predetermined user is a count exclusion target, and the determination result is stored in the result table. Settings other than these settings can be set in advance by the administrator.

FIG. 9 is a flowchart illustrating an example of the number of people counting process of the image processing system according to the embodiment of the present invention.

This process is a process of counting how many users are photographed in a predetermined area (for example, 501), and targets images stored in the server.

For example, there are 5 people in both image A501 and image B502, but by comparing the two images, you can exclude the same users, exclude certain users, or display only the number of new users. You can perform the following processing. In other words, it is possible to determine whether or not the person is to be excluded from counting based on the image, and to exclude the person who has been determined to be excluded from counting from the number of people count, regardless of whether the person is to be excluded from counting based on the image. It can be carried out.

In addition, since this process compares images, it counts the number of people who have set a predetermined number of images, and in this embodiment, the predetermined number of images is 2 for image A and image B. .

In step S901, area information is read. The area information refers to, for example, the area 106 in FIG. 1, and corresponds to an image of the area preset and photographed by the network camera 102.

In step S902, it is determined whether or not a person has been identified (recognized) within the area. If the person has been identified, the process advances to step S903 and person identification processing is performed. The person identification process will be explained with reference to FIG. 10, and the person identification process determines whether the recognized person is an exclusion target.

In step S904, it is determined whether the person recognized in step S902 is an exclusion target. If it is determined that the person is an exclusion target, the process advances to step S906; if it is determined that the person is not an exclusion target, the process proceeds to step S906. The process advances to step S905.

In step S906, the person determined to be excluded is not reflected in the number of people count, and the process advances to step S907. In step S905, the number of people recognized in step S902 is counted, and the process advances to step S907.

In step S907, it is determined whether processing has been completed for all images (in this case, two images). If the processing has not been completed, the process returns to step S903; if the processing has been completed, the process proceeds to step S908. , save the processing contents in the result table and end the processing. Note that this process may be saved all at once in this step, or may be saved each time in each step of the flowchart.

The result table saved in this flowchart will be explained using the person DB 1200 in FIG. 12. Image A is an image taken at 9:00, and image B is an image taken at 9:05.

The person DB 1200 manages, for each image, the number of recognized users and whether the users are excluded.

The person DB 1200 corresponding to the image A stores a user type 1201 and a user name. The person DB 1200 stores information on five users A to E, who are recognized based on the image 501 in FIG. 5, as user names 1202. Further, the user type 1201 of user A, who is excluded by default, is set to "default." .

The person DB 1200 corresponding to image B stores information on five users A to C, F, and G as user names 1204. The user types 1203 of users F and G, who are users (new) who do not exist in image A, are blank.

In addition, the user type 1204 of user A is set to "default," and the user types 1204 of users A to C, who were also identified in image A, are stored as "identified" (identified/A). There is. That is, the person exclusion unit 414 determines that the same user located on the same floor is to be excluded from counting even after a predetermined period of time has elapsed, and stores the determination result in the result table. Here, specified users and default users are treated as excluded targets.

FIG. 10 is a flowchart illustrating an example of face recognition processing by the image processing system according to the embodiment of the present invention.

This face recognition process corresponds to the person identification process in step S805 in FIG. 8 and step S903 in FIG. In this embodiment, a person identification process using face recognition will be described. However, as long as a person can be identified, it is not limited to the face recognition process, and any technique may be used.

In step S1001, an image of a person recognized in an image is received, and in step S1002, feature points of the person are extracted to identify the person. The process of extracting feature points of a person and identifying the person uses a technology that has already been proposed.

In step S1003, it is checked whether the identified person is registered in the result table (passing person DB 1210 in the case of FIG. 8, person DB 1200 in the case of FIG. 9). If the information has not been registered, the process advances to step S1004.

In step S1006, the person registered in the result table is returned as an excluded person (exclusion target). In the case of the person DB 1200, "registered" is determined based on whether the user types 1201 and 1203 are registered as exclusion targets. Specifically, in the case of image A, Mr. A is excluded because the user type is default, and in the case of image B, the default person A and identified persons A to C are excluded people (exclusion targets). A reply is sent and the process ends.

In addition, in the case of the passing person DB 1210, the default Mr. A and Mr. C (OUT) of the image C who moves for the second time (moved multiple times) in the same direction are returned as excluded persons (exclusion target) and processed. ends.

In step S1004, the person determined to have not been registered in step S1003 is registered (as excluded) in the result table. For example, in the person DB 1200, persons A to C, who have not been registered based on being identified in the person DB 1200 corresponding to image A, are identified by image A in the person DB 1200 corresponding to image B. is registered.

In addition, in the passing person DB 1210, Mr. C (OUT) of image C, which is a case where the same user is moving in the same direction and is not registered, and Mr. A (OUT) of image C, who is the default user, are shown.・Default) is registered as an exclusion target. Subsequently, in step S1005, the person registered in step S1004 is returned as a non-excluded person (non-excluded target), and the process ends.

FIG. 11 is a flowchart showing an example of the viewing process of the image processing system according to the embodiment of the present invention.

This process is a process for controlling the display of image-processed results on the user's terminal based on instructions from the user.

In step S1101, a camera to be displayed is selected. For example, the serial number and name of the network camera 102(A) are selected, and an image taken by the selected camera is requested in step S1102. This image may be in a still image format, a moving image format, or a LIVE moving image.

In step S1103, it is determined whether the requested image is a count of people (FIG. 5) or a count of people passing through a line (FIG. 6). If it is a count of people, the process advances to step S1104; if it is a count of people passing through the LINE, the process advances to step S1105. move on.

In step S1104, processing is performed to display a viewing screen for counting the number of people shown in FIG. 13 as the number of people information. In step S1105, processing is performed to display a viewing screen for counting the number of people passing through as shown in FIG. Details of the viewing screen will be described later using FIGS. 13 and 14.

In step S1106, processing is performed to synthesize images of items to be displayed on the viewing screen, and in step S1107, screen display control is performed to display on the client's screen, and the processing ends.

FIG. 13 is a diagram showing an example of a viewing screen for counting the number of people according to the embodiment of the present invention.

Two examples of the people count viewing screen 1301 and the people count viewing screen 1310 will be described.

On the people count viewing screen 1301, the current image 1302, the number of people recognized (5 people) 1303, the default exclusion target (1 person) 1304, the individual identified exclusion target (3 people) 1305, and new information on the screen The recognized person and number of people (2 people) 1306 are displayed.

The number of people count viewing screen 1310 is another example of the viewing screen of 1301, and can also be displayed including the display contents of 1301.

The counted number of people excluding excluded persons, the default number of excluded persons 1311, and the acquired image are displayed.

In addition, visibility may be improved by displaying the counted people with a thick frame or other frame, and Mr. A, who is the default excluded person, can be distinguished from other users with different colors and lines. may be displayed.

In addition, the display may be different, such as displaying a combined user name (for example, Mr. A or Mr. B) for users whose individuals could be identified in the person identification process, and displaying nothing for users whose individuals could not be identified. is also valid.

FIG. 14 is a diagram showing an example of a viewing screen for counting the number of people passing through according to the embodiment of the present invention.

Two examples of the passing number count viewing screen 1401 and the passing number counting viewing screen 1410 will be described.

On the passing number count viewing screen 1401, the number of people who passed through a predetermined line (position) and the direction 1402, a predetermined exclusion target (one person) 1403, and the number of people and number of people who passed through the line twice 1404 are displayed.

Passage count viewing screen 1410 is another viewing screen example of 1401, and can also be displayed including the display contents of 1401.

The number of people passing through/direction/number of exclusions 1411 and the acquired image are displayed. In the embodiment, IN (1 person), OUT (2 people), and the number of exclusions (2 people), but this number of exclusions includes both 1403 and 1404 exclusion targets.

In the image, visibility may be improved by displaying the people counted by passing the line in a frame such as a thick frame, and Mr. A, who is the default excluded person, may be displayed in a different color or with a line. It may be displayed so that it can be distinguished from other users. In addition, it is also possible to display the user 1412 who did not pass the line without displaying a coordinated display such as a frame line (for convenience, it is shown as a dotted line, but there is no need to display anything) so that the user 1412 can be identified as the person who passed the line. It is.

In addition, the display may be different, such as displaying a combined user name (for example, Mr. A or Mr. B) for users whose individuals could be identified in the person identification process, and displaying nothing for users whose individuals could not be identified. is also valid.

Next, an image processing system according to another embodiment that can be applied to the process of counting the number of people passing by will be described. In this other embodiment, the person exclusion unit 414 determines whether a person who has crossed the boundary line is a person who wanders on the boundary line, that is, a person who repeatedly passes (straddles) a predetermined position within a predetermined time (for example, about 5 minutes). This embodiment differs from the above embodiment in that it is determined whether the object is to be excluded from counting based on whether the object is a person or not. Hereinafter, this determination will also be referred to as "wandering exclusion determination."

Hereinafter, this other embodiment will be described in detail with reference to FIG. 15. FIG. 15 is a flowchart illustrating an example of a passer count process of an image processing system according to another embodiment.

This process involves counting the number of users passing through a predetermined boundary line (for example, position 604) set in a predetermined area (for example, the area shown in image 601) within a predetermined time. This process excludes people who repeatedly pass over the boundary line from the people count. This process is performed by the server 100 of the image processing system.

This process starts, for example, when a start instruction for starting this process is input by the user. Further, this process may be performed, for example, when it is determined that all persons have been analyzed in step S1504, which will be described later, when a termination instruction is input by the user, or when a preset processing time is completed, etc. finish.

First, in step S1501, boundary line information and a no-loitering interval time are acquired. The boundary line information and the no-loitering time interval are obtained, for example, from setting information set in advance by the user or the like.

The boundary line information is position information of the boundary line, and is indicated by, for example, pixel coordinates with the upper left corner of the image 601 as a reference. The non-loitering interval time is the interval at which the same person passes the boundary line multiple times, rather than simply passing through the boundary line to move from area A to area B (or from area B to area A). This is the time during which a person is considered to be wandering around on the boundary line, and is set to, for example, 5 minutes.

Subsequently, in step S1502, the image acquisition unit 411 acquires an image captured by the network camera 102. The image acquisition unit 411 acquires images taken within a predetermined time, for example.

Subsequently, in step S1503, the person recognition unit 413 recognizes all the people and their positions in the image acquired in step S1503, and creates a list of all the people and their positions on the memory.

Subsequently, in step S1504, it is determined whether or not there is a person who has not been analyzed on the list created in step S1503. Here, analysis means performing the processing of steps after step S1505, which will be described later.

If it is determined that there is no person who has not been analyzed (step S1504; NO), that is, if all the persons have been analyzed, the process ends. Note that in this case, the process may be returned to step S1503 without ending the process, the image may be acquired again, and the process from step S1504 onward may be repeated.

If it is determined that there is a person who has not been analyzed (step S1504; YES), the process advances to step S1505. That is, the processing from step S1505 onward is repeated until the analysis is completed for all persons.

Subsequently, in step S1505, information on the next person to be analyzed is acquired from among the unprocessed persons on the list created in step S1503 who have not yet been analyzed.

Subsequently, in step S1506, the person recognition unit 413 recognizes the same person as the person indicated by the person information to be analyzed next acquired in step S1505.

Subsequently, in step S1507, the person recognition unit 413 caches the position information of the person recognized as the same person in step S1506. Specifically, it is saved as a person position information DB 1600 as shown in FIG. FIG. 16 is a diagram showing an example of the person position information DB 1600.

As shown in FIG. 16, the person location information DB 1600 stores a time stamp 1601, a person ID 1602, a camera ID 1603, and location information 1604 in association with each other.

The time stamp 1601 is a character string indicating the date and time when the image in which the person was recognized was photographed by a network camera. In the example of FIG. 16, the dates and times of timestamps 1601 are stored in chronological order from the top. For example, "2018.07.27 12:00:01:00" is stored as the timestamp 1601, followed by "2018.07.27 12:00:00:00" as the timestamp 1601. In other words, the date and time whose timestamp 1601 is "2018.07.27 12:00:00:00" and the date and time whose timestamp 1601 is "2018.07.27 12:00:01:00" are chronologically different from each other. In context. Hereinafter, among the dates and times in such a context, the time before the time stamp 1601 is "2018.07.27 12:00:00:00" is simply referred to as "previous time", and the time before the time stamp 1601 is "2018.07. 27 12:00:01:00'' is also simply referred to as ``later time.''

Although illustration is omitted in the example of FIG. 16, in reality, data associated with a timestamp 1601 of a predetermined time including a date and time earlier than the date and time of the illustrated timestamp 1601 is stored. There is.

The person ID 1602 is identification information indicating the person recognized in step S1506. The camera ID 1603 is identification information indicating the network camera 102 that photographed the person recognized in step S1506. The position information 1604 is the position information of the person recognized in step S1506, and is indicated by pixel coordinates with the upper left corner of the image 601 as a reference, for example. The position information 1604a is position information indicated by the X coordinate, and the position information 1604b is position information indicated by the Y coordinate.

Referring again to FIG. 15, in the following step S1508, the moving position and moving direction of the person recognized as the same person in step S1506 within a predetermined time, such as one hour, is calculated based on the person position information DB 1600.

The movement position is a position to which the same person has moved within a predetermined period of time. The movement position is calculated as, for example, each position indicated by the position information 1604 associated with the same person ID 1602 and the time stamp 1601 within a predetermined time.

The movement direction is the direction in which the same person moved within a predetermined period of time. The movement direction is calculated, for example, as a vector indicating a direction from a position at a previous time to a position at a later time, based on each calculated position.

Subsequently, in step S1509, it is determined whether the same person has passed through the boundary line based on the boundary line information acquired in step S1501 and the movement position and movement direction calculated in step S1508.

If it is determined that the same person has not passed through the boundary line (step S1509; NO), the process returns to step S1504, and it is determined whether there is a person who has not been analyzed. If it is determined that the same person has passed through the boundary line (step S1509; YES), the process advances to step S1510.

In step S1510, boundary line passage information about the person determined to have passed through the boundary line in step S1509 is cached. Specifically, the boundary line passage information is saved as a boundary crossing information DB 1700 as shown in FIG. FIG. 17 is a diagram showing an example of the boundary crossing information DB 1700.

As shown in FIG. 17, the boundary crossing information DB 1700 stores a time stamp 1701, a person ID 1702, a camera ID 1703, and a boundary crossing direction 1704 in association with each other.

The time stamp 1701 is a character string indicating the time when the same person passed the boundary line. The person ID 1702 is identification information indicating the person determined to have passed the boundary line in step S1509. The camera ID 1703 is identification information indicating the network camera 102 that photographed the person who was determined to have passed through the boundary line in step S1509.

The boundary crossing direction 1704 is information indicating the direction in which the person who was determined to have passed the boundary line crossed the boundary line in step S1509, and is indicated by the aforementioned "IN" or "OUT". The boundary crossing direction 1704 is calculated (passing direction information acquisition means) based on, for example, the movement position and movement direction calculated in step S1508 and the boundary line information acquired in step S1501, and is stored.

Subsequently, in step S1511, the boundary crossing information DB 1700 stored in step S1510 is referred to, and a plurality of pieces of information about the same person passing the boundary line within a predetermined time (for example, one hour) is searched.

Subsequently, in step S1512, it is determined based on the information retrieved in step S1511 whether or not the vehicle is wandering through the boundary line. For example, within the wandering-free interval time (for example, 5 minutes) acquired in step S1501, there is one movement in the opposite direction (from “IN” to “OUT” or from “OUT” to “IN”). If there are any of the above, it is determined that the object is passing through the boundary line.

If it is determined in step S1512 that the object is passing through the boundary line (step S1512; YES), the process advances to step S1513. In step S1513, the person exclusion unit 414 determines that the person who has passed the boundary line is to be excluded from counting, and the number of people counting unit 412 excludes from counting the number of people who have been determined to be excluded from counting. After step S1513, the process returns to step S1504, and it is determined again whether there is a person who has not been analyzed.

Note that in this specification, excluding from the count includes not only the case where the count of the number of people is not incremented, but also the case where the count that has been incremented once is decremented.

If it is determined in step S1512 that the boundary line has not been wandered through (step S1512; NO), the process advances to step S1514, and the same process as normal people counting process is performed. Specifically, first in step S1514, it is determined whether the same person has moved in the same direction multiple times based on the information searched in step S1511.

If it is determined in step S1514 that the person has moved in the same direction multiple times (step S1514; YES), the process proceeds to step S1515, where the person exclusion unit 414 determines that the person who has passed the boundary line is to be excluded from counting, and The counting unit 412 excludes from the count the number of people who are determined to be excluded from counting. After step S1514, the process returns to step S1504, and it is determined again whether there is a person who has not been analyzed.

If it is determined in step S1514 that the number of people has not moved in the same direction multiple times (step S1514; NO), the process advances to step S1516, and the count of the number of people who have passed through the boundary line is incremented for each IN/OUT passing direction. After step S1516, the process returns to step S1504, and it is determined again whether there is a person who has not been analyzed.

The result of the number of people counted in step S1516 is stored as a line counter DB 1800 as shown in FIG. 18, for example. FIG. 18 is a diagram showing an example of the line counter DB1800.

As shown in FIG. 18, the line counter DB 1800 stores a time stamp 1801, a camera ID 1802, and a count number 1803 in association with each other.

The timestamp 1801 is a character string indicating the most recent date and time when the number of people count was incremented. The camera ID 1802 is identification information indicating the network camera 102 that photographed the person whose number of people has been incremented. The count number 1803 is the number of people incremented for each camera ID and passing direction, and is the total number of people added from a certain starting point. The count number 1803a is the number of people who passed in the "IN" direction, and the count number 1803b is the number of people who passed in the "OUT" direction.

As shown in FIG. 18, in step S1506, the camera ID and the number of people counted for each passing direction are stored.

As described above, according to the present embodiment, after the same person moves in the first direction (for example, "IN") based on the passing direction in which the person passed the boundary line, the person moves in the second direction within the predetermined wandering-free interval time. (for example, "OUT"), the same person is excluded from the passing count. Therefore, people who are not passing the boundary line to move from area A to area B (or from area B to area A), but who are simply wandering around on the boundary line, can be counted as people passing by. can be removed as noise.

Next, an image processing system according to another embodiment that can be applied to count the number of people in an image will be described. This other embodiment differs from the above embodiment in that the person exclusion unit 414 determines at least one of long-term stayers and short-time stayers in a predetermined area to be excluded from counting.

Hereinafter, this other embodiment will be described in detail with reference to FIG. 19. FIG. 19 is a flowchart illustrating an example of a process for counting the number of people in an image in an image processing system according to another embodiment.

This process counts at least one of long-term stayers and short-time stayers in a predetermined area (for example, a part of the area 601) in a people counting process that counts the number of people included in an image. This is a process to exclude from This process is performed by the server 100 of the image processing system.

This process starts, for example, when a start instruction for starting this process is input by the user. Further, this process may be performed, for example, when it is determined that all persons have been analyzed in step S1904, which will be described later, when a termination instruction is input by the user, or when a preset processing time is completed, etc. finish.

First, in step S1901, the location information of the stay area and the stay determination time are acquired. The position information of the predetermined area and the stay determination time are acquired from, for example, setting information set in advance by the user or the like.

This is a predetermined area that is part of the stay area, for example, the area 601, and is an area surrounded so as to have a predetermined shape such as a rectangle or a circle. The positional information of the stay area is indicated, for example, in pixel coordinates with the upper left corner of the image showing the area 601 as a reference. The stay determination time is a time for determining whether the same person stays in the stay area for a long time or a short time. The stay determination time includes, for example, a long stay determination time (for example, 1 hour) to determine whether the stay is long, and a short stay determination time (for example, 1 hour) to determine whether the stay is short. 10 minutes).

The subsequent processing from step S1902 to step S1907 is the same as the processing from step S1502 to step S1507 in FIG. 15 described above, so the description thereof will be omitted.

In step S1908 following step S1907, based on the location information of the stay area acquired in step S1901 and the position information of the person cached in step S1907, whether the same person recognized in step S1906 is in the stay area? Determine whether or not.

If it is determined in step S1908 that the same person is not in the stay area (step S1908; NO), the process returns to step S1904, and it is determined whether there is a person who has not been analyzed. If it is determined in step S1908 that the same person is in the stay area (step S1908; YES), the process advances to step S1909.

In step S1909, the stay time of the same person determined to be in the stay area in step S1908 is calculated. Specifically, based on the person location information DB 1600 stored in step S1907, the latest time period associated with the same person determined to be in the stay area is searched, and the oldest time in the relevant time period is searched. By calculating the difference from the latest time, the stay time of the same person is calculated.

Subsequently, in step S1910, it is determined whether the stay time calculated in step S1909 is equal to or longer than the long stay determination time acquired in step S1901. That is, it is determined whether or not the same recognized person is a long-stay person who has stayed in the stay area for a predetermined period of time (for example, one hour) or more.

If it is determined in step S1910 that the stay time is equal to or longer than the long stay determination time (step S1910; YES), the process advances to step S1911. In step S1911, the person exclusion unit 414 determines that the person who is determined to be a long-term stayer is to be excluded from counting, and the number of people counting unit 412 excludes from counting the number of people who are determined to be excluded from counting. . After step S1911, the process returns to step S1904, and it is determined again whether there is a person who has not been analyzed.

The person excluded from counting in step S1911 is stored as a subject to be excluded from counting due to long stay at the timing of exclusion from counting. Further, the number of people excluded from counting in step S1911 is incremented as the number of people excluded from counting due to a long stay at the timing of exclusion from counting.

If it is determined in step S1910 that the stay time is not longer than the long stay determination time (step S1910; NO), the process advances to step S1912. In step S1912, it is determined whether the stay time calculated in step S1909 is equal to or longer than the short stay determination time acquired in step S1901. That is, it is determined whether the same recognized person is a short-time visitor who has stayed in the stay area for less than a predetermined time (for example, 10 minutes).

If it is determined in step S1912 that the stay time is less than the short stay determination time (step S1912; YES), the process advances to step S1913. In step S1913, the person exclusion unit 414 determines that the person who is determined to be a short-time visitor is to be excluded from counting, and the people counting unit 412 excludes from counting the number of people who are determined to be excluded from counting. . After step S1913, the process returns to step S1904, and it is determined again whether there is a person who has not been analyzed.

The person excluded from counting in step S1913 is stored as a subject to be excluded from counting due to short stay at the timing of exclusion from counting. Further, the number of people excluded from counting in step S1913 is incremented as the number of people excluded from counting due to short stay at the timing of exclusion from counting.

If it is determined in step S1912 that the stay time is not less than the short stay determination time (step S1912; NO), the process advances to step S1914, and the number of people counter is incremented. That is, it is possible to count only those persons whose staying time is longer than the short-term determination time and less than the long-term determination time. Hereinafter, the time that is longer than the short time determination time and less than the long time determination time will also be referred to as "appropriate time", and a person whose staying time is the appropriate time will also be referred to as "appropriate time stayer". After step S1914, the process returns to step S1904, and it is determined again whether there is a person who has not been analyzed.

In the present embodiment, the result screen display control unit 415 displays, for example, a screen shown in FIG. 20 as a viewing screen for viewing the results of the people count.

FIG. 20 is a diagram showing an example of a viewing screen for counting the number of people. As shown in FIG. 20, on the viewing screen, a counted number of people 2001 that has been counted excluding those to be excluded from counting, and a number of people that have been excluded from counting as targets for counting 2002 are displayed.

In addition, among the number of people 2002 who were excluded from counting as a target for counting, a number of people 2003 who were excluded from being counted as long-term stayers and a number of people 2004 who were excluded from being counted as short-term stayers are displayed in a distinguishable manner. .

Furthermore, among the people staying in the stay area 2005 indicated by the dashed-dotted line, long-time stayers, short-time stayers, and proper stayers are each displayed in a distinguishable manner.

For example, long-term stayers are displayed surrounded by a solid-line rectangular frame 2006, short-time stayers are displayed surrounded by a broken-line rectangular frame 2007, and appropriate stayers are displayed without being surrounded by anything. Further, the stay time of each person in the stay area 2005 may be displayed.

As described above, according to this embodiment, long-time stayers and short-time stayers in the stay area are excluded from counting, so only those who stay for an appropriate amount of time in the stay area, who are neither long-time stayers nor short-time stayers, are counted. can do. The appropriate time can be set to a user's desired time by appropriately changing the long stay determination time and short stay determination time. Therefore, it is possible to count the number of people staying for an appropriate amount of time desired by the user.

Further, according to the present embodiment, long-term stayers, short-time stayers, appropriate stayers, and their number are each displayed in an identifiable manner. Therefore, it is possible to easily grasp how long the person staying in the stay area continues to stay in the stay area.

In addition, in this embodiment, an example has been described in which long-time stayers and short-time stayers in the stay area are excluded from counting, but the invention is not limited to this, and either long-time stayers or short-time stayers in the stay area One may be excluded from the count. For example, if only long-stay visitors are to be excluded from the count, the short-time determination time may be set to 0. Furthermore, if only short-time visitors are to be excluded from the count, the long-time determination time may be set to a very large value.

Although the embodiments have been described above, the present invention can also be embodied as a system, device, method, program, storage medium, etc., and specifically, it can be applied to a system composed of a plurality of devices. Alternatively, the present invention may be applied to a device consisting of one device.

Note that the present invention includes a system or device in which a software program that implements the functions of the embodiments described above is supplied directly or remotely. The present invention also includes a case where the present invention is achieved by the computer of the system or device reading and executing the supplied program code.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself installed in the computer also realizes the present invention. That is, the present invention also includes the computer program itself for realizing the functional processing of the present invention.

In that case, as long as it has the function of a program, it may be in the form of an object code, a program executed by an interpreter, script data supplied to the OS, or the like.

Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, MOs, CD-ROMs, CD-Rs, and CD-RWs. There are also magnetic tapes, nonvolatile memory cards, ROMs, DVDs (DVD-ROM, DVD-R), and the like.

Another method of supplying the program is to connect to an Internet home page using a browser on a client computer. It can also be supplied by downloading the computer program of the present invention itself or a compressed file containing an automatic installation function from the homepage to a recording medium such as a hard disk.

It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, the present invention also includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

Additionally, the program of the present invention can be encrypted, stored on a storage medium such as a CD-ROM, and distributed to users, and users who meet certain conditions can download the key information for decrypting it from a homepage via the Internet. let It is also possible to execute the encrypted program by using the downloaded key information and install it on the computer.

Further, the functions of the embodiments described above are realized by the computer executing the read program. In addition, based on instructions from the program, an OS running on the computer performs part or all of the actual processing, and the functions of the embodiments described above can also be realized by this processing.

Furthermore, the program read from the recording medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instructions of the program, the CPU included in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by this processing.

The embodiments described above are merely examples of implementation of the present invention, and the technical scope of the present invention should not be construed as limited by these embodiments. That is, the present invention can be implemented in various forms without departing from its technical idea or main features.

100 Server 101 Client PC
102 Network camera

Claims (7)

acquisition means for acquiring a first image taken of a predetermined area and a second image taken of the area at a different time from the first image;
Based on the person identified from each of the first image and the second image, a person who is shown in the second image and who is also shown in the first image, and An image processing system comprising: an output control unit configured to output information indicating a person who is not shown in the first image to a viewing screen as text information. 2. The output control means controls to output the number of people who are shown in the second image and who are not shown in the first image. Image processing system. 3. The output control means controls to output a name of a person who is shown in the second image and who is not shown in the first image. Image processing system described. 4. The apparatus according to claim 1, wherein the output control means controls to output the number of people who are shown in the second image and who are also shown in the first image. The image processing system according to any one of the items. 5. The apparatus according to claim 1, wherein the output control means controls to output a name of a person who is shown in the second image and who is also shown in the first image. The image processing system according to any one of the items. A method for controlling an image processing system, the method comprising:
an acquisition step in which the acquisition means acquires a first image taken of a predetermined area and a second image taken of the area at a different time from the first image;
The output control means determines, based on the person identified from each of the first image and the second image, the person who is shown in the second image and who is also shown in the first image. A method for controlling an image processing system, comprising: an output control step for controlling to output information indicating a person who is present in the first image and a person who is not included in the first image as text information on a viewing screen . A program executable on an image processing system,
The image processing system,
acquisition means for acquiring a first image taken of a predetermined area and a second image taken of the area at a different time from the first image;
Based on the person identified from each of the first image and the second image, a person who is shown in the second image and who is also shown in the first image, and A program for functioning as an output control means for controlling information indicating a person not shown in the first image to be output as text information to a viewing screen.

Images