JP2006165729A - Photographing program, photographing method and photographing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photographing program, a photographing method, and a photographing system capable of properly utilizing information with respect to a particular object with high convenience. <P>SOLUTION: The photographing method includes: a photographing step of photographing an object; a step of extracting three-dimensional information of the object from image information of the photographed object; a step of extracting at least one object specifying information among attributes and states of the object from the image information; and a step of relating the image information, the three-dimensional information and the object particularizing information with each other and storing them on the basis of the object. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

    The present invention relates to a photographing system that photographs a specific target by a plurality of photographing devices connected to a network, and stores and distributes video information and information obtained by analyzing the video in association with each other.

  2. Description of the Related Art Conventionally, a system has been proposed in which a specific target is observed (captured) from a plurality of image capturing apparatuses in a multifaceted manner, and three-dimensional information about the target is acquired from the obtained video information.

  For example, as shown in FIG. 13 (a), these system configurations can be processed by inputting video information output from a plurality of photographing devices to one computer (processing terminal), as shown in FIG. 13 (b). In this way, a configuration has been proposed in which a photographing apparatus is connected to a network and processed by a computer disposed at a remote location.

  Further, as shown in FIG. 13C, a computer is provided for each photographing apparatus for the purpose of reducing the load on the network, and only the processing result is transmitted via the network, or the photographing apparatus as shown in FIG. A distributed / cooperative system is also proposed in which processing results obtained for each are transmitted and received.

On the other hand, there has been proposed a technique for facilitating video storage management by adding information related to the video, copyright information, and the like to the captured video. Such information added to video is generally called meta information, and environment information at the time of video acquisition, automatic analysis of video, scene information input by the user, cut information, etc. are stored in association with video. The As a result, video search, editing, copyright management, and the like can be automatically performed by a computer, which is expected to contribute to the promotion of safe and convenient video distribution (see Patent Document 1). .
JP 2000-030057 (paragraphs 0022 to 0029, FIG. 1, etc.)

  However, in the conventional three-dimensional information acquisition system as described above, the three-dimensional information of the specific target is acquired from the video, but the accumulation of the spatial information of the target and the temporal statistical information are not acquired. In general, video information and three-dimensional information are managed separately, and the correspondence and association of each information is not considered.

  In addition, the system that adds meta information to video focuses on information about the content and copyright of the video itself, acquires 3D information from multiple videos, accumulates target spatial information and temporal statistical information As with the three-dimensional information acquisition system, the acquisition and addition are not considered.

  Therefore, in the conventional three-dimensional information acquisition system, the relationship between the acquired video and the three-dimensional information is poor, and the information cannot be fully utilized when constructing a system using the actually acquired information. There is a problem of low expansibility and expandability.

  One of exemplary purposes of the present invention is to realize a highly convenient shooting program, shooting method, and shooting system that can appropriately use information on a specific target.

  An imaging program as one aspect of the present invention includes an imaging step of imaging an object, a step of extracting three-dimensional information of the object from image information of the imaged object, an attribute of the object from the image information, and The method includes a step of extracting at least one target specifying information from the state, and a step of storing image information, three-dimensional information, and target specifying information in association with each other based on the target.

  According to the present invention, a specific target is photographed, and the captured video information and the information obtained by analyzing the video information are stored in association with each other, and the three-dimensional information and the target specific information of the specific target are stored in the video information. Since the information is stored in association with each other, it is possible to share information appropriately and easily as compared with the conventional case, and to realize a highly convenient photographing system.

  Examples of the present invention will be described below.

  FIG. 1 is a configuration block diagram of a photographing system in Embodiment 1 of the present invention.

  Reference numeral 101 denotes an imaging terminal (imaging unit), which has an imaging unit 142 and a communication unit 141 that acquire an image of an object (object), and is connected to a network 131. Therefore, communication with other computers and photographing terminals can be performed via the communication unit 141.

  Note that the photographing terminal may be configured to provide one photographing unit (imaging device) like the photographing terminal 101, but two photographing units 144 and 145 like the photographing terminal 102 are provided. It is good also as a terminal for photography provided with a plurality of photography parts.

  Further, in the photographing system of the present embodiment, a plurality of photographing terminals may be connected via a network such as photographing terminals 101, 102, 103, and 104 according to the scale and purpose of the system configuration. In order to acquire three-dimensional information such as a specific target, at least two photographing unit terminals in the entire system may be provided.

  Reference numeral 111 denotes a storage terminal (storage unit) for storing a 3D information extraction unit 151 that acquires 3D information of a specific target from a plurality of videos acquired by the shooting terminals 101 to 104, and video information and the like. Storage section 152, significant information extraction section 153 that statistically processes the three-dimensional information in time, and a communication section 154.

  The user terminal (computer) 121 includes a display unit 161 and a communication unit 162 as display interfaces for various information to the user, and enables the user to access the photographing system.

  The photographing system of this embodiment is controlled by a microcomputer (CPU) provided in each terminal. For example, the whole photographing system is controlled using the CPU of the user terminal 121 and is performed at each terminal. The processing and communication described later are configured to be controlled, and the CPU in each terminal controls processing such as communication, storage processing, update processing, recognition processing, and image processing in each terminal.

  Next, person recognition processing of the photographing terminal and sharing of person information between the terminals in the present embodiment will be described with reference to the conceptual diagram of FIG.

  A storage terminal 602 and photographing terminals 603 and 604 are connected via a network 601. Now, it is assumed that there are shooting targets A605 and B606, and the shooting terminals 603 and 604 capture two shooting targets within the shooting range of the shooting unit provided in each shooting terminal.

  In the photographing terminal 603, an agent A607 and an agent B608 corresponding to the photographing target B are created as agents that are software objects that track the photographing target A and hold information. Similarly, an agent A 609 corresponding to the shooting target A and an agent B 610 corresponding to the shooting target B are created in the shooting terminal 604.

  Each agent stores and updates the information of the corresponding object to be photographed, and the information is shared among the agents, thereby improving tracking accuracy and coordinating tasks.

  In particular, agents in charge of the same shooting object across the shooting terminals such as agent A 607 of shooting terminal 603, agent A 609 of shooting terminal 604, agent B 608 and agent B 610 form a group called an agency, By sharing information related to the imaging target, it is possible to acquire the three-dimensional information of the imaging target in addition to the improvement in tracking accuracy and task sharing.

  The agent keeps extracting information by tracking all persons (specific targets) that are generated within the shooting range of the shooting terminal (shooting unit) until the person in charge is out of the shooting range of the shooting terminal.

  Creation and destruction of agents and agencies are processed in real time, and their state changes dynamically as the person moves or changes state. Information sharing between agents is realized by centrally managing information extracted from the photographing terminals 603 and 604 by the storage terminal 602 and transmitting it to each terminal as necessary.

  Next, FIG. 2 shows an application example of the photographing system of this embodiment to monitoring / marketing in a store, and FIG. 3 shows a sequence of the photographing system.

  FIG. 2 is a schematic diagram when the photographing system of the present embodiment is applied to a general store. The store 201 includes two floors: an entrance floor 202 having an entrance / exit 204 to the outside, and a sales floor 203 on which merchandise is displayed and sold.

  Reference numerals 221 and 222 denote customer characters, and the store 201 cannot enter the sales floor 203 unless it always passes through the entrance / exit 204 and the entrance floor 202. The photographing terminals 211 to 215, the storage terminal 216 including the display unit 217, the user terminal 218, and a POS (Point Of Sales) terminal 219 (not shown) are connected via a high-speed network such as 100BASE-T standard and can communicate with each other. It is in a state.

  Here, the photographing terminals 211 to 215 include a photographing unit such as a CCD camera or a CMOS camera and a video processing unit (not shown), and can perform real-time processing of the video photographed by the video processing unit. Furthermore, the photographing terminal is a camera operation, that is, the overhead photographing units 211 to 213 fixed without performing panning, tilting, and the like, and the noticeable photographing terminal 214 capable of physical camera operations such as panning, tilting, and zooming. 215, and by combining photographing terminals having different functions, it is possible to photograph various persons 221 and 222 entering the store 201.

  The storage terminal 216 is a processing terminal that stores, accumulates, and analyzes the customer personal information as the processing result and the video imaged (acquired) by the imaging terminals 211 to 215, and at the same time from the user terminal 218 and the POS terminal 219. It has a function of distributing significant information (target specifying information) that is spatio-temporal statistical information of a photographing target extracted according to a request in association with stored video information about each person.

  The user terminal 218 is a PDA (Personal Digital Assistant) connected to the network by wireless communication. Here, a store clerk of the store 201 as a user has access to a system such as acquisition of various information and execution of processing. Is possible.

  Such access to the system by the user can also be performed via the storage terminal 216 and the POS terminal 219, and the configuration can be flexibly changed according to the number and arrangement of store clerk. Further, the POS terminal 219 has a function of accumulating accounting and purchase records by scanning a tag such as a barcode attached to a product.

  FIG. 3 is a flowchart of the system sequence. When the imaging system of the present embodiment is activated, the imaging system is first initialized (S301). Here, calibration of the photographing units (cameras) included in the photographing terminals 211 to 215, a communication test between each unit and the computer, and acquisition of terminal information of the photographing terminals 211 to 215 by the storage terminal 216 are performed.

  As a result, the storage terminal 216 can grasp the status of all terminals included in the imaging system, and the individual terminals can inquire the storage terminal 216 as a result. It will be shared.

  If the imaging system is initialized and the status between the terminals arranged can be confirmed, the imaging system enters a standby state (S302). After that, the process on the entrance floor 202 (S303 to S307) and the process on the sales floor 203 (S309 to S314) are respectively performed in parallel.

  In the entrance floor 202, the overhead view of the entrance floor 202 including the entrance / exit 204 is performed using the overhead view photographing terminal 211 in the overhead view of the entrance floor 202 (S303). This overhead view of the entrance floor 202 is continuously executed until a system termination command is issued from the user (S304), and the obtained overhead view video is newly processed by real-time processing by the video processing unit provided in the overhead view photographing terminal 211. A person 221 entering or exiting the entrance floor 202 from the entrance / exit 204 is detected (S305), and more detailed person information is acquired for the person entering the room by further performing attention imaging using the attention imaging terminal 214 (S306). .

  The detailed person information of the person entering the room thus obtained is transmitted to the storage terminal 216 via the network and stored. And it is shared among all terminals in the system as customer personal information.

  Similarly, the person who has exited from the entrance / exit 204 is stored in the storage terminal 216 each time, so that the latest person information existing in the store 201 is constantly updated.

  On the other hand, in the processing on the sales floor 203 that is executed in parallel, in the overhead view shooting on the sales floor 203, the overhead view of the entire sales floor 203 is taken by the overhead view imaging terminals 212 and 213 arranged to take an image of the sales floor 203 without blind spots. Shooting is performed (S309). The overhead view of the sales floor 203 is continuously executed until a system termination command is issued from the user, and the obtained overhead view video is processed in real time by the video processing unit provided in the overhead view terminals 212 and 213 so that the inside of the sales floor 203 is obtained. Is detected (S310), and the behavior is continuously tracked until the person leaves the sales floor 203.

  Furthermore, if a specific event defined in advance in the imaging system is detected from the video imaged in the overhead view (S311), the process proceeds to step 312 and further by performing attention imaging by the attention imaging terminal 215, the details will be described. It is possible to acquire information on a person. In step 313, processing corresponding to the detected specific event is executed.

  The specific events defined in advance in the system include a person's fall, contact accident, theft, etc., and if such an event is detected, the imaging system will allow the clerk to perform a predetermined response to the user terminal. Event display / notification is performed via the display unit 217 and the POS terminal.

  Information on the person who travels on the sales floor 203 and the specific event obtained in this way is integrated with the detailed person information acquired by the entrance floor 202 and stored in the storage terminal 216, and is photographed via the network. In addition to being shared among all terminals in the system, the purchase record acquired by the POS terminal 219 at the time of accounting can be mutually referred to (S314).

  If a system termination command is issued from the user in step 304 and step 310, the imaging system executes the termination process in step 308. The termination processing includes processing such as release of user memory at the time of system startup, arrangement of various extracted information, and editing.

  Next, the processing of each system sequence shown in FIG. 3 will be described in detail.

  First, the system initialization process in step 301 of FIG. 3 will be described with reference to FIG. In addition, since each terminal which comprises a system performs a process in parallel, the process for every terminal is isolate | separated, respectively, and the flowchart is described.

  First, a system activation command is executed by a store clerk at a store via a POS terminal 219, a user terminal 218, or a storage terminal 216, which is a terminal having an interface to the user.

  The POS terminal 219 and the user terminal 218 perform communication tests in steps 402 and 404, respectively, and then proceed to steps 403 and 405 to transmit each terminal information to the storage terminal 216. The terminal information to be transmitted includes an ID unique to the terminal, a communication test result, function defect information, and the like.

  In the overhead view photographing units 211 to 213 and the noticeable photographing units 214 to 215, the camera calibration performed in steps 408 and 412 is performed by using a plurality of photographing terminals, for example, photographing terminals 211 and 214, with reference objects whose three-dimensional coordinates are known. This is a process of calculating camera parameters representing the position, orientation, angle of view, etc. of the photographing unit of each terminal from images taken at the same time. In addition, since the photographing unit included in the target photographing terminal has panning / tilting and zooming functions, the range and speed of the camera operation are obtained by acquiring camera motion / capturable area information in step 413. The camera parameter may be calculated directly using a position / orientation detection sensor or the like in addition to the method using the image.

  If the parameters relating to the photographing unit are obtained in this way, a background image is created for each photographing terminal in steps 409 and 414. This is used when detecting a moving region by the background subtraction method in a process of detecting a person or a specific event from a captured image. In particular, a target photographing terminal capable of camera operation needs to acquire a background image and synthesize a plurality of images within a camera operable range.

  In this embodiment, the background subtraction method is used as a method for detecting a moving region from a captured image. However, the detection algorithm is not limited to this, and a template matching method, an optical flow method, and the like widely used for this type of processing are used. Combination methods of these methods are also applicable.

  In steps 410 and 415, a communication test is performed in the same manner as the POS terminal 219 and the user terminal 218. In steps 411 and 416, terminal information including the calculated parameters relating to the imaging unit is transmitted to the storage terminal 216.

  On the other hand, the storage terminal 216 performs a communication test in the same manner as other terminals in step 406, and receives and stores terminal information transmitted from all other terminals constituting the photographing system in step 407.

  Through the processing for each terminal as described above, the photographing system enters a standby state in step 417. When the photographing system shifts to the standby state in step 417, the POS terminal 219, the user terminal 218, and the storage terminal 216 accept processing input from the user, and the photographing terminals 211 to 215 each start photographing.

  Next, customer personal information acquisition / update processing in the entrance floor 202 from step 302 to step 307 shown in FIG. 3 will be described using the flowchart of FIG.

  When the person 221 enters the shooting range of the overhead shooting terminal 211 that has started the overhead view shooting of the entrance floor 202 from the standby state of the shooting system (S501), it is for shooting at the system initialization in step 301 in FIG. Person recognition is performed using a background difference method of detecting a change area in an image by calculating a difference between the background image acquired for each terminal and the current image (S502), and two-dimensionally related to the person 221 recognized simultaneously. Information is extracted.

  It is determined whether the recognized person is a newly invaded person within the shooting range, or whether the state of the already existing person (photographed person) has changed (S503), and is determined to be a newly invaded person If so, information about the person 221 is stored (held) within the shooting range of the overhead view shooting terminal 211, and a software object called an agent that performs tracking is generated (S504).

  When it is determined that the state of a person that has already existed has been changed, two-dimensional person characteristics obtained from the photographed image, two-dimensional positions on the image, and person information such as the agent in charge are updated ( S505).

  If an agent responsible for recognizing and tracking the person 221 who has entered the shooting range of the overhead view photographing terminal 211 is generated in this way, the overhead view photographing terminal 211 stores the personal information about the recognized person in the storage terminal 216. (S506). The transmitted personal information is received by the storage terminal 216, and the personal information is stored / updated in the storage terminal as well as the overhead view photographing terminal 211 (S507). Then, the storage terminal 216 centrally manages the personal information recognized in the video acquired by all the photographing terminals as well as the overhead photographing terminal 211, and constantly updates the information.

  Further, the storage terminal 216 stores past customer person information, and by storing the person information and the person image in a linked form, it is possible to search for the person image from the person information and vice versa. Accordingly, in step 508, it is determined whether the customer personal information of the person recognized in step 502 is stored in the storage terminal 216 by searching the entire customer personal information based on the acquired personal information.

  If it is determined that the customer person information is already stored, it is determined whether or not the update is necessary from the last update date and time of the customer person information (S509). If it is determined that the update is not necessary It returns to the standby state of step 501. If the customer personal information is not stored in the storage terminal 216 in step 508 or if it is determined that the information stored in step 509 needs to be updated, the storage terminal 216 performs system initialization processing. The information of all terminals existing in the imaging system collected in this way is referred to (S510), and the terminal for attention imaging capable of attention imaging is searched on the entrance floor 202 (S511).

  In this embodiment, the target imaging terminal installed on the entrance floor 202 is one target imaging terminal 214, but the same operation is possible even when a plurality of target imaging terminals are installed. As a result of searching for the target imaging terminal in step 511, when a plurality of target imaging terminals capable of capturing the currently processed person are detected, the storage terminal 216 focuses on the plurality of target imaging terminals. An imaging command signal may be output, and an attention imaging operation command signal may be output for the imaging of interest for which one camera is selected.

  When a noticeable photographing terminal capable of photographing the recognized person is detected, information on the person currently being processed is transmitted, and the CPU of the storage terminal 216 pays attention to the noticeable photographing terminal 214 for noticeable photographing. Requests shooting (command signal output).

  The attention imaging terminal 214 that has received the attention imaging request controls the imaging unit (camera) based on the received person information (S513), captures the attention person in the imaging range, and performs the same process as in step 502 to perform the person processing. It recognizes (S514), compares the obtained two-dimensional features of the person with the person information received in step 513, and confirms the person of interest. Thereafter, it is determined whether the recognized person is newly captured in the shooting range by the camera operation in step 513 or has already existed in the shooting range before the camera operation in step 513 (S515). If it is determined that the object is captured within the imaging range, a software object called an agent that tracks the target person 221 is generated within the imaging range of the target imaging terminal 214 (S517).

  On the other hand, if it is determined that the person has already existed, an agency that is a group of agents having the same purpose as the agent in charge of recognition and tracking of the same person of interest in the overhead view photographing terminal 211 generated in step 504 is selected. Form (S516).

  In step 518, each control of the panning operation, tilting operation, and zoom operation of the noticeable photographing terminal 214 is performed in real time so that detailed photographing is possible according to the action of the noticeable person. For example, it is possible to perform control such that the entire image of the person of interest is positioned at a desired ratio in the center of the shooting range, or control so that the face area of the person is captured in the shooting range using a face recognition algorithm. If the desired image of the person of interest is captured in this way, the noticeable photographing terminal proceeds to step 519 and performs a photographing operation.

  At this time, instead of starting the shooting of the noticeable shooting terminal 214 for the first time in step 519, the shooting may be continued from the time when the person-of-interest information is received in step 513 or when the system is activated. This corresponds to recording time position information in which a detailed image of a person of interest exists in continuously shot images. Thereafter, the photographed person image is subjected to image analysis (S520), and the extracted two-dimensional person information and the two-dimensional person information of the person of interest in the overhead view photographing terminal 211 received in step 513 are referred to. It is possible to calculate the three-dimensional information of the person of interest.

  The obtained detailed image of the attention person and the person information including the three-dimensional information are transmitted to the storage terminal 216 (S521), and the detailed image of the attention person, the features in the two-dimensional image, the agency in charge of tracking, and the affiliation Agent information to be associated is stored and updated as customer personal information (S522). When the storing / updating process of the customer person information in the storage terminal 216 is completed, the process returns to step 501 to be in a standby state until the overhead view photographing terminal 211 recognizes a new person.

  All persons passing through the entrance floor 202 by the system sequence of the imaging system as described above are stored with personal information in the storage terminal 216, and all terminals in the system store personal characteristics and three-dimensional information in the storage terminal 216. By using the inquiry, it is possible to obtain the latest customer person information including image information of the person concerned.

  Next, the overhead view shooting of the sales floor 203 in step 309 of FIG. 3 will be described using the flowchart of FIG.

  First, when a person 222 enters the shooting range of the overhead view shooting terminal 212 that is performing the overhead view shooting of the sales floor 203 from the standby state (S701), the background image acquired for each shooting terminal in the system initialization of step 301 The person is recognized by using the background difference method for detecting the change area in the image by calculating the difference between the current image and the two-dimensional information regarding the recognized person is extracted (S702).

  It is determined whether the recognized person is a newly invaded person within the shooting range or whether the state of the person already existing in the shooting range has changed (S703). If so, a software object called an agent that tracks the person 222 within the shooting range of the overhead view shooting terminal 212 is generated (S704). On the other hand, if it is determined that the state of a person who has already existed has been changed, the two-dimensional person characteristics obtained from the captured image, the two-dimensional position on the image, and the update of person information such as the agent in charge Is performed (S705).

  If an agent responsible for the recognition process and tracking of the person 222 who has entered the shooting range of the overhead view terminal 212 is generated, the overhead view terminal 212 stores personal information about the recognized person in the storage terminal 216 and This is transmitted to another overhead view photographing terminal 213 (S706). The storage terminal 216 that has received the person information stores and updates the person information (S707). Subsequently, all terminal information connected to the storage terminal 216 is referred to (S708), and the same person is in charge using the person information received in step 707 from the photographing terminals operating on the sales floor 203. The agent and the terminal are searched (S709). When an agent tracking the same person is detected, the agent belonging to the overhead view photographing terminal 212 and the newly discovered agent are grouped to form an agency (S710).

  The agent and agency information discovered in this way are transmitted to the overhead view photographing terminals 212 and 13 as person information (S711), and the information is shared. Upon receiving the personal information from the storage terminal 216, the overhead view photographing terminal 212 updates its own personal information (S717), and grasps the agents belonging to other terminals that form an agency with the agent being tracked It becomes possible to do. Also, the image captured by the overhead view terminal 212 is transmitted to the storage terminal 216 (S718).

  On the other hand, in the other overhead view photographing terminal 213 that has received the person information of the overhead view photographing terminal 212 transmitted in step 706, the personal information is updated in the same manner as the storage terminal 216 (S712). An agent that tracks the person received in step 712 is searched from agents corresponding to the person being tracked within the shooting range of the overhead view terminal 213 (S713), and an agent that tracks the same person is detected. In this case, an agency that is a group of agents that track the same person between different terminals is formed (S714). Then, the person information including the newly formed agency information is updated, and the information is transmitted to the storage terminal 216 and the overhead view terminal 212 (S715).

  Further, the image photographed by the overhead view photographing terminal 213 is transmitted to the storage terminal 216 (S716), and image storage processing is performed (S719). A transition is made to a standby state in which a new person is recognized in the overhead view photographing terminal 212 or 213.

  Thereafter, the overhead view photographing terminals 212 and 213 successively detect the persons who have entered the photographing range, generate a responsible agent, and transmit information related to the recognized person to the storage terminal 216 and other overhead view photographing terminals. The other overhead view photographing terminals that have received the person information determine whether or not they are the same person by comparing the received person information and the person information within the shooting range of the person, and the agency of the agent who is responsible for the same person Form.

  Further, in the storage terminal 216, when it is determined that the same person is being tracked between the overhead view photographing terminals grasped by the storage terminal 216, the agents in charge of each correspond to form an agency. Send personal information. By sharing the position and camera parameters of all the overhead view shooting terminals and the person data that falls within the shooting range, person identification is performed between the overhead view shooting terminals, and in the image transmitted to the storage terminal The captured image of the same person from a plurality of viewpoints is specified, and the three-dimensional information of the detected person can be acquired.

  Through the overhead view shooting of the sales floor 203 as described above, a person who goes back and forth on the sales floor 203 is always captured within the shooting range of any one of the overhead view shooting terminals, and by sharing the extracted person information, Acquisition and storage of three-dimensional information is performed.

  Under such circumstances, that is, when an abnormal event defined in advance in the system, such as an accident, dangerous behavior, or theft, occurs in a bird's-eye shooting and attention shooting operation on the entrance floor 202 and the sales floor 203, It is detected immediately by the photographing terminals 212 and 213, and a notification to the store clerk and recording of customer personal information are performed.

  FIG. 8 is a detailed flowchart of event detection / attention shooting on the sales floor in steps 311 to S313 in FIG.

  When a person 222 enters the photographing range of the overhead view photographing terminals 212 and 213 that are performing the overhead view photographing of the sales floor 203 from the system standby state (S801), each photographing terminal is initialized in the S301 system initialization similarly to the overhead view photographing processing. The person is recognized using the background difference method that detects the change area in the image by calculating the difference between the acquired background image and the current image, and the two-dimensional information about the recognized person is extracted at the same time. (S802).

  It is determined whether or not the recognized person 222 is an abnormal person defined in advance in the system (S803). If the recognized person 222 is not an abnormal person, the acquired person information is updated and then the standby state is returned (S804). ).

  On the other hand, if the recognized person is recognized as an abnormal person in step 803, the agent of the target photographing terminal is searched for in the agency to which the agent in charge of tracking the recognized person 222 belongs (S805). In other words, the recognizable person 222 searches for a terminal for attention photographing that is currently within the photographing range.

  When there is no agent belonging to the focused imaging terminal in the agency, that is, there is no focused imaging terminal in which the recognized person 222 is within the imaging range, personal information is transmitted from the overhead imaging terminal to the storage terminal 216. The personal information of the storage terminal 216 is updated (S806). When the personal information is updated in the storage terminal 216, reference is made to all the terminal information connected to the storage terminal 216 (S807), and the target photographing for capturing a person recognized as an abnormal person in step S803 Terminal search is performed (S808).

  In this embodiment, the operation is performed by one attention photographing terminal 215 installed on the sales floor 203, but the same operation is possible even when a plurality of attention photographing terminals are installed, and attention photographing at step 808 is performed. As a result of searching for a terminal, when a plurality of attention photographing terminals capable of photographing an abnormal person are detected, the storage terminal 216 outputs a command signal for attention photographing to the plurality of attention photographing terminals. In addition, a command signal may be output so that one of the plurality of attention imaging terminals is selected and the attention imaging is performed.

  When a noticeable photographing terminal capable of photographing an abnormal person is detected, the person information of the recognized person 222 is transmitted and a noticeable photograph request is made (S809). On the other hand, in step 805, when the recognizing person 222 is captured in the photographing range and the noticeable photographing terminal generating the tracking agent is detected, the overhead view photographing terminal directly contacts the relevant noticeable photographing terminal 215. Send personal information.

  The notable shooting terminal 215 that has received the person information from the overhead view shooting terminal or the storage terminal 216 performs camera control based on the received person information (S810), captures the notable person within the shooting range, and is similar to step 802. The person is recognized by a simple process, and the feature of the obtained two-dimensional person is compared with the received person information to confirm the person of interest (S811). Then, an agent corresponding to the recognized abnormal person is searched (S812), and if it is a person newly captured in the shooting range by the process of Step 811, the corresponding agent is generated (S813), With reference to the agent information received from the overhead view photographing terminal or the storage terminal 216, an agency that is a group of agents in charge of the corresponding abnormal person is generated (S [814).

  Thereafter, panning, tilting, and zooming control of the notable photographing terminal 215 is performed in real time so that detailed photographing can be performed according to the action of the abnormal person to be noticed (S815). For example, it is possible to perform control such that the entire image of the person of interest is positioned at a desired ratio in the center of the shooting range, or control so that the face area of the person is captured in the shooting range using a face recognition algorithm.

  If the desired image of the person of interest is thus captured, the noticeable photographing terminal 215 performs a photographing operation (S816). At this time, instead of starting the imaging of the attention imaging terminal 215 for the first time, the imaging may be continued from the time when the person-of-interest information is received in step 810 or the system activation time. In this case, the imaging in step 816 is continuous. This corresponds to recording time position information in which a detailed image of an abnormal person is present among the images photographed automatically.

  The photographed abnormal person image is subjected to image analysis (S817), and both the extracted two-dimensional person information and the two-dimensional person information of the abnormal person in the overhead view photographing terminal or storage terminal 216 received in step 810 are referred to. By doing so, it becomes possible to calculate the three-dimensional information of the abnormal person. The obtained detailed information of the attention person and the person information including the three-dimensional information are transmitted to the storage terminal 216 (S818), and belong to the agency responsible for tracking the detailed image of the abnormal person, the feature in the two-dimensional image, and the feature. Agent information is associated and stored and updated as customer personal information (S819).

  At the storage terminal 216, customer personal information is stored and updated (S819), and abnormal personal information is specially notified and displayed on the user terminal 218 and the POS terminal 219 to report to the user (S820, S821, S822). As a result, the abnormal person is detected and the detailed information is acquired on the sales floor 203 and the user is notified efficiently, and the user can quickly deal with the abnormal person.

  As described above, in the present embodiment, acquisition / update of customer personal information at the entrance floor 202 (see FIG. 5), overhead view shooting at the sales floor 203 (see FIG. 7), event detection / attention shooting at the sales floor 203 (FIG. 5). 8), the imaging system detects all persons who have entered the store 201, and the customer personal information of the detected person is a detailed image and a feature in the two-dimensional image. The three-dimensional information of the agency in charge of tracking, the agent belonging to the person, and the person is accumulated, and significant information is extracted as a result of statistical processing of the accumulated information temporally and spatially, and stored in association with each other.

  Table 1 shows two-dimensional information, three-dimensional information, and significant information extracted and stored in the present embodiment.

  As shown by the monitoring operation in the store 201 from FIG. 3 to FIG. 8, the shooting system of the present embodiment relates to the person who invades and stays in the store 201, and the bird's-eye view image and the terminal for attention shooting with respect to the person who stays in the store 201 The attention image by is acquired. Then, two-dimensional information about a person can be obtained by processing the image information independently at the acquired individual terminal.

  As the two-dimensional information of a person, a two-dimensional position in the image, a two-dimensional shape, a two-dimensional count, a two-dimensional density, a two-dimensional movement speed, a two-dimensional movement locus, a two-dimensional stay time, and color information can be obtained. As a method for obtaining such information, it is possible to select an image processing algorithm such as an edge extraction method, a pattern matching method, an optical flow method, or the like in consideration of the calculation accuracy and load balance suitable for the photographing system.

  Next, by sharing the obtained two-dimensional information among other photographing terminals connected via the storage terminal, it becomes possible to observe the target person from various perspectives. Obtainable.

  As the three-dimensional information of a person, a three-dimensional position in a defined space, a three-dimensional shape, a three-dimensional count, a three-dimensional density, a three-dimensional movement speed, a three-dimensional movement trajectory, a three-dimensional stay time, and a three-dimensional movement are obtained. be able to. Since these three-dimensional information includes spatial (stereoscopic) information as compared with two-dimensional information, it is highly useful as highly accurate and reliable information. Furthermore, it is possible to accumulate two-dimensional and three-dimensional information about the obtained person over time, and to define and extract statistical information specific to the photographing system as spatiotemporal information as significant information.

  As significant information in this embodiment, customer person information history, purchase behavior history, store layout information, clerk placement information, security countermeasure information, POS system automatic input information, and the like can be extracted.

  FIG. 9 shows customer personal information stored in the storage terminal 216 in a hierarchical manner. Person information 901 and 902 is accumulated in accordance with the customer person, and the content is the frequency of use from the history of the whole body image 921 and the face image 922 photographed by the attention photographing terminal as the detailed image 911 and the three-dimensional movement locus 912. Behavior pattern 923 such as high travel route and stop-by order, three-dimensional movement speed 913, average speed 924 and maximum speed 925 to date, height 926, three-dimensional shape 914, height 926, physique 927, frequently performed actions and characteristic The store visit frequency 929, the store visit pattern 930, and the average stay time 931 are obtained and stored by taking temporal statistics of higher-order information as statistics of the behavior 928 and stay time / date and time 915, respectively.

  Further, the customer person information 901 and 902 can be cross-referenced with the POS information 903. For example, the customer history information 916 can be inquired from a specific person characteristic, or the customer person information can be automatically entered by referring to the customer person information when the POS information 903 is inputted. It is possible.

  Furthermore, these customer person information 901 and 902 are arranged hierarchically as store information 1001 as shown in FIG. 10, and for example, according to a request from the user, for example, the three-dimensional movement trajectory 1011 of all the customer persons, the three-dimensional density 1012 From the information such as the three-dimensional moving speed 1013 and the three-dimensional shape 1014, the congested section 1021, the human logistics stagnation section 1022, the abnormal behavior frequent occurrence section 1023, and the customer attraction section 1024 can be extracted together with video information as significant information in the store 201. Thus, the information can be used as reference information for implementing the store layout 1031, the store clerk arrangement 1032, and the crime prevention measure 1033 in advance.

  Specifically, when the customer area 1024 with many people passing through the store is determined by the statistical processing of the three-dimensional movement trajectory 1011, the store layout is changed to display the products to be noticed. The position where the three-dimensional movement speed is low although it is small and the three-dimensional density is low, the passage design is reviewed as the human logistics stagnation section 1022, and the abnormal action frequent occurrence section 1023 which is determined to be abnormal behavior frequent occurrence from the three-dimensional shape of the person. In addition to changing the store layout, it is possible to closely arrange the store clerk and to simulate crime prevention countermeasures with reference to abnormal action images that have occurred in the past in advance.

  FIG. 11 is an operation flowchart of the photographing system according to the second embodiment of the present invention. Since the outline of the photographing system, the person recognition processing of the photographing terminal, the sharing of personal information between the terminals, and the application of the photographing system to a general store are the same as those in the first embodiment, description thereof will be omitted.

  In FIG. 11, the system initialization and standby state (S1011 and S1012), steps 1103 to 1108 in the entrance floor 202 executed in parallel and steps 309 to 314 in the sales floor 203 are the same as those in the first embodiment. This is performed in the same manner as in FIG.

  In the first embodiment, in addition to the processing operations on the entrance floor 202 and the sales floor 203, if a person and a specific event are detected by the overhead view photographing terminal, a command signal or the like is given to the noticeable photographing terminal to perform noticeable photographing. After outputting the image and requesting attention imaging, the attention imaging terminal performs person detection, imaging, image analysis, and 3D information acquisition, and then transmits all information to the storage terminal as customer personal information.

  On the other hand, in this embodiment, the terminal for attention photographing that has received the instruction signal for attention photographing performs detection, photographing, and image analysis of a person, but the three-dimensional information acquisition and significant information acquisition processing is a request from the user, that is, It is executed in response to a user operation.

  In steps 1115 to 1117, the process is a three-dimensional information request process from the user. When a request for three-dimensional information is made from the user in step 1115, the storage terminal analyzes the stored video and requests the requested person. The three-dimensional information is calculated by referring to the image analysis results of a plurality of viewpoints taken at the same time and the terminal information that has been taken and stored together (S1116). The calculated three-dimensional information is transmitted to the requesting user (terminal) together with the corresponding video (S1117).

  Similarly, steps 1118 to 1121 are significant information request processing. When a request for significant information is made from the user in step 1118, the storage terminal analyzes the stored video and responds to the requested condition. The three-dimensional information is calculated by referring to the image analysis results of a plurality of viewpoints obtained by photographing the person at the same time and the terminal information of the photographed image stored together (S1119). Then, significant information is calculated together with the corresponding video by calculating temporal statistics of the calculated three-dimensional information (S1120). The calculated three-dimensional information and significant information are transmitted to the requested user (terminal) (S1121).

  As described above, in this embodiment, the calculation of three-dimensional information and significant information with a large processing load is not performed in real time at the same time as video acquisition, but is performed by offline processing at the time of user request or after the end of shooting. It is possible to reduce the computational load, storage capacity, and communication load of the imaging system required for.

  FIG. 12 is a schematic configuration diagram of an imaging system in Embodiment 3 of the present invention. In the first and second embodiments, as shown in FIG. 1, the photographing system is composed of a plurality of photographing terminals 101 to 104 connected to a network, a storage terminal 111 and a user terminal 121. In other words, various types of information including video acquired by the photographing terminals 101 to 104 are once collected in the storage terminal 111, and the server / client system is configured to centrally manage the information by the storage terminal 111.

  On the other hand, in this embodiment, a peer-to-peer system configuration is adopted as shown in FIG. Specifically, the photographing storage terminals 1201 to 1203 exchange information (communication) with each other without using a server, and the photographing storage terminal 1201 includes a photographing unit 1225 for acquiring a target video, and a plurality of videos. A three-dimensional information extraction unit 1224 for extracting more target three-dimensional information, a storage unit 1223 for storing video information and the like, a significant information extraction unit 1222 for statistically processing the three-dimensional information temporally, and a communication unit. Information can be exchanged with other terminals via the network 1201.

  The user terminal 1211 includes a display unit 1231 and a communication unit 1232 that are interfaces of various types of information to the user, and enables the user to access the photographing system.

  As described above, in this embodiment, the photographing storage terminals 1201 to 1204 communicate information acquired by communicating with each other, and information is shared to extract three-dimensional information and significant information. In other words, since each shooting and storage terminal has the same function in a peer-to-peer system configuration, the system construction flexibility compared to the server-client type and the entire system when a malfunction occurs in the terminal Improves stability.

1 is a schematic configuration diagram of an imaging system in Embodiment 1 of the present invention. 1 is an abstract diagram in which a photographing system according to Embodiment 1 of the present invention is applied to a store. FIG. 3 is an operation flowchart of the photographing system according to the first embodiment of the present invention. 1 is a flowchart relating to system initialization of an imaging system in Embodiment 1 of the present invention. FIG. 3 is a flowchart of customer personal information acquisition / update processing (entrance floor) of the photographing system according to the first embodiment of the present invention. 1 is an information sharing conceptual diagram of an imaging system in Embodiment 1 of the present invention. The flowchart figure (sales floor) which concerns on the overhead view imaging | photography of the imaging | photography system in Example 1 of this invention. The flowchart figure (sales floor) which concerns on the event detection and attention imaging | photography in Example 1 of this invention. The hierarchy figure which shows the customer person information in Example 1 of this invention. The hierarchy figure which showed the customer person information in Example 1 of this invention as shop information. FIG. 7 is an operation flowchart of the imaging system according to the second embodiment of the present invention. FIG. 6 is a schematic configuration diagram of an imaging system in Embodiment 3 of the present invention. 1 is a schematic configuration diagram of a conventional photographing system.

Explanation of symbols

101 Shooting terminal (shooting unit)
111 Storage terminal (storage unit)
121 User terminal 131 Network 142 Imaging unit 141 Communication unit 151 Three-dimensional information extraction unit 152 Storage unit 153 Significant information extraction unit

Claims (11)

A shooting step for shooting the object;
Extracting three-dimensional information of the object from the image information of the imaged object;
Extracting at least one target specifying information from the image information among the attributes and states of the target;
And a step of storing the image information, the three-dimensional information, and the target specifying information in association with each other based on the target.   The imaging program according to claim 1, further comprising a step of storing the color information, the two-dimensional information, the target specifying information, and the image information of the target object in association with each other.   The imaging program according to claim 1, wherein the target specifying information is temporal and spatial statistical information in an imaging situation.   The three-dimensional information includes a three-dimensional position, a three-dimensional shape, a three-dimensional count, a three-dimensional density, a three-dimensional velocity, a three-dimensional trajectory, and a three-dimensional motion of the object. The shooting program according to any one of the above.   The photographing program according to any one of claims 1 to 4, wherein photographing of the object is performed by a plurality of photographing units. Photographing means for photographing the object;
3D information extraction means for extracting 3D information of the object from image information of the imaged object;
Target specifying information extracting means for extracting at least one target specifying information from the image information among the attribute and state of the target;
Storage means for storing the image information, the three-dimensional information, and the target specifying information;
An imaging system comprising: control means for associating and storing the image information, the three-dimensional information, and the target specifying information in the storage unit based on the object.   The imaging system according to claim 6, wherein the control unit stores the color information, the two-dimensional information, the target specifying information, and the image information of the target object in association with each other.   The imaging system according to claim 6 or 7, wherein the target specifying information is temporal and spatial statistical information in an imaging situation. A computer comprising: display means for displaying the image information, the three-dimensional information, and the target specifying information; and communication means for communicating with each of the means,
The imaging system according to claim 6, wherein the control unit extracts the target specifying information based on an output signal from the computer.   The imaging system according to claim 9, wherein the control unit extracts the target specifying information at a predetermined timing.   The imaging system according to claim 6, wherein imaging of the object is performed by a plurality of imaging units.

Images