WO2022249277A1 - 画像処理装置、画像処理方法、及びプログラム - Google Patents

画像処理装置、画像処理方法、及びプログラム Download PDF

Info

Publication number
WO2022249277A1
WO2022249277A1 PCT/JP2021/019792 JP2021019792W WO2022249277A1 WO 2022249277 A1 WO2022249277 A1 WO 2022249277A1 JP 2021019792 W JP2021019792 W JP 2021019792W WO 2022249277 A1 WO2022249277 A1 WO 2022249277A1
Authority
WO
WIPO (PCT)
Prior art keywords
person
image
images
image processing
target group
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/019792
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
登 吉田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US18/275,769 priority Critical patent/US20240087289A1/en
Priority to JP2023523757A priority patent/JP7556463B2/ja
Priority to PCT/JP2021/019792 priority patent/WO2022249277A1/ja
Publication of WO2022249277A1 publication Critical patent/WO2022249277A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • G06V10/987—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns with the intervention of an operator
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present invention relates to an image processing device, an image processing method, and a program.
  • a device that performs this processing cuts out a person image from an image and classifies the cut out person image for each person. In this classification, the device may include other people's person images in one person's group of person images.
  • Japanese Patent Application Laid-Open No. 2002-200001 describes that an image processing system is provided with correcting means for correcting this error.
  • Patent Document 2 describes that a tracking support device performs the following processing when a moving object to be tracked is tracked by displaying images for each of a plurality of cameras on a display device.
  • the tracking support device sets the person specified by the monitor as a track target.
  • the tracking support device sequentially selects the person with the highest link score for each camera.
  • the tracking support device extracts, for each camera, an image that is most likely to include the person to be tracked as a confirmation image, and displays the timeline screen on which the confirmation image is displayed on the monitor. If an inappropriate confirmation video is found in this timeline image, the observer performs an operation to instruct editing of the tracking result.
  • the tracking support device displays a tracking result edit screen. On this screen, the image of the camera corresponding to the confirmation image is displayed. The supervisor performs an editing operation on this screen so that the video display period of the confirmation video is appropriate.
  • the device When the device performs the process of classifying human images by person, the device may make various types of errors. It is an object of the present invention to make it easier for the user to correct these multiple types of errors.
  • a plurality of person images each including a person, person identification information generated for each of the person images and given to the person included in the person images, and an acquisition means for acquiring time-series information indicating a time-series of person images;
  • Classification means for classifying the person images having the same person identification information into the same group;
  • At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • display control means for determining a display position of the person image belonging to the target group using the time-series information; a correction executing means for executing a correction process according to the information entered in the item input field;
  • An image processing apparatus is provided.
  • a computer is configured to: a plurality of person images each including a person, person identification information generated for each of the plurality of person images and assigned to the person included in the person image, and a time series showing the time series of the plurality of person images an acquisition process for acquiring information; a classification process for classifying the person images having the same person identification information into the same group; At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • a display control process for determining a display position of the person image belonging to the target group using the time-series information; a correction execution process for executing a correction process according to the information entered in the item input field;
  • the computer is configured to: a plurality of person images each including a person, person identification information generated for each of the plurality of person images and assigned to the person included in the person image, and a time series showing the time series of the plurality of person images a retrieval function that retrieves information; a classification function for classifying the person images having the same person identification information into the same group; At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • a display control function for determining the display position of the person image belonging to the target group using the time-series information
  • a correction execution function for executing correction processing according to the information entered in the item input field;
  • a program is provided to have a
  • the user can easily correct multiple errors that may occur when classifying human images by person.
  • FIG. 1 illustrates an example functional configuration of an image processing apparatus according to a first embodiment
  • FIG. 1 shows a first example of information stored in an image storage unit
  • 3 shows a second example of information stored in the image storage unit
  • It is a figure which shows the hardware structural example of an image processing apparatus.
  • 4 is a flowchart showing an example of processing performed by the image processing apparatus
  • 6 is a diagram showing an example of a confirmation screen displayed on the display in step S40 of FIG. 5
  • FIG. 6 is a flowchart showing a first example of processing performed in step S60 of FIG. 5
  • FIG. 8 is a diagram showing an example of a selection screen displayed on the display in step S110 of FIG. 7
  • FIG. 6 is a flowchart showing a second example of processing performed in step S60 of FIG. 5; FIG. It is a figure which shows an example of the screen displayed on a display by step S220.
  • FIG. 6 is a flowchart showing a third example of processing performed in step S60 of FIG. 5;
  • FIG. FIG. 12 is a diagram showing a modification of the selection screen displayed on the display in step S110 of FIG. 7 and/or step S310 of FIG. 11;
  • FIG. 6 is a flowchart showing a fourth example of processing performed in step S60 of FIG. 5;
  • FIG. FIG. 14 is a diagram showing a first example of a screen displayed on the display in step S410 of FIG. 13;
  • FIG. 14 is a diagram showing a second example of a screen displayed on the display in step S410 of FIG. 13;
  • FIG. 10 is a diagram showing an example of a functional configuration of an image processing apparatus according to a second embodiment;
  • FIG. 1 is a diagram showing an example of the functional configuration of an image processing apparatus 10 according to this embodiment.
  • the image processing device 10 processes a plurality of person images.
  • each of the plurality of person images includes a person and has person identification information (for example, person ID) that identifies the person.
  • person identification information for example, person ID
  • This person identification information is assigned to each person by image recognition processing. That is, when the same person is photographed in different person images, the same person identification information is given to these person images.
  • various errors can occur in the correspondence between the person image and the person identification information. Also, if the person image is clipped from another image, this clipping may be erroneous. An example of this error is when an area other than a person is cut out as a person's image, or when an area in which a person exists is not cut out as a person's image.
  • the image processing apparatus 10 is used when the user corrects these errors.
  • the plurality of human images handled by the image processing apparatus 10 may be cut out from, for example, a plurality of frame images forming the same moving image, or may be images generated by a plurality of different cameras (for example, a plurality of surveillance cameras). may be cut out from Also, at least part of the plurality of person images may be the image itself generated by the camera.
  • the image processing apparatus 10 is used as part of a system that tracks a person by image processing.
  • the image processing apparatus 10 includes an acquisition unit 110, a classification unit 120, a display control unit 130, and a correction execution unit 140.
  • the acquisition unit 110 acquires the plurality of person images, person identification information, and time-series information described above.
  • Person identification information and time-series information are assigned to each of a plurality of person images.
  • the time series information indicates the time series of a plurality of person images.
  • the time-series information may indicate the order of the frame images that are the basis of each person image.
  • the time-series information may also be the shooting date and time of the person image or the image from which the person image is based (sometimes even milliseconds are indicated).
  • the acquisition unit 110 acquires the above information from the image storage unit 150.
  • An example of information stored in the image storage unit 150 will be described later using other drawings.
  • the classification unit 120 classifies human images having the same personal identification information into the same group.
  • the display control unit 130 causes the display 160 to simultaneously display at least one person image belonging to a group to be processed (hereinafter referred to as a target group) and an item input field for inputting correction item information.
  • a target group a group to be processed
  • Correction item information indicates correction items to be made to the target group. Examples of modification items are dividing the target group into multiple groups, combining other groups into the target group, and deleting at least one person image from the target group. Further, when a plurality of person images are clipped from a plurality of frame images forming the same moving image, the correction item indicates to clip a new person image to be included in the target group from one of the frame images. good too.
  • the display control unit 130 determines the display position of the person image belonging to the target group using time-series information. For example, when a plurality of person images belong to the target group, the display control unit 130 arranges the plurality of person images in chronological order.
  • the correction execution unit 140 executes correction processing according to the information entered in the item input fields. An example of correction processing will be described later using other drawings.
  • the image processing device 10 further includes the display 160 and the input section 170 described above.
  • the input unit 170 acquires various inputs that the user makes to the image processing apparatus 10 .
  • the display 160 is a touch panel, the display 160 may also serve as the input unit 170 .
  • the display 160 and the input unit 170 may be located outside the image processing apparatus 10 .
  • FIG. 2 shows a first example of information stored in the image storage unit 150.
  • the image storage unit 150 stores an image that is the source of the person image.
  • the image storage unit 150 stores, for each moving image that is the source of a person image, information identifying the moving image (hereinafter referred to as moving image identification information), information identifying the camera that generated the moving image, (hereinafter referred to as camera identification information) and video data (hereinafter also referred to as video).
  • moving image identification information information identifying the moving image
  • camera identification information information identifying the camera that generated the moving image
  • video data hereinafter also referred to as video
  • FIG. 3 shows a second example of information stored in the image storage unit 150.
  • the image storage unit 150 also stores information about human images. Specifically, the image storage unit 150 stores, for each person image, information for identifying the person image (hereinafter referred to as person image identification information), person identification information of a person included in the person image, and image data (hereinafter referred to as person image identification information). , and a portrait image), and information about an image from which the portrait image is cut out (hereinafter referred to as original image information).
  • An example of original image information is moving image identification information and time series information. A frame image from which the person image is based is specified by the moving image identification information and the time-series information.
  • some of the person images stored in FIG. 3 may be images other than persons. This is because an error may occur in the process of cutting out the person image from the original image.
  • the image storage unit 150 may store various scores generated during image processing for each human image. Examples of these scores are a detection score, or probability of being a person, and a tracking score, or probability of person identification.
  • FIG. 4 is a diagram showing a hardware configuration example of the image processing apparatus 10. As shown in FIG.
  • the image processing apparatus 10 has a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input/output interface 1050 and a network interface 1060 .
  • the bus 1010 is a data transmission path for the processor 1020, the memory 1030, the storage device 1040, the input/output interface 1050, and the network interface 1060 to exchange data with each other.
  • the method of connecting processors 1020 and the like to each other is not limited to bus connection.
  • the processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.
  • the memory 1030 is a main memory implemented by RAM (Random Access Memory) or the like.
  • the storage device 1040 is an auxiliary storage device realized by a HDD (Hard Disk Drive), SSD (Solid State Drive), memory card, ROM (Read Only Memory), or the like.
  • the storage device 1040 stores program modules that implement each function of the image processing apparatus 10 (for example, the acquisition unit 110, the classification unit 120, the display control unit 130, and the correction execution unit 140). Each function corresponding to the program module is realized by the processor 1020 reading each program module into the memory 1030 and executing it.
  • the storage device 1040 also functions as the image storage section 150 .
  • the input/output interface 1050 is an interface for connecting the main part of the image processing apparatus 10 and various input/output devices.
  • processor 1020 communicates with display 160 and input 170 via input/output interface 1050 .
  • a network interface 1060 is an interface for connecting the image processing apparatus 10 to a network.
  • This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network).
  • a method for connecting the network interface 1060 to the network may be a wireless connection or a wired connection.
  • FIG. 5 is a flowchart showing an example of processing performed by the image processing device 10.
  • FIG. 5 the image storage unit 150 already stores the information shown in FIGS.
  • the acquisition unit 110 reads a plurality of person images and information incidental thereto (hereinafter referred to as incidental information) from the image storage unit 150 .
  • the incidental information includes person identification information and original image information.
  • the original image degree information includes time series information.
  • the acquiring unit 110 also reads this moving image (for example, moving image indicated by the moving image identification information linked to the person image) from the image storage unit 150 (step S10).
  • the classification unit 120 classifies a plurality of person images into a plurality of groups using the person identification information. Specifically, the classification unit 120 puts together a plurality of person images having the same person identification information into one group. If there is only one person image having the person identification information, only one person image belongs to the group (step S20).
  • the display control unit 130 acquires information specifying the target group, for example, person identification information corresponding to the group to be selected as the target group.
  • the display control section 130 may acquire this information from the user via the input section 170 . Further, the display control unit 130 may recognize all pieces of personal identification information acquired from the acquisition unit 110 and select one piece of personal identification information from the pieces of personal identification information (step S30).
  • the display control unit 130 causes the display 160 to display a confirmation screen.
  • the confirmation screen includes at least one person image belonging to the target group and item input fields for inputting correction item information.
  • a specific example of the confirmation screen will be described later using other drawings (step S40).
  • the user of the image processing apparatus 10 uses the confirmation screen to recognize correction items to be performed on the target group. Then, the user enters correction item information in the item input field of the confirmation screen.
  • the correction execution unit 140 acquires this correction item information (step S50). Then, the correction executing unit 140 recognizes the correction item indicated by the acquired correction item information, and executes processing according to this correction item (step S60). The details of the processing performed here will be described later using other drawings. Also, the information stored in the image storage unit 150 is updated according to the result of the processing performed in step S60.
  • the image processing apparatus 10 repeats the processing shown in steps S30 to S60 until the termination condition is satisfied (step S70).
  • An example of the termination condition is that the user inputs information to the input unit 170 indicating that the modification has been completed, or that all groups are selected, but the conditions are not limited to these.
  • FIG. 6 is a diagram showing an example of a confirmation screen displayed on the display 160 in step S40.
  • the confirmation screen has a person display area 210 , an item input field 220 and a video playback field 230 .
  • the person display area 210 is an area for displaying images of persons belonging to the target group.
  • the person display area 210 may display all person images belonging to the target group, or may display only some person images. In the latter case, the display control unit 130 may select frame images at regular intervals (for example, every 10 frames) in time series, and display only person images corresponding to the selected frame images in the person display area 210. .
  • the display control unit 130 may change the display method between the person image to be watched and the other person image. Items to be changed include, for example, the following. ⁇ Thinning rules (for example, thinning intervals) when displaying only some person images ⁇ Presence or absence of a frame for emphasis ⁇ Presence or absence of at least one of marks, letters, and sentences
  • a person image to be watched is, for example, a person image that is highly likely to be excluded from the target group.
  • An image to be watched is specified, for example, as follows. 1) A person image with a low detection score, that is, a probability of being a person. 2) A person image with a low tracking score, ie, a low probability of person identification information. 3) When a plurality of people are shown in the frame image from which the person image is cut out 4) When the posture of the person changes more than a reference value in the frame images before and after 5) When the degree of clarity of the face is less than the reference value Person image 6) When there is information loss in the original frame image of the person image Specific examples of information loss in 6) are as follows. - Part of the face is hidden.
  • a part of the face is covered by a mask or sunglasses.
  • Part of the attitude information is lost.
  • hiding a part of the body is that a part of the body overlaps possessions, other parts of the person's body (self-hiding), and/or other people.
  • the person display area 210 may display a person image in the person display area 210 so that the user can recognize the existence of the missing frame.
  • An example of this display is to arrange a plurality of person images in the same order as the frame images from which each person image is based, and to provide a space (that is, blank) in the area corresponding to the missing frame.
  • the display control unit 130 may highlight the person image to be watched. Examples of highlighting include framing, coloring, marking, resizing, having a mode that displays only the person image to look at, and displaying the person image to look at to one side (e.g. right or left). ) to display side by side (sort).
  • the item input field 220 displays a plurality of correction items in a selectable manner.
  • the item input field 220 has a plurality of buttons 212 corresponding respectively to a plurality of correction items.
  • the correction items are "separate", “merge”, “delete” and "find”.
  • To "divide” is to divide a target group into a plurality of groups.
  • Cold means to group other groups into a target group.
  • “Erase” is to delete at least one person image from the target group.
  • To “find” is to cut out a new person image to be included in the target group from one of the frame images when a plurality of person images are cut out from a plurality of frame images forming the same moving image.
  • the item input field 220 may display a plurality of correction items in a pull-down format so that they can be selected.
  • the video playback column 230 is a column in which the video read out in step S10 is played.
  • the video playback field 230 is displayed at the same time as the person display area 210 and the item input field 220, the user can easily find an error regarding the target group.
  • the video playback field 230 may have various operation buttons such as a playback start button, a pause button, a fast forward button, and a rewind button.
  • FIG. 7 is a flow chart showing a first example of the processing performed in step S60 of FIG. This figure corresponds to the case where "separate" is selected in FIG.
  • the correction executing unit 140 displays a plurality of person images belonging to the target group on the display 160 in a selectable state (step S110).
  • the image displayed on the display 160 will be referred to as a selection screen.
  • the user of the image processing apparatus 10 selects a person image to be divided into another group while confirming the plurality of person images displayed on the selection screen.
  • the user may select a plurality of person images or may select one person image.
  • the user may specify the boundary between the person images to be left in the target group and the person images to be divided into other groups. Also, when there are a plurality of person images to be divided into other groups, these plurality of person images are often continuous.
  • the user may specify sections of the person images to be divided into other groups (for example, the first person image and the last person image).
  • the correction execution unit 140 gives the same new person identification information to the selected person image.
  • the selected person images are classified into a new group (step S130).
  • step S120 the user may select a person image to be left as the target group.
  • the correction executing unit 140 gives the same new person identification information to the person images that have not been selected in step S130.
  • FIG. 8 is a diagram showing an example of the selection screen displayed on the display 160 in step S110 of FIG.
  • a plurality of person images are cut out from a plurality of frame images forming the same moving image.
  • the plurality of person images are arranged in chronological order.
  • the user specifies the person images to be divided into other groups, and selects the specified person images.
  • FIG. 9 is a flow chart showing a second example of the process performed in step S60 of FIG. This figure corresponds to the case where "summarize" is selected in FIG.
  • the correction execution unit 140 selects at least one candidate group (hereinafter referred to as candidate group).
  • candidate group the correction execution unit 140 calculates the degree of similarity between a person belonging to the target group and a person belonging to another group, and this degree of similarity falls within a reference range (for example, a reference value or more). ) group is selected as a candidate group (step S210).
  • the reference range used here is set based on information obtained from the outside.
  • the user of the image processing apparatus 10 inputs this reference range setting information to the correction executing section 140 via the input section 170 .
  • the user can appropriately set the similarity reference range according to the state of the person image (for example, sharpness and resolution).
  • the correction execution unit 140 causes the display 160 to simultaneously display at least part of the person image belonging to the target group and at least part of the person image belonging to the candidate group (step S220).
  • the user of the image processing apparatus 10 can recognize groups to be grouped into the target group by viewing this screen.
  • the user inputs to the image processing apparatus 10 information specifying groups to be grouped into the target group (hereinafter referred to as group specifying information).
  • group specifying information As an example, the user places a cursor on a person image belonging to a group to be designated, and performs predetermined input to an input device such as a mouse.
  • the correction execution unit 140 can acquire the group designation information (step S230).
  • the correction execution unit 140 selects a group indicated by the group designation information, and puts the selected groups together into a target group.
  • the correction executing unit 140 changes the person identification information associated with the selected group to person designation information associated with the target image (step S240).
  • FIG. 10 is a diagram showing an example of the screen displayed on the display 160 in step S220.
  • the correction executing unit 140 causes the display 160 to simultaneously display at least part of the person images belonging to the target group and at least part of the person images belonging to the candidate group.
  • the correction executing unit 140 determines the display position of the person image belonging to the target group and the display position of the person image belonging to the other group using time-series information. For example, when the person image belonging to the target group and the person image belonging to each candidate group are clipped from the same moving image, the correction executing unit 140 selects the person image clipped from the same frame image among the plurality of person images. are arranged at the same position in a first direction (eg, laterally or longitudinally).
  • the correction execution unit 140 selects a plurality of candidate groups (groups 1 to 3).
  • the correction executing unit 140 determines the arrangement position of the person image belonging to each candidate group by calculating the difference between the position in the frame image of the person image belonging to the candidate group and the position in the frame image of the person image belonging to the target group. determined using When a person belonging to the target group and a person belonging to a certain candidate group are the same, the positions of these persons in the frame image are almost the same or the difference is small. Therefore, the correction execution unit 140 arranges the person images belonging to the candidate group closer to the person images belonging to the target group as the difference becomes smaller. By doing so, the user of the image processing apparatus 10 can easily recognize the group to be selected.
  • the correction execution unit 140 determines the position in the direction intersecting (for example, perpendicular to) the first direction using the difference described above.
  • the correction executing unit 140 may set the display positions of the plurality of candidate groups using the degree of similarity used in step S210. For example, the correction executing unit 140 may display the candidate group closer to the target group as the degree of similarity increases.
  • FIG. 11 is a flow chart showing a third example of the process performed in step S60 of FIG. This figure corresponds to the case where "delete" is selected in FIG.
  • the correction executing unit 140 displays a plurality of person images belonging to the target group on the display 160 in a selectable state (step S310).
  • the screen displayed here is the same as the selection screen shown in FIG.
  • the user of the image processing apparatus 10 selects a person image to be deleted from the target group while confirming the plurality of person images displayed on the selection screen.
  • the user may select a plurality of person images, or may select one person image (step S320).
  • the correction executing section 140 deletes the person identification information from the selected person image. As a result, the selected person image is deleted from the target group (step S330).
  • the correction execution unit 140 may delete the selected person image itself.
  • An example of a person image deleted here is an image other than a person.
  • the user may select a person image to be left as the target group in step S320.
  • the correction executing unit 140 deletes the person identification information from the person images that have not been selected in step S130.
  • FIG. 12 is a diagram showing a modification of the selection screen displayed on the display 160 in step S110 of FIG. 7 and/or step S310 of FIG.
  • the correction executing unit 140 displays the moving image on which the person image is based on the display 160 .
  • the correction executing unit 140 displays the mark indicating the position of the person image and the person identification information corresponding to the person image in an overlapping manner in the moving image.
  • a user selects a person image by selecting at least one of a mark and person identification information.
  • Correction execution unit 140 receives information indicating the selection result as the selection result of the person image.
  • correction execution unit 140 may cause the display column for moving images shown in FIG. 12 to be displayed on the display 160 at the same time as the display column for person images shown in FIG.
  • FIG. 13 is a flowchart showing a fourth example of the processing performed in step S60 of FIG. This figure corresponds to the case where "find" is selected in FIG.
  • the correction execution unit 140 identifies undetected frame images.
  • An undetected frame image is a frame image in which a person image belonging to the target group has not been cut out from among a plurality of frame images forming a moving image. Then, the correction execution unit 140 causes the display 160 to display at least part of the person images belonging to the target group and the undetected frame images (step S410).
  • the correction executing unit 140 cuts out a human image to be newly added to the target group from the undetected frame images (step S420).
  • the correction executing unit 140 may cut out a person candidate area from the undetected frame image using information about the person image (hereinafter referred to as the reference person image) belonging to the target group.
  • the correction executing unit 140 uses the position of the reference person image in the frame image (hereinafter referred to as the specific position) to cut out the person candidate area from the undetected frame image.
  • the correction execution unit 140 identifies a person candidate area included in the undetected frame image, performs image analysis on this person candidate area, and estimates a person score and a similarity score with the target group. Then, when both scores satisfy the reference (for example, if they are equal to or greater than the reference value), the correction execution unit 140 cuts out the human region from the human candidate region.
  • the user may specify an area to be used as a new person image in the undetected frame image.
  • the correction execution unit 140 performs the same processing as the above-described person candidate area for this area.
  • the correction execution unit 140 adds the clipped person image to the target group. For example, the correction executing unit 140 associates the extracted human image with the same object identification information as that of the target group (step S430).
  • FIG. 14 is a diagram showing a first example of the screen displayed on the display 160 in step S410 of FIG.
  • a plurality of person images belonging to the target group are cut out from the same moving image.
  • the correction executing unit 140 arranges the plurality of person images belonging to the target group according to the order of the frame images from which the person images are based.
  • the correction executing section 140 provides a space for an area corresponding to the undetected frame image. Then, display the undetected frame image so as to associate it with that space.
  • FIG. 15 is a diagram showing a second example of the screen displayed on the display 160 in step S410 of FIG.
  • the example shown in this figure is the same as the example shown in FIG. 14 except that moving images are displayed instead of undetected frame images.
  • the correction executing unit 140 displays the mark indicating the position of the person image and the person identification information corresponding to the person image in an overlapping manner in the moving image. This allows the user to easily recognize the human image to be newly cut out, that is, the undetected human image.
  • the user of the image processing apparatus 10 can easily correct multiple errors that may occur when classifying human images by person.
  • FIG. 16 is a diagram showing an example of the functional configuration of the image processing apparatus 10 according to this embodiment.
  • the image processing apparatus 10 shown in this figure is the same as the image processing apparatus 10 according to the first embodiment except that it further has an image clipping unit 180 .
  • the image cropping unit 180 generates a person image by processing the moving image stored in the image storage unit 150 . Then, the image clipping unit 180 causes the image storage unit 150 to store the information shown in FIG.
  • the user of the image processing apparatus 10 can easily correct multiple errors that may occur in the processing result of the image clipping unit 180 .
  • a plurality of person images each including a person, person identification information generated for each of the plurality of person images and assigned to the person included in the person image, and a time series showing the time series of the plurality of person images an acquisition means for acquiring information; Classification means for classifying the person images having the same person identification information into the same group; At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • the display control means for determining a display position of the person image belonging to the target group using the time-series information; a correction executing means for executing a correction process according to the information entered in the item input field;
  • An image processing device comprising: 2.
  • the display control means is an image processing device that differentiates a display method between the person image that satisfies a predetermined condition among the plurality of person images and the other person images. 3.
  • the image processing device wherein the display control means displays a plurality of correction items in the item input field in a selectable manner. 4.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the display control means further displays a moving image reproduction field for reproducing the moving image on the display at the same time as the person image and the item input field. 5.
  • the information entered in the item input field indicates that the target group is to be divided into a plurality of groups,
  • the image processing device, wherein, as the modification process, the modification execution means causes the selected person image or the unselected person image among the person images belonging to the target group to belong to the new group. . 6.
  • the correction executing means performs the correction process as follows: selecting at least one candidate group using the person images belonging to the target group, and displaying at least part of the person images belonging to the candidate group; An image processing apparatus that selects groups to be combined into the target group from the candidate groups according to information input from the outside, and adds the selected groups to the target group. 7. 6. In the image processing device according to 6 above, The correction execution means is selecting, as the candidate group, the group containing the person image similar to the person image belonging to the target group; An image processing device that sets a similarity reference range for selecting the candidate group based on information acquired from the outside. 8. 8.
  • the image processing device wherein the correction executing means determines a display position of the person image belonging to the target group and a display position of the person image belonging to the candidate group using the time-series information.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image, When a plurality of the candidate groups are selected, the correction executing means adjusts the display position of the person image belonging to each of the plurality of candidate groups to the position in the frame image of the person image belonging to the candidate group and the An image processing device that determines using a difference between the positions of the person images belonging to the target group in the frame images. 10.
  • the information entered in the item input field indicates that at least one person image is to be deleted from the target group;
  • the image processing device wherein the correction executing means deletes the selected person image or the unselected person image among the person images belonging to the target group from the target group as the correction process.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image,
  • the correction execution means is displaying the moving image and superimposing a mark indicating the position of the person image and the person identification information corresponding to the person image in the moving image; An image processing device that receives selection of at least one of the mark and the person identification information as selection of the person image. 12.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the information entered in the item input field indicates that the new person image to be included in the target group is extracted from one of the frame images
  • the correction execution means is displaying the frame image in which the person image belonging to the target group is not cut out; An image processing device that cuts out the new person image from the frame image. 13. 13. In the image processing device described in 12 above, The image processing device, wherein the correction executing means cuts out the new person image using information about the person image included in the target group. 14. 13. In the image processing device described in 12 above, The image processing device, wherein the correction executing means cuts out the new person image using an input from a user. 15.
  • the display control means is An image processing device that displays the plurality of person images belonging to the target group in chronological order, and vacates a position corresponding to the frame image from which the person image belonging to the target group is not cut out. 16.
  • the computer a plurality of person images each including a person, person identification information generated for each of the plurality of person images and assigned to the person included in the person image, and a time series showing the time series of the plurality of person images an acquisition process for acquiring information; a classification process for classifying the person images having the same person identification information into the same group; At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • a display control process for determining a display position of the person image belonging to the target group using the time-series information; a correction execution process for executing a correction process according to the information entered in the item input field;
  • An image processing method comprising: 17.
  • the computer in the display control process, uses different display methods for the person image satisfying a predetermined condition among the plurality of person images and other person images. 18.
  • the computer selectably displays a plurality of correction items in the item input field in the display control process. 19.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the image processing method wherein in the display control processing, the computer further displays a moving image reproduction field for reproducing the moving image on the display simultaneously with the person image and the item input field.
  • the information entered in the item input field indicates that the target group is to be divided into a plurality of groups
  • the image processing method wherein, as the correction process, the selected person image or the unselected person image among the person images belonging to the target group belongs to the new group. 21.
  • the information entered in the item input field indicates that the other groups are grouped into the target group
  • the computer as the correction process, selecting at least one candidate group using the person images belonging to the target group, and displaying at least part of the person images belonging to the candidate group;
  • An image processing method comprising selecting groups to be combined into the target group from the candidate groups according to information input from the outside, and adding the selected groups to the target group. 22.
  • the computer in the modification execution process, selecting, as the candidate group, the group containing the person image similar to the person image belonging to the target group; An image processing method, wherein a similarity reference range for selecting the candidate group is set based on externally acquired information.
  • the computer determines a display position of the person image belonging to the target group and a display position of the person image belonging to the candidate group using the time-series information.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image,
  • the computer adjusts the display position of the person image belonging to each of the plurality of candidate groups within the frame image of the person image belonging to the candidate group.
  • An image processing method wherein the determination is made using a difference between a position and a position within the frame image of the person image belonging to the target group. 25.
  • the information entered in the item input field indicates that at least one person image is to be deleted from the target group;
  • the image processing method wherein, as the correction processing, the computer deletes the selected person image or the unselected person image among the person images belonging to the target group from the target group. 26. 26.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the computer in the modification execution process, displaying the moving image and superimposing a mark indicating the position of the person image and the person identification information corresponding to the person image in the moving image;
  • the image processing method wherein selection of at least one of the mark and the person identification information is received as selection of the person image.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the information entered in the item input field indicates that the new person image to be included in the target group is extracted from one of the frame images
  • the computer in the modification execution process, displaying the frame image in which the person image belonging to the target group is not cut out; An image processing method for cutting out the new person image from the frame image.
  • the image processing method wherein the computer cuts out the new person image using an input from a user in the correction execution process.
  • the computer in the display control process, An image processing method comprising displaying the plurality of person images belonging to the target group in chronological order, and leaving a position corresponding to the frame image from which the person image belonging to the target group is not cut out. 31.
  • a plurality of person images each including a person, person identification information generated for each of the plurality of person images and assigned to the person included in the person image, and a time series showing the time series of the plurality of person images a retrieval function that retrieves information; a classification function for classifying the person images having the same person identification information into the same group; At least one person image belonging to a target group, which is the group to be processed, and an item input field for inputting correction item information indicating correction items to be performed on the target group are simultaneously displayed on the display.
  • a display control function for determining the display position of the person image belonging to the target group using the time-series information
  • a correction execution function for executing correction processing according to the information entered in the item input field;
  • a program that has 32.
  • the display control function is a program for differentiating a display method between the person image that satisfies a predetermined condition among the plurality of person images and other person images.
  • the display control function is a program that selectably displays a plurality of correction items in the item input field.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image,
  • the display control function further displays a moving image playback field for reproducing the moving image on the display at the same time as the person image and the item input field. 35.
  • the information entered in the item input field indicates that the target group is to be divided into a plurality of groups, A program in which the correction execution function, as the correction processing, causes the selected or unselected person images among the person images belonging to the target group to belong to the new group.
  • the information entered in the item input field indicates that the other groups are grouped into the target group,
  • the correction execution function includes, as the correction process, selecting at least one candidate group using the person images belonging to the target group, and displaying at least part of the person images belonging to the candidate group;
  • the correction execution function includes: selecting, as the candidate group, the group containing the person image similar to the person image belonging to the target group; A program for setting a similarity reference range for selecting the candidate group based on externally acquired information. 38. In the program according to 36 or 37 above, The program, wherein the correction execution function determines a display position of the person image belonging to the target group and a display position of the person image belonging to the candidate group using the time-series information. 39.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the correction execution function determines the display positions of the person images belonging to each of the plurality of candidate groups to the positions in the frame images of the person images belonging to the candidate groups and the A program for determining by using the difference between the position of the person image belonging to the target group within the frame image. 40.
  • the information entered in the item entry field indicates that at least one person image is to be deleted from the target group; A program in which the correction execution function deletes the selected person image or the unselected person image among the person images belonging to the target group from the target group as the correction process.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image,
  • the correction execution function includes: displaying the moving image and superimposing a mark indicating the position of the person image and the person identification information corresponding to the person image in the moving image; A program for accepting selection of at least one of the mark and the person identification information as selection of the person image. 42.
  • the plurality of person images are cut out from a plurality of frame images that constitute a moving image
  • the information entered in the item input field indicates that the new person image to be included in the target group is to be extracted from one of the frame images
  • the correction execution function includes: displaying the frame image in which the person image belonging to the target group is not cut out; A program for cutting out the new person image from the frame image. 43. In the program according to 42 above, The program according to claim 1, wherein the correction execution function cuts out the new person image using information about the person image included in the target group. 44. In the program according to 42 above, The program, wherein the correction execution function cuts out the new person image using an input from a user. 45.
  • the display control function is A program for displaying the plurality of person images belonging to the target group in chronological order, and leaving a position corresponding to the frame image from which the person image belonging to the target group is not cut out.
  • Image processing device 110 Acquisition unit 120 Classification unit 130 Display control unit 140 Correction execution unit 150 Image storage unit 160 Display 170 Input unit 180 Image clipping unit 210 Person display area 212 Button 220 Item input field 230 Video playback field

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2021/019792 2021-05-25 2021-05-25 画像処理装置、画像処理方法、及びプログラム Ceased WO2022249277A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/275,769 US20240087289A1 (en) 2021-05-25 2021-05-25 Image processing apparatus, image processing method, and non-transitory computer-readable storage medium
JP2023523757A JP7556463B2 (ja) 2021-05-25 2021-05-25 画像処理装置、画像処理方法、及びプログラム
PCT/JP2021/019792 WO2022249277A1 (ja) 2021-05-25 2021-05-25 画像処理装置、画像処理方法、及びプログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/019792 WO2022249277A1 (ja) 2021-05-25 2021-05-25 画像処理装置、画像処理方法、及びプログラム

Publications (1)

Publication Number Publication Date
WO2022249277A1 true WO2022249277A1 (ja) 2022-12-01

Family

ID=84229557

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/019792 Ceased WO2022249277A1 (ja) 2021-05-25 2021-05-25 画像処理装置、画像処理方法、及びプログラム

Country Status (3)

Country Link
US (1) US20240087289A1 (https=)
JP (1) JP7556463B2 (https=)
WO (1) WO2022249277A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010507139A (ja) * 2006-10-11 2010-03-04 ヒューレット−パッカード デベロップメント カンパニー エル．ピー． 顔に基づく画像クラスタリング
JP2014016786A (ja) * 2012-07-09 2014-01-30 Canon Inc 画像処理装置、画像処理方法、及びプログラム
WO2016006090A1 (ja) * 2014-07-10 2016-01-14 株式会社東芝 電子機器、方法及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010507139A (ja) * 2006-10-11 2010-03-04 ヒューレット−パッカード デベロップメント カンパニー エル．ピー． 顔に基づく画像クラスタリング
JP2014016786A (ja) * 2012-07-09 2014-01-30 Canon Inc 画像処理装置、画像処理方法、及びプログラム
WO2016006090A1 (ja) * 2014-07-10 2016-01-14 株式会社東芝 電子機器、方法及びプログラム

Also Published As

Publication number Publication date
JP7556463B2 (ja) 2024-09-26
US20240087289A1 (en) 2024-03-14
JPWO2022249277A1 (https=) 2022-12-01

Similar Documents

Publication Publication Date Title
JP5510167B2 (ja) ビデオ検索システムおよびそのためのコンピュータプログラム
US6339431B1 (en) Information presentation apparatus and method
JP6446971B2 (ja) データ処理装置、データ処理方法、及び、コンピュータ・プログラム
US7978936B1 (en) Indicating a correspondence between an image and an object
US20080079693A1 (en) Apparatus for displaying presentation information
JP3361587B2 (ja) 動画像検索装置及び方法
US11308158B2 (en) Information processing system, method for controlling information processing system, and storage medium
US8626755B2 (en) Electronic album editing system, electronic album editing method, and electronic album editing program
JP2000090121A (ja) メディアブラウザ、メディアファイルブラウジング方法、及びグラフィカルユ―ザインタフェ―ス
US6629107B1 (en) Multimedia information collection control apparatus and method
JPH1084525A (ja) ビデオに索引をつける方法
JP3312105B2 (ja) 動画像インデックス生成方法および生成装置
EP0636994B1 (en) Method of and apparatus for retrieving dynamic images and method of and apparatus for managing images
JP2002344946A (ja) 監視装置
JPWO2006025272A1 (ja) 映像分類装置、映像分類プログラム、映像検索装置、および映像検索プログラム
JP4797761B2 (ja) 動画表示装置
JP6203188B2 (ja) 類似画像検索装置
JP4492124B2 (ja) 書込解析装置、方法、およびプログラム
WO2012157611A1 (ja) 類似画像検索システム
US20040119839A1 (en) Method and apparatus for processing images
US20050262527A1 (en) Information processing apparatus and information processing method
WO2022249277A1 (ja) 画像処理装置、画像処理方法、及びプログラム
DeCamp et al. A human-machine collaborative approach to tracking human movement in multi-camera video
JP6635208B1 (ja) 検索装置、検索方法、およびプログラム
JPH0895986A (ja) 動画像のデータベース装置及びその登録方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21942933

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18275769

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2023523757

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21942933

Country of ref document: EP

Kind code of ref document: A1