Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

• the disclosed technology relates to an information processing method, an information processing device, and an information processing program.
• a target object has been detected from multi-view images obtained by photographing the target object from a plurality of different viewpoints.
• an image monitoring device has been proposed that captures images of people in different background images by capturing images from different directions using a ceiling camera and a wall camera.
• This device projects the pixels of the changing area in the wall camera image onto the ceiling camera image to obtain an epipolar line, extracts the area of the epipolar line that has the same characteristics as the pixels of the changing area, and based on the existing area of the area.
• the projection area is generated using Further, this device combines the projected area and the changed area in the ceiling camera image to obtain a combined changed area, and detects a person in the ceiling camera image based on the combined changed area.
• machine learning models such as neural networks are used to detect objects from images.
• a large amount of data with correct labels indicating the position information of an object in an image is required.
• preparing a large amount of data with correct answer labels requires a huge amount of work cost. Therefore, the position information of the object detected by the machine learning model is used as a pseudo label, and in addition to the data with the correct answer label prepared in advance, the data with the pseudo label is also used to perform machine learning of the machine learning model.
• Supervised learning methods have also been proposed.
• the disclosed technology aims to accurately calculate position information of an object in an image.
• the disclosed technology acquires a plurality of images taken by each of a plurality of cameras that take images of a target object from a plurality of different viewpoints. Further, the disclosed technology calculates three-dimensional position information of the object based on two-dimensional position information of the object detected from each of the plurality of images and camera parameters of each of the plurality of cameras. presume. The disclosed technology projects three-dimensional position information of the object onto the at least one image based on camera parameters of a camera that has taken at least one of the plurality of images, and projects the three-dimensional position information of the object onto the at least one image. Calculate two-dimensional position information of the object at .
• One aspect is that the positional information of the object in the image can be calculated with high accuracy.
• FIG. 2 is a schematic diagram showing a connection between an information processing device and a camera according to the present embodiment.
• FIG. 3 is a diagram for explaining machine learning of a detector that detects 2D-BBOX and detection of 2D-BBOX.
• FIG. 3 is a diagram for explaining machine learning of a detector using semi-supervised learning.
• 1 is a functional block diagram of an information processing device according to an embodiment.
• FIG. FIG. 2 is a diagram for explaining a 2D-BBOX.
• FIG. 3 is a diagram for explaining two-dimensional posture information of a target object.
• FIG. 3 is a diagram for explaining projection of three-dimensional posture information of a target object onto an image and calculation of two-dimensional posture information of the target object.
• FIG. 3 is a diagram for explaining machine learning of a detector that detects 2D-BBOX and detection of 2D-BBOX.
• FIG. 3 is a diagram for explaining machine learning of a detector using semi-supervised learning.
• 1 is a functional
• FIG. 7 is a diagram for explaining the effect of calculating two-dimensional posture information by projecting three-dimensional posture information.
• FIG. 6 is a diagram for explaining selection of pseudo labels based on spatial restrictions.
• FIG. 6 is a diagram for explaining selection of pseudo labels based on time restrictions.
• FIG. 3 is a diagram for explaining selection based on evaluation of pseudo labels.
• 1 is a diagram showing a schematic configuration of a computer functioning as an information processing device according to the present embodiment. 3 is a flowchart illustrating an example of information processing according to the present embodiment.
• FIG. 3 is a diagram illustrating an example of a pseudo label generation result by the information processing device according to the present embodiment.
• FIG. 2 is a diagram for explaining application of the information processing device according to the present embodiment to a scoring system for gymnastics competitions.
• the information processing apparatus 10 includes a plurality of cameras 30n each of which photographs an object (in the example of FIG. 1, the object is a person) 90 at a viewpoint n from a different direction. connected to.
• n 0, 1, and 2
• a camera 300 that takes pictures from viewpoint 0 a camera 301 that takes pictures from viewpoint 1
• a camera 302 that takes pictures from viewpoint 2 are connected to the information processing device 10.
• the number of cameras 30n connected to the information processing device 10 is not limited to the example in FIG. 1, and may be two or four or more.
• the camera 30n is installed at an angle and position where the object 90 falls within the photographing range. Images captured by the camera 30n are sequentially input to the information processing device 10. Note that a synchronization signal is sent to each camera 30n, and the images taken by each camera 30n are synchronized.
• the information processing device 10 generates a refined object based on two-dimensional position information of the object 90 detected from each of a plurality of images taken from a plurality of different viewpoints (hereinafter referred to as "multi-view images"). Two-dimensional position information of the object 90 is calculated.
• a detector that is a machine learning model such as a neural network is used.
• this detector is generated by machine learning using a large amount of images with correct labels indicating two-dimensional position information of the target object 90.
• the coordinates [x 1 , y 1 ] and The coordinates [x 2 , y 2 ] of the lower right point are used as the correct label.
• the position of the object 90 can be determined from that image. The 2D-BBOX shown is detected.
• Position bias refers to a deviation in the position of the area indicated by the pseudo label from the actual area of the object 90 on the image, such as a shift in the position, or a deviation in the size of the area, such as being larger or smaller. .
• the object 90 can take various postures, such as a gymnast, it is difficult to generate balanced pseudo-labels for all postures.
• the two-dimensional position information of the target object 90 in the image is calculated with high accuracy so that false positive and false negative false labels can be reduced. Furthermore, in this embodiment, a pseudo label is generated in which the bias of the position of the object 90 with respect to the actual area is corrected. Furthermore, in this embodiment, a well-balanced pseudo label corresponding to the diversity of postures of the object 90 is generated.
• the information processing device 10 according to this embodiment will be described in detail below.
• the information processing device 10 functionally includes an acquisition section 11, an estimation section 12, a generation section 13, a selection section 14, and a machine learning section 15. Further, a detector 22 and a camera parameter database (DB) 24 are stored in a predetermined storage area of the information processing device 10.
• the detector 22 is a machine learning model for detecting a 2D-BBOX indicating the area of the target object 90 from an image, which is generated by machine learning using images with correct answers as training data.
• the camera parameter DB 24 stores internal parameters and external parameters of each camera 30n.
• the generation unit 13 is an example of a “calculation unit” of the disclosed technology.
• the acquisition unit 11 acquires time-series multi-view images captured by a plurality of cameras 30n.
• the estimation unit 12 estimates the three-dimensional shape of the object 90 based on the 2D-BBOX indicating the area of the object 90 detected from each image included in the multi-view image and the camera parameters of each camera that captured each image. Estimate location information.
• the estimation unit 12 uses the detector 22 to detect a 2D-BBOX 42n indicating the area of the target object 90 from the image 40n taken by the camera 30n. Then, the estimation unit 12 uses a recognition model (not shown) generated in advance by machine learning to recognize one or more parts of the person, which is the object 90, from the detected 2D-BBOX 42n. Estimate the two-dimensional position information of each part. For example, as shown in FIG. 6, when the recognition model recognizes the position of each joint, etc. of a person (object 90) (black circles in FIG. 6), the estimation unit 12 calculates the position of each joint, etc. The coordinate values are estimated as two-dimensional position information of the object 90.
• a group of two-dimensional position information of each part such as a joint of the object 90 will be referred to as two-dimensional posture information.
• the estimation unit 12 uses the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional posture information of the object 90 to determine the three-dimensional shape of each part of the object 90 by triangulation.
• Estimate location information a group of three-dimensional position information of each part of the object 90, such as joints, will be referred to as three-dimensional posture information.
• the recognition model recognizes n parts such as joints for one person, which is the object 90
• the three-dimensional posture information is expressed as ⁇ [P X 1 , P Y 1 , P Z 1 ], [P X 2 , P Y 2 , P Z 2 ], ..., [P X n , P Y n , P Z n ] ⁇ .
• the generation unit 13 projects three-dimensional posture information of the object 90 onto the image 40n based on the camera parameters of the camera 30n that captured the image 40n included in the multi-view image, Two-dimensional posture information of the refined object 90 in the image 40n is calculated. Specifically, three-dimensional posture information ⁇ [P X 1 , PY 1 , P Z 1 ], [P X 2 , PY 2 , P Z 2 ], ..., [ P n , P Z n ] ⁇ , the two-dimensional posture information corresponding to ⁇ [p x 1 , p y 1 ], [p x 2 , p y 2 ], ..., [p x n , p y n ] ⁇ . In this case, the generation unit 13 calculates two-dimensional posture information using equation (1) below. Note that in equation (1), H is a three-dimensional to two-dimensional projection matrix determined from the camera parameters of the camera 30n.
• the generation unit 13 generates a pseudo label 44n indicating the area of the target object 90 based on the calculated two-dimensional posture information. Specifically, as shown in equation (2) below, the generation unit 13 uses the maximum and minimum values of the two-dimensional coordinates of each point included in the two-dimensional posture information to generate the upper left corner of the pseudo label 44n. The coordinates of the point [x 1 , y 1 ] and the coordinates of the lower right point [x 2 , y 2 ] are calculated.
• w and h in equation (2) are the width and height of the circumscribed rectangle of the object 90 indicated by the calculated two-dimensional posture information.
• An area obtained by adding a predetermined margin to a w ⁇ h circumscribed rectangle is calculated as the range of the pseudo label 44n.
• the margin is not limited to the value obtained by multiplying the width w or the height h by the constant ⁇ .
• the range of the pseudo label 44n may be a range added in the vertical and horizontal directions of the w ⁇ h range, with a predetermined pixel (for example, 5 pixels) as a margin.
• the multi-view image includes images 400, 401, and 402, and the estimation unit 12 detects 2D-BBOX 420 and 421 from images 400 and 401, and from image 402, 2D-BBOX 420 and 421 are detected from image 402. Assume that BBOX 422 is not detected. Even in this case, the generation unit 13 can generate the pseudo label 442 from the image 402 by reprojecting the three-dimensional posture information onto the image 402. That is, false negative false labels can be reduced.
• the generation unit 13 corrects the positional bias occurring in the 2D-BBOXs 420 and 421 by reprojecting the three-dimensional posture information on the images 400 and 401 and generating pseudo labels 440 and 441. can do.
• the selection unit 14 selects a pseudo label to be used for machine learning of the detector 22 from the pseudo labels 44n generated by the generation unit 13 based on spatial and temporal restrictions.
• the selection unit 14 selects in advance the position of the object 90 in the three-dimensional space (hereinafter referred to as "three-dimensional position") indicated by the three-dimensional posture information that is the projection source when generating the pseudo label 44n. If it is included in the predetermined range, that pseudo label 44n is selected.
• the predetermined range may be a competition area depending on the competition event. More specifically, in the case of an event that uses equipment, a predetermined range including the equipment may be defined as the competition area, and if the event is on the floor, a predetermined range including the prescribed performance range may be defined as the competition area.
• the selection unit 14 selects the pseudo labels 440A and 441A as the pseudo labels 44n used for machine learning.
• the three-dimensional position 46B from which the pseudo label 440B generated from the image 400 and the pseudo label 441B generated from the image 401 are projected is outside the competition area.
• the selection unit 14 excludes the pseudo labels 440B and 441B from the pseudo labels 44n used for machine learning.
• the selection unit 14 selects the pseudo label 44n as the pseudo label 44n to be used for machine learning.
• the predetermined time range may be a time range corresponding to the time from the start to the end of the performance.
• the selection unit 14 selects a start frame corresponding to the start of the performance and an end frame corresponding to the end of the performance from each frame of a series of time-series multi-view images. Identify.
• the selection unit 14 specifies, as the start frame, a frame that is a predetermined frame before the moment when the athlete enters the competition area and his or her feet first leave the floor.
• the selection unit 14 specifies a frame that is a predetermined frame before the player leaves the competition area as the end frame.
• the selection unit 14 selects a pseudo label 44n generated from a frame (image 40n) included in the target time, with the target time being from the start frame to the end frame.
• the selection unit 14 excludes pseudo labels 44n generated from non-target frames outside the target time. As a result, it is possible to exclude pseudo labels 44n based on the posture of the athlete who is simply standing before the start of a performance, and it is possible to select a well-balanced pseudo label 44n that corresponds to the variety of postures of the athlete. can.
• the selection unit 14 evaluates the quality of the generated pseudo label 44n, and selects it as the pseudo label 44n to be used for machine learning of the detector 22 if the evaluation result satisfies the criteria. Specifically, the selection unit 14 determines the degree of overlap between the 2D-BBOX 42n detected by the estimation unit 12 using the detector 22 and the pseudo label 44n generated by the generation unit 13 based on the 2D-BBOX 42n. calculate.
• the degree of overlap may be, for example, the area of the overlapped portion/the area of the pseudo label 44n. As shown in FIG. 11, the selection unit 14 selects pseudo labels 44n whose degree of overlap is greater than or equal to a predetermined threshold, and excludes pseudo labels 44n whose degree of overlap is less than the threshold.
• the selection unit 14 presents the pseudo labels 44n whose degree of overlap is less than the threshold value to the user, accepts the user's decision to accept or reject the pseudo labels 44n, and uses the pseudo labels 44n adopted by the user as pseudo labels 44n for use in machine learning of the detector 22. It may be selected as the label 44n.
• the user is made to make a decision only about the pseudo labels 44n that do not meet the criteria, so the burden on the user can be reduced.
• the machine learning unit 15 executes machine learning of the detector 22 using the pseudo-labeled image obtained by adding the pseudo label 44n selected by the selection unit 14 to the image 40n and the correct-answered image as training data.
• the machine learning unit 15 causes the acquisition unit 11, the estimation unit 12, the generation unit 13, and the selection unit 14 to repeatedly execute the processing, and repeatedly executes machine learning of the detector 22 using the obtained pseudo label 44n.
• the number of images with pseudo labels increases, so the detection accuracy of the 2D-BBOX 42n by the detector 22 improves, and the generation accuracy of the pseudo labels 44n also improves.
• the selection unit 14 uses only the pseudo labels 44n whose quality evaluation results meet the standards, thereby further improving the detection accuracy of the 2D-BBOX 42n by the detector 22.
• the information processing device 10 may be realized, for example, by a computer 50 shown in FIG. 12.
• the computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a nonvolatile storage device 53.
• the computer 50 also includes an input/output device 54 such as an input device and a display device, and an R/W (Read/Write) device 55 that controls reading and writing of data to and from a storage medium 59.
• the computer 50 also includes a communication I/F (Interface) 56 connected to a network such as the Internet.
• the CPU 51, memory 52, storage device 53, input/output device 54, R/W device 55, and communication I/F 56 are connected to each other via a bus 57.
• the storage device 53 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like.
• An information processing program 60 for causing the computer 50 to function as the information processing device 10 is stored in the storage device 53 as a storage medium.
• the information processing program 60 includes an acquisition process control instruction 61 , an estimation process control instruction 62 , a generation process control instruction 63 , a selection process control instruction 64 , and a machine learning process control instruction 65 .
• the storage device 53 has an information storage area 70 in which information constituting the detector 22 and camera parameter DB 24 is stored.
• the CPU 51 reads the information processing program 60 from the storage device 53, expands it onto the memory 52, and sequentially executes control commands included in the information processing program 60.
• the CPU 51 operates as the acquisition unit 11 shown in FIG. 4 by executing the acquisition process control instruction 61. Further, the CPU 51 operates as the estimation unit 12 shown in FIG. 4 by executing the estimation process control instruction 62. Further, the CPU 51 operates as the generation unit 13 shown in FIG. 4 by executing the generation process control instruction 63. Further, the CPU 51 operates as the selection unit 14 shown in FIG. 4 by executing the selection process control instruction 64. Further, the CPU 51 operates as the machine learning section 15 shown in FIG. 4 by executing the machine learning process control instruction 65.
• the CPU 51 reads information from the information storage area 70 and develops the detector 22 and camera parameter DB 24 in the memory 52. Thereby, the computer 50 that has executed the information processing program 60 functions as the information processing device 10. Note that the CPU 51 that executes the program is hardware.
• the functions realized by the information processing program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.
• the information processing device 10 executes the information processing shown in FIG. 13.
• the information processing is an example of an information processing method of the disclosed technology.
• step S11 the acquisition unit 11 acquires a plurality of time-series multi-view images.
• step S12 the estimation unit 12 uses the detector 22 to detect the 2D-BBOX 42n indicating the area of the target object 90 from each image 40n included in the multi-view image. Then, the estimation unit 12 estimates two-dimensional posture information of the object 90 from the detected 2D-BBOX 42n using the recognition model.
• step S13 the estimating unit 12 uses the camera parameters of the camera 30n stored in the camera parameter DB 24 and the estimated two-dimensional posture information of the object 90 to determine the shape of the object 90 by triangulation. Estimate three-dimensional posture information.
• step S14 the generation unit 13 projects the three-dimensional posture information of the object 90 onto each image 40n based on the camera parameters of the camera 30n that captured each image 40n, and Two-dimensional posture information of the object 90 is calculated. Then, the generation unit 13 generates a pseudo label 44n based on the calculated two-dimensional posture information.
• step S15 the selection unit 14 selects a pseudo label to be used for machine learning of the detector 22 from the pseudo labels 44n generated in step S14, based on spatiotemporal restrictions. Specifically, when the three-dimensional position of the object 90 indicated by the three-dimensional posture information that is the projection source when generating the pseudo label 44n is included in a predetermined range, the selection unit 14 selects the pseudo label 44n. Select 44n. Further, the selection unit 14 selects the pseudo label 44n when the photographing time of the image for which the pseudo label 44n has been generated is included in a predetermined time range.
• step S16 the selection unit 14 evaluates the quality of the pseudo label 44n selected in step S15, and if the evaluation result satisfies the criteria, selects it as the pseudo label 44n to be used for machine learning of the detector 22. do.
• step S17 the machine learning unit 15 uses the pseudo labeled image obtained by adding the pseudo label 44n selected in step S16 above to the image 40n and the correct answer image as training data to train the detector 22. Run machine learning.
• step S18 the machine learning unit 15 determines whether the end condition of the machine learning of the detector 22 is satisfied. For example, when the number of repetitions reaches a predetermined number, when the detection accuracy of the detector 22 reaches a predetermined value, when the detection accuracy of the detector 22 converges, etc., the machine learning unit 15 determines that the termination condition is satisfied. judge. If the termination condition is not satisfied, the process returns to step S11, and if the termination condition is satisfied, the information processing is terminated.
• the information processing apparatus determines the three-dimensional position of the object based on the two-dimensional position information of the object detected from each image included in the multi-view image and the camera parameters. Estimate information. Then, the information processing device projects the three-dimensional position information of the object onto each image based on the camera parameters, and calculates refined two-dimensional position information of the object. Thereby, the positional information of the object in the image can be calculated with high accuracy. Furthermore, by generating a pseudo label based on this two-dimensional position information, it is possible to reduce false negatives of the pseudo label and correct the bias in the position of the pseudo label.
• the information processing device selects a pseudo label to be used for machine learning of a detector from generated pseudo labels based on spatio-temporal restrictions, thereby reducing the diversity of poses of the target object. It is possible to generate well-balanced pseudo-labels according to the
• FIG. 14 shows an example of a pseudo label generation result by the information processing device according to the present embodiment.
• the left three diagrams in FIG. 14 schematically show an example of the detection results obtained by the method of detecting 2D-BBOX using the detector before applying semi-supervised learning in this embodiment (hereinafter referred to as the "comparison method").
• FIG. 14 the three diagrams on the right in FIG. 14 schematically show an example of detection results obtained by a method of detecting 2D-BBOX using a detector applying semi-supervised learning in this embodiment (hereinafter referred to as "this method").
• the information processing device can be applied to, for example, a scoring system for gymnastics competitions.
• a scoring system for gymnastics competitions for example, a scoring system for gymnastics competitions.
• the scoring system detects a region of a person from each image included in the multi-view image. Next, the scoring system determines whether the person indicated by the detected area is a player or a non-player based on whether the position where the person is present is in the competition area, etc., and identifies the area indicating the player.
• the scoring system tracks players by associating regions representing the same player in time-series multi-view images.
• the scoring system recognizes the player's two-dimensional skeletal information from each of the series of tracked images using a recognition model or the like.
• the scoring system estimates three-dimensional skeletal information from two-dimensional skeletal information using camera parameters. Then, the scoring system performs post-processing such as smoothing on the time-series three-dimensional skeletal information, estimates the phases (breaks) of the performance, and then recognizes the techniques.
• three-dimensional posture information which is estimated three-dimensional position information
• the image may be projected onto at least one of the multi-view images, such as by targeting an image in which the 2D-BBOX is not detected by the detector.
• the disclosed technology is not limited to cases where the object is a gymnast, but can be applied to various people such as athletes of other sports and ordinary pedestrians. Furthermore, it is also possible to apply the present invention to objects other than people, such as animals and vehicles.
• the information processing program is stored (installed) in the storage device in advance, but the information processing program is not limited thereto.
• the program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, DVD-ROM, or USB memory.
• Information processing device 11 Acquisition unit 12 Estimation unit 13 Generation unit 14 Selection unit 15 Machine learning unit 22 Detector 24 Camera parameter DB 30n Camera 40n Image 42n 2D-BBOX 44n Pseudo label 50 Computer 51 CPU 52 Memory 53 Storage device 54 Input/output device 55 R/W device 56 Communication I/F 57 Bus 59 Storage medium 60 Information processing program 61 Acquisition process control instruction 62 Estimation process control instruction 63 Generation process control instruction 64 Selection process control instruction 65 Machine learning process control instruction 70 Information storage area 90 Object

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Image Analysis (AREA)

Priority Applications (2)

Applications Claiming Priority (1)

Publications (1)

Family

ID=89705798

Family Applications (1)

Country Status (2)

Citations (3)

Family Cites Families (8)

• 2022
  • 2022-07-28 JP JP2024536705A patent/JPWO2024024055A1/ja active Pending
  • 2022-07-28 WO PCT/JP2022/029178 patent/WO2024024055A1/ja not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events