JP7202047B1 - Program, information processing system and information processing method - Google Patents

Abstract

A program, an information processing system, and an information processing method capable of realizing highly real-time signal processing irrespective of the number of channels without user devising special algorithms are provided. A program of an information processing device in an information processing system causes at least one computer to execute the following steps. In the reading step, the signal processing algorithm implemented by the user is read. The first receiving step receives a signal output from at least one information input device connected to the computer. In the parallel processing step, parallel threads corresponding to the number of signal channels and dedicated threads independent of algorithms are established to execute signal processing. [Selection drawing] Fig. 1

Description

Application of Article 30, Paragraph 2 of the Patent Act On January 3, 2022, “MIMO DSP Platform” https://mimodsp. liberal logic. co. jp/ja/ and https://mimodsp. liberal logic. co. Announced at jp/en/ [Publications, etc.] Online meeting with NTT DoCoMo, Inc. on January 31, 2022, online briefing on February 10, 2022, and March 25, 2022 [Publications, etc.] On April 1, 2020, Crescent Co., Ltd.'s website https://crescentinc. co. Announced at jp/product/mimo/p_top/ [Publications, etc.] At the actual machine demonstration briefing held for Keisoku Giken Co., Ltd. held at Liberal Logic Co., Ltd. on May 13, 2022. Presentation

The present invention relates to a program, an information processing system and an information processing method.

Japanese Patent Application Laid-Open No. 2002-200002 discloses an image processing apparatus that performs image processing in parallel as an example of an apparatus that performs signal processing.

In this image processing device, the image processing unit performs filtering on the pixels of the divided image and the pixels of the marginal pixels stored in the pixel storage unit that do not refer to the pixels of the divided image that are not stored in the pixel storage unit. I do. In addition, the image processing unit that performs filter processing in parallel with the image processing unit performs filter processing on pixels that do not refer to the pixels of the divided image among the pixels of the divided image.

JP 2015-138417 A

However, like the technology disclosed in Patent Document 1, when researching and developing signal processing algorithms, users need to write source code for parallel processing of multiple channels for each product, which is time-consuming. It was hanging

In view of the above circumstances, the present invention provides a program or the like that can realize highly real-time signal processing regardless of the number of channels without the user having to devise special algorithms.

According to one aspect of the invention, a program is provided. This program causes at least one computer to perform the following steps. In the reading step, the signal processing algorithm implemented by the user is read. The first receiving step receives a signal output from at least one information input device connected to the computer. In the parallel processing step, parallel threads corresponding to the number of signal channels and dedicated threads independent of algorithms are established to execute signal processing.

According to this aspect, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm.

1 is a configuration diagram showing an information processing system 1 according to this embodiment; FIG. 2 is a block diagram showing the hardware configuration of the information processing device 2; FIG. 3 is a block diagram showing functions realized by a control unit 23 and the like in the information processing apparatus 2 according to the embodiment; FIG. 4 is an activity diagram showing an example of an information processing method by the information processing system 1 according to this embodiment; FIG.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments shown below can be combined with each other.

By the way, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium (Non-Transitory Computer-Readable Medium), or may be downloaded from an external server. It may be provided as possible, or may be provided so that the program is activated on an external computer and the function is realized on the client terminal (so-called cloud computing).

Further, in the present embodiment, the term “unit” may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be specifically realized by these hardware resources. . In addition, various information is handled in the present embodiment, and these information are, for example, physical values of signal values representing voltage and current, and signal values as binary bit aggregates composed of 0 or 1. It is represented by high and low, or quantum superposition (so-called quantum bit), and communication and operation can be performed on a circuit in a broad sense.

A circuit in a broad sense is a circuit implemented by appropriately combining at least circuits, circuits, processors, memories, and the like. Application Specific Integrated Circuits (ASICs), programmable logic devices (e.g., Simple Programmable Logic Devices (SPLDs), Complex Programmable Logic Devices (CPLDs), and field It includes a programmable gate array (Field Programmable Gate Array: FPGA).

[Embodiment]
1. Hardware Configuration This section describes the hardware configuration of this embodiment.

1.1 Information processing system 1
FIG. 1 is a configuration diagram showing an information processing system 1 according to this embodiment. In one example, an information processing system 1 includes an information processing device 2, a plurality of imaging devices 3 (an example of an information input device), and a display device 4 (an example of an information output device). A plurality of imaging devices 3 are connected to the information processing device 2 via a general-purpose or dedicated communication standard. The display device 4 is connected to the information processing device 2 via a general-purpose or dedicated communication standard. Here, the system exemplified by the information processing system 1 consists of one or more devices or components. Therefore, even the information processing device 2 alone is included in the system exemplified by the information processing system 1 . Each component included in the information processing system 1 will be further described below.

1.3 Information processing device 2
FIG. 2 is a block diagram showing the hardware configuration of the information processing device 2. As shown in FIG. The information processing device 2 has a communication unit 21 , a storage unit 22 , a control unit 23 and an input unit 25 , and these components are electrically connected inside the information processing device 2 via the communication bus 20 . It is connected. Each component will be further described.

The communication unit 21 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but wireless LAN network communication, mobile communication such as 3G/LTE/5G, Bluetooth (registered trademark), etc. Communication and the like may be included as desired. That is, it is more preferable to implement as a set of these communication means. That is, the information processing device 2 may communicate various information from the outside via the communication unit 21 and the communication network. Particularly preferably, the communication unit 21 is connected to each of the plurality of imaging devices 3 and display devices 4 via a general-purpose or dedicated communication standard.

The storage unit 22 stores various information defined by the above description. For example, it can be used as a storage device such as a solid state drive (SSD) for storing various programs related to the information processing device 2 executed by the control unit 23, or as a temporary storage device related to program calculation. It can be implemented as a memory such as a Random Access Memory (RAM) that stores information (arguments, arrays, etc.) required for the . The storage unit 22 stores various programs, variables, etc. related to the information processing device 2 executed by the control unit 23 .

The control unit 23 processes and controls overall operations related to the information processing device 2 . The control unit 23 is, for example, a central processing unit (CPU) (not shown). The control unit 23 implements various functions related to the information processing device 2 by reading a predetermined program stored in the storage unit 22 . That is, information processing by software stored in the storage unit 22 can be specifically realized by the control unit 23 which is an example of hardware, and can be executed as each functional unit included in the control unit 23 . These are further detailed in the next section. Note that the control unit 23 is not limited to a single unit, and may be implemented to have a plurality of control units 23 for each function. A combination thereof may also be used. That is, the control unit 23 is an example of a processor capable of executing a program for executing each step described later.

The input unit 25 may be included in the housing of the information processing device 2 or may be externally attached. For example, the input unit 25 may be implemented as a touch panel integrated with the display device 4 described later. With a touch panel, the user can input a tap operation, a swipe operation, or the like. Of course, a switch button, a mouse, a QWERTY keyboard, or the like may be employed instead of the touch panel. That is, the input unit 25 receives an operation input made by the user. The input is transferred as a command signal to the control unit 23 via the communication bus 20, and the control unit 23 can execute predetermined control and calculation as necessary.

The information processing device 2 may be a computer installed with a general-purpose OS, or may be a dedicated device equipped with a dedicated circuit for signal processing.

1.3 Imaging device 3
The imaging device 3 is a so-called vision sensor (camera) configured to be able to acquire information of the outside world as an image. The resolution of the imaging device 3 is not particularly limited, and an imaging device 3 with high spatial resolution (resolution) may be employed, or an imaging device 3 with high temporal resolution (frame rate) is preferably employed. In one example, imaging devices 3-1, 3-2, 3-3 . . . 3-n are connected to the information processing device 2 as the n imaging devices 3. FIG.

The resolution of the imaging device 3 may be full HD or lower, full HD, WQHD, 2K, 4K, 8K, or higher.

Specifically, the frame rate of the imaging device 3 is, for example, 325,350,375,400,425,450,475,500,525,550,575,600,625,650,675,700,725,750,775,800,825,850,875,900,925, 950, 975, 1000, 1025, 1050, 1075, 1100, 1125, 1150, 1175, 1200, 1225, 1250, 1275, 1300, 1325, 1350, 1375, 1400, 1425, 1450, 1475, 1500, 1525, 1550, 1575, 1600, 1625, 1650, 1675, 1700, 1725, 1750, 1775, 1800, 1825, 1850, 1875, 1900, 1925, 1950, 1975, 2000 fps, and between any two of the numerical values exemplified here may be within the range. Alternatively, a processing rate having one second or more per frame may be used.

1.4 Display device 4
The display device 4 is configured to be able to display time-series images (video) output from the information processing device 2 . The display device 4 is, for example, a CRT display, a liquid crystal display, an organic EL display, a plasma display, or the like. Moreover, the display device 4 may include an audio output unit (not shown). The audio output unit is configured to be capable of outputting the audio of the video output from the information processing device 2 . Note that the information processing system 1 may be implemented so as to have a plurality of display devices 4 .

2. Functional Configuration This section describes the functional configuration of this embodiment. As described above, information processing by software stored in the storage unit 22 is specifically realized by the control unit 23, which is an example of hardware, so that each function unit included in the control unit 23 can be executed.

FIG. 3 is a block diagram showing functions realized by the control unit 23 and the like in the information processing apparatus 2 according to this embodiment. Specifically, the information processing device 2 includes a reading unit 231, a reception unit 232, a parallel processing unit 233, an optimization unit 234, and an output unit 235 as functional units.

The reading unit 231 is configured to read various information stored in the storage unit 22 by writing the information to the cache memory of the storage unit 22 .

The reception unit 232 is configured to receive various types of information from the outside. Preferably, the image output from the imaging device 3 is received.

The parallel processing unit 233 is configured to perform allocation of cores of the control unit 23 for performing signal processing on received video of multiple channels, for example, n channels, so that parallel processing is possible.

The optimization unit 234 is configured to control the operation of the information processing system 1 so as to optimize the data flow for parallel processing.

The output unit 235 displays various types of information, screens, images, etc. containing these information in a form that can be visually recognized on the display device 4 . The output unit 235 may generate only rendering information for displaying visual information on the display device 4 . Also, the output unit 235 may output audio data, or may output metadata.

3. Information Processing Method FIG. 4 is an activity diagram showing an example of an information processing method by the information processing system 1 according to the present embodiment. Each activity in this figure will be described below. Here, an example in which images captured by the imaging devices 3 having N channels are synthesized in the information processing device 2 and a high-resolution output image is displayed on the display device 4 will be taken up. More specifically, for example, the imaging device 3 is a camera having a resolution of 8K, and the output image to be displayed on the display device 4 is an ultra-high resolution image, for example, a 64K image.

First, the user implements in advance an algorithm AL related to the synthesis process (activity A001). Preferably, the program is a software development kit for signal processing, and the user implements the algorithm AL using the software development kit. According to this aspect, the present program can be used as middleware to more easily realize software development for signal processing. Note that various libraries for image processing or signal processing may be employed to implement the algorithm AL. Such libraries can include, for example, at least OpenCV, CUDA and OpenGL. Algorithm AL also does not include instructions regarding the data flow regarding the external input and external output of the target data itself. In other words, algorithm AL does not contain instructions for thread processing of signals.

Subsequently, the storage unit 22 in the information processing device 2 stores the implemented algorithm (activity A002).

Subsequently, by connecting the imaging device 3 having N channels to the input side of the information processing device 2 and connecting the display device 4 to the output side of the information processing device 2, the information processing system 1 shown in FIG. constructed (activity A003).

From here, the process of synthesizing the captured images captured by the imaging devices 3 with N channels in the information processing device 2 and displaying the high-resolution output image on the display device 4 starts. The reading unit 231 of the information processing device 2 reads the signal processing algorithm AL implemented by the user from the storage unit 22 (activity A004). In addition, a captured image captured by the imaging device 3 having N channels is transmitted to the information processing device 2 . In other words, the reception unit 232 of the information processing device 2, as a first reception step, is an example of the signal SG output by at least one imaging device 3 (an example of an information input device) connected to the information processing device 2. (Activity A005). Note that the reception unit 232 may receive audio data together with the captured image. In other words, the signal SG is preferably a signal relating to images or sounds.

Subsequently, the parallel processing unit 233 in the information processing device 2 constructs parallel processing for executing the algorithm AL based on the contents of the algorithm AL, the type of the signal SG and the number of channels N (activity A006). By the way, as described above, the algorithm AL does not include instructions regarding the data flow regarding the input of the target data itself from the outside and the output to the outside. Even in such a case, the parallel processing unit 233 establishes parallel threads corresponding to the number N of channels of the signal SG and an independent dedicated thread for the algorithm AL to execute signal processing. Here, the receiving unit 232 continuously receives captured images, which are an example of the signal SG, for each frame (activity A007), and the above-described image processing is executed using parallel threads defined by the parallel processing unit 233. (Activity A008). More preferably, the optimization section 234 optimizes the data flow of each signal SG processed in the parallel processing section 233 . According to such an aspect, it is possible to handle a data flow obtained by appropriately dividing the signal SG of a plurality of channels, and realize highly real-time signal processing.

Also, the parallel processing unit 233 executes the algorithm AL without compiling based on at least one of the algorithm AL and the signal SG and preset reference information. The reference information is information about various input signals, their signal processing details, and preferable parallel processing, and may be rule-based information, or a trained model in which the relationship between the two is learned in advance. . More preferably, the reference information may include a predetermined file F containing at least the number of channels N of the signal SG. In such a case, preferably, the reading unit 231 further reads a predetermined file containing the number of signal channels, and the parallel processing unit 233 establishes parallel threads based on the predetermined file and executes the algorithm. According to such an aspect, signal processing of one or more channels can be realized without recompiling the algorithm AL. More specifically, by preparing reference information corresponding to various algorithms AL and input signals SG, the algorithms AL and signals SG can add functions to the information processing system 1 in the form of plug-ins. possible, and can be implemented without recompiling the algorithm AL itself.

Here, preferably, the signal SG is a plurality of data strings, and the parallel processing section 233 processes the data strings with FIFO (First-In First-Out). That is, parallel processing is performed so that the order of input/output of the signal SG is preserved. According to this aspect, the front stage and the rear stage of the processing can be operated in parallel by separate CPU cores, so that the calculation processing can be scaled with the number of CPU cores present in the system. As a result, real-time performance can be enhanced.

Finally, as a result of executing the algorithm AL by parallel processing, a desired image obtained by synthesizing the captured image, which is an example of the signal SG, is output (activity A009). In other words, the output unit 235 in the information processing device 2 outputs the output signal SG to which the algorithm AL is applied based on the optimized data flow. The display device 4 displays a synthesized image, which is an example of the output signal SG. According to this aspect, a desired output result can be obtained through highly real-time signal processing.

After that, the input of the captured image, the image processing, and the output of the desired image shown in activities A007 to A009 are repeated for each frame of the signal SG input to the information processing apparatus 2 for each frame. Also, the processing related to input, the image processing, and the processing related to output may be performed simultaneously by parallel threads. Specifically, input of a frame at a certain time, image processing for a frame at a first past time that is past the certain time, and processing for a frame at a second past time that is past the first past time. can be performed concurrently by parallel threads.

In summary, the information processing system 1 according to this embodiment includes a control unit 23 (processor) capable of executing a program. This program causes at least one information processing device 2, which is an example of a computer, to execute the following steps. The reading unit 231 reads out the signal processing algorithm AL implemented by the user. The receiving unit 232 receives a captured image, which is an example of the signal SG, output by the imaging device 3, which is an example of at least one information input device, connected to a computer. The parallel processing unit 233 establishes parallel threads corresponding to the number N of channels of the signal SG and an independent dedicated thread for the algorithm AL to execute signal processing.

Alternatively, this program causes at least one information processing device 2, which is an example of a computer, to execute the following steps. The reading unit 231 reads out the signal processing algorithm AL implemented by the user. The receiving unit 232 receives a captured image, which is an example of the signal SG, output by the imaging device 3, which is an example of at least one information input device, connected to a computer. The parallel processing unit 233 establishes parallel threads corresponding to the number of channels N of the signal SG and executes the algorithm AL when the algorithm AL does not include an instruction related to thread processing of the signal SG.

That is, this information processing method comprises each step of the above program.

According to this aspect, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm. In particular, it is possible to realize highly real-time signal processing for high-resolution video and audio that require a wide band.

Furthermore, without special algorithm AL design for the imaging device 3 (input side) and the display device 4 (output side) connected to the information processing device 2, input/output can be flexibly adapted according to individual requirements. An information input device and an information output device, which are devices, can be selected. Specifically, various plug-ins such as input/output using the SDI video card SDK, input/output using the NDI API, signal generation plug-in related to the input side, and input/output using the network API can be realized.

[others]
The foregoing embodiments may be further elaborated as follows.

The user may be able to change the predetermined file F, which is one aspect of the reference information. In other words, the accepting unit 232 accepts a user input to change the number of channels N included in the predetermined file F as the second accepting step. The parallel processing unit 233 sets up parallel threads based on the changed predetermined file F and executes the algorithm AL. According to this aspect, the user's desired number of channels N can be realized without recompiling the algorithm AL.

The aforementioned signal processing is an example, and may be image processing or audio data processing. The number of channels N of signals handled by the information processing device 2 does not have to match the number of information input devices exemplified by the imaging device 3 . The imaging device 3 may be a reproducing device that reproduces pre-captured video or audio. The display device 4 may be an audio reproduction device that does not have a display function.

A signal input from an information input device such as the imaging device 3 to the information processing device 2 may be a signal relating to metadata. Metadata may be, for example, metadata associated with images or audio. In relation to the metadata, for example, the imaging device 3 may be a ToF (Time-of-Flight) camera. In such a case, the depth information output from the ToF camera is an example of metadata. The imaging device 3 may be an AI camera capable of recognizing a specific target. In this case, the positional information of the specific target output from the AI camera and the time information about the time are examples of the metadata. According to such an aspect, various signal processing using metadata can be realized with improved real-time performance.

The information processing device 2 may be configured to be connectable to other external devices through a network line. In such a case, the user can realize signal processing with efficient data flow as a SaaS service regardless of the number of channels N without devising a special algorithm AL.

The information processing device 2 may be implemented as a sound card specialized for audio processing or a graphic card specialized for image processing. In such a case, the information processing system 1 according to this embodiment can be realized by installing such a dedicated card in an existing computer.

Furthermore, it may be provided in each aspect described below.

(1) A program causing at least one computer to execute the following steps, reading a signal processing algorithm implemented by a user in a reading step, and connecting to the computer in a first receiving step receive a signal output from at least one information input device, and in the parallel processing step, set up parallel threads corresponding to the number of channels of the signal and an independent dedicated thread for the algorithm, and execute the signal processing ,thing.

According to this aspect, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm.

(2) A program causing at least one computer to execute the following steps, reading a signal processing algorithm implemented by a user in a reading step, and connecting to the computer in a first receiving step; receiving a signal output from at least one information input device, and in the parallel processing step, if the algorithm does not include an instruction regarding thread processing of the signal, parallel threads are set up according to the number of channels of the signal. , which executes said algorithm.

According to this aspect, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm.

(3) In the program according to (1) or (2) above, the reading step further reads a predetermined file containing the number of channels of the signal, and the parallel processing step reads the parallel processing based on the predetermined file. One that spawns a thread and runs the algorithm.

According to such an aspect, signal processing of one or more channels can be realized without recompiling algorithms.

(4) In the program described in (3) above, further, the second receiving step receives user input to change the number of channels included in the predetermined file, and the parallel processing step receives the changed predetermined number of channels. Based on a file, spawn said parallel threads to run said algorithm.

According to this aspect, the number of channels desired by the user can be realized without recompiling the algorithm.

(5) In the program according to any one of (1) to (4) above, in the parallel processing step, based on at least one of the algorithm and the signal and preset reference information, and run the algorithm without compiling.

According to this aspect, by preparing reference information corresponding to various algorithms and input signals, it becomes possible for the algorithms and signals to give functions to the system in the form of plug-ins. can be realized without recompiling.

(6) The program according to any one of (1) to (5) above, wherein the optimization step further optimizes the data flow of each of the signals processed in the parallel processing step.

According to such an aspect, it is possible to handle a data flow in which the signals of a plurality of channels are appropriately separated, and realize highly real-time signal processing.

(7) In the program described in (6) above, the output step further outputs an output signal to which the algorithm is applied based on the optimized data flow.

According to this aspect, a desired output result can be obtained through highly real-time signal processing.

(8) In the program according to any one of (1) to (7) above, the signal is a plurality of data strings, and in the parallel processing step, the data strings are stored in FIFO (First-In First- Out).

According to this aspect, the front stage and the rear stage of the processing can be operated in parallel by separate CPU cores, so that the calculation processing can be scaled with the number of CPU cores present in the system. As a result, real-time performance can be enhanced.

(9) In the program according to any one of (1) to (8) above, the signal is a signal relating to image or sound.

According to this aspect, it is possible to realize highly real-time signal processing for high-resolution video and audio that require a wide band.

(10) In the program according to any one of (1) to (9) above, the signal is a signal relating to metadata.

According to such an aspect, various signal processing using metadata can be realized with improved real-time performance.

(11) The program according to any one of (1) to (10) above, which is a software development kit for the signal processing.

According to this aspect, the present program can be used as middleware to more easily realize software development for signal processing.

(12) An information processing system comprising a processor capable of executing the program according to any one of (1) to (11) above.

According to this aspect, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm.

(13) An information processing method, comprising steps of the program according to any one of (1) to (11) above.

According to such a method, the user can realize highly real-time signal processing regardless of the number of channels without devising a special algorithm.
Of course, this is not the only case.

Finally, while various embodiments of the invention have been described, these have been presented by way of example and are not intended to limit the scope of the invention. The novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

1: information processing system 2: information processing device 20: communication bus 21: communication unit 22: storage unit 23: control unit 25: input unit 231: reading unit 232: reception unit 233: parallel processing unit 234: optimization unit 235: Output unit 3: Imaging device 4: Display device AL: Algorithm F: Predetermined file N: Number of channels SG: Signal

Claims (11)

A software development kit program comprising:
causing at least one computer to perform each of the following steps;
In the reading step, the signal processing algorithm implemented by the user is read out via the software development kit program ,
In a first receiving step, receiving a signal output from at least one information input device connected to the computer;
In the parallel processing step, even if the algorithm does not include instructions related to thread processing of the signal, parallel threads corresponding to the number of channels of the signal and independent dedicated threads of the algorithm are set up to perform the signal processing. The one that runs the . The software development kit program according to claim 1,
The reading step further reads a predetermined file containing the number of channels of the signal,
In the parallel processing step, the parallel threads are established based on the predetermined file to execute the algorithm. In the software development kit program according to claim 2 ,
Furthermore, in a second receiving step, a user's change input for the number of channels included in the predetermined file is received,
In the parallel processing step, the parallel threads are set up and the algorithm is executed based on the changed predetermined file. The software development kit program according to claim 1,
In the parallel processing step, the algorithm is executed without compiling based on at least one of the algorithm and the signal and preset reference information. The software development kit program according to claim 1,
Further, the optimizing step optimizes the data flow of each of said signals processed in said parallel processing step. In the software development kit program according to claim 5 ,
Further, the output step outputs an output signal to which the algorithm is applied based on the optimized data flow. The software development kit program according to claim 1,
The signal is a plurality of data strings,
In the parallel processing step, the data string is processed by FIFO (First-In First-Out). The software development kit program according to claim 1,
The signal is a signal relating to image or sound. The software development kit program according to claim 1,
The signal is a signal relating to metadata. An information processing system,
10. A processor capable of executing the software development kit program according to any one of claims 1-9 . An information processing method,
A method comprising the steps of a software development kit program according to any one of claims 1-9 .

Images