WO2024082996A1

WO2024082996A1 - Interaction method, in-vehicle infotainment system, vehicle comprising same, and storage medium

Info

Publication number: WO2024082996A1
Application number: PCT/CN2023/123723
Authority: WO
Inventors: 孙艘; 袁安贝; 刘潇
Original assignee: 蔚来汽车科技(安徽)有限公司
Priority date: 2022-10-19
Filing date: 2023-10-10
Publication date: 2024-04-25
Also published as: CN115686205A

Abstract

An interaction method (20), an in-vehicle infotainment system (30), a vehicle (100) comprising same, and a storage medium. The interaction method (20) comprises: receiving control data about extended reality content, the control data comprising first inertial measurement data in the current environment (S22); generating second inertial measurement data in the current environment (S24); and, on the basis of the first inertial measurement data and the second inertial measurement data, generating control rays for interacting with the extended reality content (S26).

Description

Interaction method, vehicle system, vehicle including the same, and storage medium

Technical Field

The present application relates to the field of extended reality display, and more specifically, to an interaction method, a vehicle system, a vehicle including the same, and a storage medium.

Background technique

At present, Bluetooth ring controllers are commonly used external control devices in the field of extended reality, but they are rarely used in vehicle environments. First, there is little demand for Bluetooth ring controllers in vehicle environments; second, due to the instability of driving conditions, the operation of passengers is easily affected by the driving conditions, so the interactive information generated by Bluetooth ring controllers will also be greatly affected.

In view of this, an improved interaction mechanism needs to be proposed.

Summary of the invention

Embodiments of the present application provide an interactive method, a vehicle system, a vehicle including the same, and a storage medium, for improving the operating accuracy of a control device such as a ring controller in a vehicle environment.

According to one aspect of the present application, an interaction method is provided. The method includes: receiving control data about extended reality content, wherein the control data includes first inertial measurement data in a current environment; generating second inertial measurement data in the current environment; and generating a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data.

In some embodiments of the present application, optionally, the control data also includes key control data and/or touch control data for interacting with the extended reality content.

In some embodiments of the present application, optionally, the control data comes from a ring controller.

In some embodiments of the present application, optionally, generating a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data includes: correcting the first inertial measurement data using the second inertial measurement data to generate drawing data about the control ray.

In some embodiments of the present application, optionally, the method further includes: sending the drawing data to a rendering device of the extended reality content for rendering the control ray.

In some embodiments of the present application, optionally, the second inertial measurement data is used to measure the first The inertial measurement data is corrected, comprising: extracting the overall motion situation in the current environment according to the second inertial measurement data; and removing the motion data caused by the overall motion situation from the first inertial measurement data to generate drawing data about the control ray.

In some embodiments of the present application, optionally, the first inertial measurement data and the second inertial measurement data are input into a neural network to generate the drawing data.

In some embodiments of the present application, optionally, the extended reality content includes at least one of the following: augmented reality content, virtual reality content.

According to another aspect of the present application, a vehicle system is provided. The system includes: a receiving unit configured to receive control data about extended reality content, wherein the control data includes first inertial measurement data in a current environment; a measuring unit configured to generate second inertial measurement data in the current environment; and a generating unit configured to generate a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data.

In some embodiments of the present application, optionally, the receiving unit communicates with the ring controller based on a Bluetooth protocol.

In some embodiments of the present application, optionally, the generating unit is configured to correct the first inertial measurement data using the second inertial measurement data to generate drawing data about the control ray.

In some embodiments of the present application, optionally, the generating unit is further configured to send the drawing data to a rendering device of the extended reality content for rendering the control ray.

In some embodiments of the present application, optionally, the generating unit is configured to: extract the overall motion situation in the current environment according to the second inertial measurement data; and remove the motion data caused by the overall motion situation from the first inertial measurement data to generate drawing data about the control ray.

In some embodiments of the present application, optionally, the generating unit is based on a neural network, and the generating unit inputs the first inertial measurement data and the second inertial measurement data into the neural network to generate the drawing data.

According to another aspect of the present application, a vehicle system is provided, wherein the system comprises: a memory configured to store instructions; and a processor configured to execute the instructions so as to perform any one of the methods described above.

According to another aspect of the present application, a vehicle is provided, wherein the vehicle comprises any vehicle system as described above.

According to another aspect of the present application, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and wherein when the instructions are executed by a processor, the processor is caused to execute any one of the methods described above.

The interactive method, vehicle system, vehicle including the same, and storage medium provided according to some embodiments of the present application can improve the operating accuracy of control devices such as ring controllers in a vehicle environment, thereby correctly reflecting the control intentions of the occupants and improving the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the present application will become more fully apparent from the following detailed description in conjunction with the accompanying drawings, wherein the same or similar elements are represented by the same reference numerals.

FIG1 shows an interaction scenario according to an embodiment of the present application;

FIG2 shows an interaction method according to an embodiment of the present application;

FIG3 shows a vehicle system according to an embodiment of the present application;

FIG. 4 shows a vehicle system according to an embodiment of the present application.

Detailed ways

For the purpose of brevity and illustration, the principles of the present application are described herein mainly with reference to its exemplary embodiments. However, those skilled in the art will readily recognize that the same principles can be equally applied to all types of interactive methods, vehicle-mounted systems and vehicles including them, and storage media, and these same or similar principles can be implemented therein, and any such changes do not depart from the true spirit and scope of the present application.

FIG1 shows an interactive scenario according to an embodiment of the present application, which depicts a scenario of using various extended reality devices (e.g., virtual reality devices, augmented reality devices) in a vehicle. Unlike using extended reality devices in a stationary space, when using extended reality devices in a vehicle environment, the vehicle is also in motion, and the extended reality device and the vehicle have the same speed in a certain direction.

In addition, the use of extended reality devices in a vehicle environment is susceptible to various unexpected vehicle movements. For example, unexpected vehicle movements may significantly affect the control direction of the occupant operator. When the occupant operator wants to point to a fixed point in the virtual reality scene, if the vehicle suddenly accelerates or encounters bumps, the occupant operator's torso may be affected and cannot point to the original desired location.

The following will describe the principle of using various extended reality devices in the vehicle 100 in conjunction with FIG. 1. The vehicle 100 may include multiple seats 101, and the number of passenger operators 200 (also referred to as objects 200 in this article) may not be unique. The number of objects 200 can be at most equal to the number of seats 101 equipped in the vehicle 100. The object 200 can interact with the extended reality screen 300 by wearing a ring controller 201, and the extended reality screen 300 can be generated by the head-mounted extended reality device (rendering device) of the object 200. The object 200 can wear multiple ring controllers 201 to implement complex operations. In addition, the extended reality screen 300 can be shared by multiple objects 200 in the vehicle 100. At this time, each object 200 in the vehicle 100 can wear a ring controller 201, and these ring controllers 201 can interact with the extended reality screen 300.

When the object interacts with the extended reality screen 300 through the ring controller 201, the extended reality screen 300 may display control rays 301 and 302 representing the current direction of the object 200. As shown in the figure, the root (vertex) of the control ray 302 will be connected to the ring controller 201 of the object 200 or its graphical representation. For example, the extended reality screen 300 may display a virtual hand, gun, laser pen, etc. as a graphical representation of the ring controller 201 of the object 200.

In the conventional solution, the ring controller 201 directly communicates with the head-mounted augmented reality device of the subject 200. In the present application, the ring controller 201 may not directly communicate with the head-mounted augmented reality device, but may process the control data through the vehicle system of the vehicle 100 and reflect the processing result on the augmented reality screen 300.

One aspect of the present application provides an interaction method. As shown in FIG2 , the interaction method 20 includes the following steps: receiving control data about extended reality content in step S22, wherein the control data includes first inertial measurement data in the current environment; generating second inertial measurement data in the current environment in step S24; and generating control rays for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data in step S26. The interaction method 20 can be executed by a vehicle system in the vehicle 100. By executing the above steps, the vehicle system can accurately reflect the original operation intention of the object 200 in the extended reality screen 300, thereby improving the interaction experience.

The interaction method 20 receives control data about the extended reality content in step S22, wherein the control data includes inertial measurement data in the current environment (referred to herein as first inertial measurement data to distinguish it from the second inertial measurement data described below). The control data here is generated by the object 200 through a device such as a ring controller 201, and the control data reflects the content of the interaction that the object 200 wants to perform with the extended reality screen 300.

As described in FIG. 1 , the positions of the control rays 301 and 302 are generated based on the motion data of the ring controller 201. For example, the position of the corresponding control ray can be determined based on the displacement of the ring controller 201 relative to the original position. To determine the real-time positions of the control rays 301 and 302, the inertial measurement data generated by the ring controller 201 can be sent to the vehicle system of the vehicle 100 in real time for processing.

It should be understood that when the user wants to interact with the extended reality screen 300 in the turbulent vehicle 100, the control data (specifically, the first inertial measurement data) received in step S22 will reflect the superposition properties of the torso movement of the object 200 and the movement of the vehicle 100 to a certain extent. However, the movement of the vehicle 100 is not what the object 200 expects when operating the ring controller 201. In other words, the movement of the vehicle 100 "interferes" with the object 200 operating the ring controller 201, and thus this interference factor needs to be eliminated.

In some embodiments of the present application, the control data received in step S22 may also include key control data and touch control data for interacting with the extended reality content. In addition to being used for indicating directions, the ring controller 201 may also have keys and touch control components. For example, keys can be used to perform operations such as clicking, and touch control can also achieve similar effects. Therefore, the control data may also include key control data and touch control data to achieve fine interaction with the extended reality content.

The interactive method 20 generates inertial measurement data in the current environment in step S24, which is also referred to as the second inertial measurement data in this article. The inertial measurement data generated by the vehicle system in step S24 is referred to as the second inertial measurement data here only to illustrate that there are differences in the subjects of the two, that is, the sources of the inertial measurement data are different. However, the first inertial measurement data and the second inertial measurement data can have the same data format, etc. Since the vehicle system is fixedly arranged relative to the vehicle 100, the second inertial measurement data is actually a representation of the movement of the vehicle 100 measured by the vehicle system.

In step S26, the interaction method 20 generates a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data. According to the above description, the first inertial measurement data received in step S22 reflects the superposition property of the torso movement of the object 200 and the movement of the vehicle 100, while the second inertial measurement data generated in step S24 is the vehicle system's measurement of the movement of the vehicle 100. Therefore, the influence of the second inertial measurement data can be deducted from the first inertial measurement data, and the result obtained is the expected interaction action of the object 200 with the extended reality screen 300. This interaction action can be represented by the control rays 301 and 302 in FIG. 1 .

For example, in some examples, the displacement of the control ray 301 or 302 can be obtained by subtracting the displacement in a certain direction (the direction of movement of the vehicle 100) from the first inertial measurement data, wherein the displacement in this direction can be derived through the second inertial measurement data. Specifically, if the second inertial measurement data has a large displacement in a certain direction, then the displacement may be caused by the movement of the vehicle 100; and the smaller displacement of the second inertial measurement data in other directions may be caused by the bumps of the vehicle 100. Therefore, the displacement of the first inertial measurement data in other directions can be filtered using an anti-shake algorithm. For example, the displacement of the first inertial measurement data in other directions can be smoothed. The smoothing method can be carried out according to the existing scheme, which will not be described in detail in this article.

In some embodiments of the present application, in step S26, the first inertial measurement data can be corrected using the second inertial measurement data to generate drawing data about the control ray. In some examples, this correction can be mathematically expressed as an operation of motion vectors in three dimensions of space. In some other examples, the correction can also be performed in a way that cannot be clearly expressed as a mathematical formula. For example, a correction scheme can be established under different second inertial measurement data scenarios based on preliminary experiments. When it is confirmed that a certain second inertial measurement data scenario has been entered, the correction scheme for the first inertial measurement data can be determined by methods such as a table lookup method. In addition, this "correction" will also filter out interference caused by vehicle bumps.

In some embodiments of the present application, in step S26, using the second inertial measurement data to correct the first inertial measurement data includes: extracting the overall motion situation in the current environment according to the second inertial measurement data; and removing the motion data caused by the overall motion situation from the first inertial measurement data to generate drawing data about the control ray. For example, the first inertial measurement data has a displacement of 0.2 on the spatial X-axis at a certain moment, and the second inertial measurement data has a displacement of 0.1 on the spatial X-axis at this time (the overall motion situation of the vehicle 100 and the object 200), then it can be inferred that the finger ring controller 201 has a displacement of 0.2-0.1=0.1. In other examples, considering that the object 200 will also perform a certain degree of self-correction in the scene where the speed changes sharply, it is inferred that the finger ring controller 201 can be 0.2-0.1*0.8=0.12 Displacement, where 0.8 is a correction coefficient. The above examples are for illustrative purposes only, and the actual situation is much more complicated. For example, the coefficient may not be fixed to 0.8, but can be adjusted with acceleration.

In some embodiments of the present application, in step S26, the first inertial measurement data and the first Second, the inertial measurement data is input into the neural network to generate the drawing data. The use of neural network technology to process the inertial measurement data will have the characteristics of adaptive self-learning. At the same time, it can automatically grasp the characteristics of the environment, has good fault tolerance, and strong anti-interference ability. Since the neural network does not need to establish a concrete mathematical relationship, the development process will be greatly shortened.

In some embodiments of the present application, the interaction method 20 further includes the following steps (not shown in FIG. 2 ): sending the drawing data to the rendering device of the extended reality content for rendering the control ray. The drawing data generated by the vehicle system is a mathematical expression of the control rays 301 and 302, which can be sent to the head-mounted rendering device to realize imaging, so that the object 200 can interact with the extended reality screen 300 according to the control rays 301 and 302.

As mentioned above, the extended reality device can be a virtual reality device, an augmented reality device, etc., so the extended reality content in this application can be augmented reality content, virtual reality content, or a combination of the two. The extended reality content can also be visual content generated by other augmented reality solutions developed in the future.

Another aspect of the present application provides a vehicle system. As shown in FIG3 , the vehicle system 30 includes a receiving unit 31, a measuring unit 32, and a generating unit 33. The vehicle system 30 can accurately reflect the original operation intention of the object 200 in the extended reality screen 300, thereby improving the interactive experience. In addition to the detailed description below, other features of the vehicle system 30 can be carried out with reference to the above-mentioned interactive method 20.

The receiving unit 31 of the vehicle system 30 is configured to receive control data about the extended reality content, wherein the control data includes the first inertial measurement data in the current environment. The control data here is generated by the object 200 through a device such as a ring controller 201, and the control data reflects the interaction content that the object 200 wants to perform with the extended reality screen 300.

In some embodiments of the present application, the control data received by the receiving unit 31 may also include key control data and touch control data for interacting with the extended reality content. In addition to being used for indicating directions, the ring controller 201 may also have keys and touch control components. For example, keys can be used to perform operations such as clicking, and touch control can also achieve similar effects. Therefore, the control data may also include key control data and touch control data to achieve fine interaction with the extended reality content.

In some embodiments of the present application, the receiving unit 31 communicates with the ring controller 201 based on the Bluetooth protocol. In other examples, other wireless communication protocols may be used to implement communication between the vehicle system 30 and the ring controller 201.

The measuring unit 32 of the vehicle system 30 is configured to generate a second inertial measurement number under the current environment. Here, the inertial measurement data generated by the vehicle system 30 through the measurement unit 32 is referred to as the second inertial measurement data only to illustrate that there are differences in the subjects of the two, that is, the sources of the inertial measurement data are different. However, the first inertial measurement data and the second inertial measurement data can have the same data format, etc. Since the vehicle system 30 is fixedly arranged relative to the vehicle 100, the second inertial measurement data is actually a representation of the movement of the vehicle 100 measured by the vehicle system 30.

The generation unit 33 of the vehicle system 30 is configured to generate a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data. According to the above description, the first inertial measurement data received by the receiving unit 31 reflects the superposition property of the torso movement of the object 200 and the movement of the vehicle 100, and the second inertial measurement data generated by the measuring unit 32 is a representation of the movement of the vehicle 100 measured by the vehicle system 30. Therefore, the influence of the second inertial measurement data can be deducted from the first inertial measurement data, and the result obtained is the expected interaction action of the object 200 with the extended reality screen 300. This interaction action can be expressed as control rays 301 and 302 in Figure 1.

For example, in some examples, the displacement of the control ray 301 or 302 can be obtained by subtracting the displacement in a certain direction (the direction of movement of the vehicle 100) from the first inertial measurement data, wherein the displacement in this direction can be derived through the second inertial measurement data. Specifically, if the second inertial measurement data has a large displacement in a certain direction, then the displacement may be caused by the movement of the vehicle 100; and the smaller displacement of the second inertial measurement data in other directions may be caused by the bumps of the vehicle 100. Therefore, the displacement of the first inertial measurement data in other directions can be filtered using an anti-shake algorithm. For example, the displacement of the first inertial measurement data in other directions can be smoothed. The smoothing method can be carried out according to the existing scheme, which is not described in detail in this article.

In some embodiments of the present application, the generation unit 33 is configured to correct the first inertial measurement data using the second inertial measurement data to generate drawing data about the control ray. In some examples, this correction can be mathematically expressed as an operation of a motion vector in three dimensions of space. In some other examples, the correction can also be performed in a way that cannot be clearly expressed as a mathematical formula. For example, a correction scheme can be established under different second inertial measurement data scenarios based on preliminary experiments. When it is confirmed that a certain second inertial measurement data scenario has been entered, the correction scheme for the first inertial measurement data can be determined by a method such as a table lookup. In addition, this "correction" will also filter out interference caused by vehicle bumps.

In some embodiments of the present application, the generating unit 33 is configured to: extract the overall motion situation in the current environment according to the second inertial measurement data; and remove the overall motion situation from the first inertial measurement data. The motion data caused by the dynamic situation is used to generate drawing data about the control ray. For example, the first inertial measurement data has a displacement of 0.2 in the spatial X-axis at a certain moment, and the second inertial measurement data has a displacement of 0.1 in the spatial X-axis at this time (the overall motion situation of the vehicle 100 and the object 200), then it can be inferred that the ring controller 201 has a displacement of 0.2-0.1=0.1. In other examples, considering that the object 200 will also perform a certain degree of self-correction in the scenario where the speed changes sharply, it is inferred that the ring controller 201 can be a displacement of 0.2-0.1*0.8=0.12, where 0.8 is the correction coefficient. The above examples are for illustrative purposes only, and the actual situation is much more complicated. For example, the coefficient may not be fixed at 0.8, but can be adjusted with acceleration.

In some embodiments of the present application, the generating unit 33 is based on a neural network, and the generating unit 33 inputs the first inertial measurement data and the second inertial measurement data into the neural network to generate drawing data. The use of neural network technology to process inertial measurement data will have the characteristics of adaptive self-learning. At the same time, it can also automatically grasp environmental characteristics, has good fault tolerance, and strong anti-interference ability. Since the neural network does not need to establish a concrete mathematical relationship, the development process will be greatly shortened.

In some embodiments of the present application, the generation unit 33 is further configured to send the drawing data to the rendering device of the extended reality content for rendering the control ray. The drawing data generated by the vehicle system 30 is a mathematical expression of the control rays 301 and 302, which can be sent to the head-mounted rendering device to realize imaging, so that the object 200 can interact with the extended reality screen 300 according to the control rays 301 and 302.

As mentioned above, the extended reality content can be augmented reality content, virtual reality content, or a combination of the two. The extended reality content can also be visual content generated by other augmented reality solutions developed in the future.

Another aspect of the present application provides a vehicle system 40. As shown in FIG4 , the vehicle system 40 includes a memory 41 and a processor 42. The processor 42 can read data from the memory 41 and can write data thereto. The memory 41 is configured to store instructions, and the processor 42 is configured to execute any one of the interaction methods described above when executing the instructions stored in the memory 41. The memory 41 can have the characteristics of a computer-readable storage medium as described below, and the details will be described below.

Another aspect of the present application provides a vehicle, wherein the vehicle includes any vehicle system as described above.

According to another aspect of the present application, a computer-readable storage medium is provided, wherein instructions are stored, and when the instructions are executed by a processor, the processor executes any one of the interaction methods described above. Method. The computer-readable medium referred to in this application includes various types of computer storage media, which can be any available medium that can be accessed by a general or special computer. For example, the computer-readable medium may include RAM, ROM, EPROM, ^E2PROM , register, hard disk, removable disk, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other temporary or non-temporary medium that can be used to carry or store the desired program code unit in the form of instructions or data structures and can be accessed by a general or special computer, or a general or special processor. As used herein, the disk usually copies data magnetically, while the dish uses a laser to optically copy data. The above combination should also be included in the protection scope of the computer-readable medium. The exemplary storage medium is coupled to the processor so that the processor can read and write information from/to the storage medium. In an alternative solution, the storage medium can be integrated into the processor. The processor and the storage medium can reside in the ASIC. The ASIC can reside in the user terminal. In an alternative solution, the processor and the storage medium can reside in the user terminal as discrete components.

The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art can think of other feasible changes or substitutions based on the technical scope disclosed in the present application, and such changes or substitutions are all included in the protection scope of the present application. In the absence of conflict, the implementation modes of the present application and the features in the implementation modes can also be combined with each other. The protection scope of the present application shall be subject to the description of the claims.

Claims

An interactive method, characterized in that the method comprises:

receiving control data regarding the extended reality content, wherein the control data includes first inertial measurement data in a current environment;

generating second inertial measurement data under the current environment; and

A control ray for interacting with the augmented reality content is generated based on the first inertial measurement data and the second inertial measurement data.
The method according to claim 1, wherein the control data further includes key control data and/or touch control data for interacting with the extended reality content.
The method of claim 1, wherein the control data is from a finger ring controller.
The method according to claim 1, wherein generating a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data comprises:

The first inertial measurement data is corrected using the second inertial measurement data to generate rendering data related to the control ray.
The method according to claim 4 further comprises: sending the drawing data to a rendering device of the extended reality content for rendering the control ray.
The method according to claim 4, wherein correcting the first inertial measurement data using the second inertial measurement data comprises:

Extracting the overall motion situation in the current environment according to the second inertial measurement data; and

The motion data caused by the overall motion condition is removed from the first inertial measurement data to generate rendering data about the control ray.
The method of claim 4, wherein the first inertial measurement data and the second inertial measurement data are input into a neural network to generate the rendering data.
The method according to claim 1, wherein the extended reality content includes at least one of the following: augmented reality content, virtual reality content.
A vehicle computer system, characterized in that the system comprises:

a receiving unit configured to receive control data regarding the extended reality content, wherein the control data includes first inertial measurement data in a current environment;

a measurement unit configured to generate second inertial measurement data under the current environment; and

A generating unit is configured to generate a control ray for interacting with the extended reality content based on the first inertial measurement data and the second inertial measurement data.
The system according to claim 9, wherein the control data further includes key control data and/or touch control data for interacting with the extended reality content.
The system of claim 9, wherein the control data is from a finger ring controller.
The system according to claim 11, wherein the receiving unit communicates with the ring controller based on a Bluetooth protocol.
The system according to claim 9, wherein the generating unit is configured to correct the first inertial measurement data by using the second inertial measurement data to generate the rendering data about the control ray.
The system according to claim 13, wherein the generating unit is further configured to send the drawing data to a rendering device of the extended reality content for rendering the control ray.
The system according to claim 13, wherein the generating unit is configured to:

Extracting the overall motion situation in the current environment according to the second inertial measurement data; and

The motion data caused by the overall motion condition is removed from the first inertial measurement data to generate rendering data about the control ray.
The system according to claim 13, wherein the generating unit is based on a neural network, and the generating unit inputs the first inertial measurement data and the second inertial measurement data into the neural network to generate the drawing data.
The method according to claim 9, wherein the extended reality content includes at least one of the following:

Augmented reality content, virtual reality content.
A vehicle computer system, characterized in that the system comprises:

a memory configured to store instructions; and

A processor configured to execute the instructions so as to perform the method according to any one of claims 1 to 8.
A vehicle, characterized in that the vehicle comprises the vehicle system as described in any one of claims 9-18.
A computer-readable storage medium stores instructions, wherein when the instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 8.