Available translations
loading results
Data Mining - Indicator Bots - Data Mining Workflow
Summary: When running on a data mining operation, indicators follow a cycle designed to fulfill their purpose: to process data others have produced and make the result available for others to consume.
Indicators take datasets produced by sensors or by other indicators as inputs. These inputs are determined by the defined data dependencies.
The calculations procedure and the data building procedure produce the intended transformations. The data building procedure stores the results in datasets that others may use as input, and the calculations procedure may make certain pieces of information available without ever writing it in the files.
The following is the breakdown of the cycle.
1. Loading of data dependencies
Datasets produced by indicators are sets of text files with data stored in the form of arrays. Each value in a record is separated by colons, and each record is delimited by square brackets. Records are separated by commas. The format in which data is stored in files aims to make files as lightweight as possible.
For example, the following fragment features three records in the 24-hours Multi-Time-Frame-Market Bollinger Bands Data.json file:
... ,[1586044800000,1586131199999,6339.464,440.8934655492186,881.7869310984372],[1586131200000,1586217599999,6440.327,424.9041072418576,849.8082144837152],[1586217600000,1586303999999,6530.541,382.3824677453191,764.7649354906382], ... That is what the system encounters at the time of reading a typical file in a dataset, which is the first step in the runtime cycle.
2. Inflation of files into JSON objects with named properties
Once a file is loaded in memory, it is inflated into a JSON object structure, with named properties. This is the second step in the cycle and aims to make accessing each record—from within the code—much easier.
3. Calculation of properties not stored in the dataset
Because performance is inversely related to the weight of files, developers may choose not to store all indicator properties in files. They may choose to leave properties that are easy to calculate out of the dataset and make them available as calculated properties instead. Calculated properties are calculated by the calculations procedure and fed to the record definition formulas to be made available as an integral part of the object other indicators may access.
For example, picture an indicator that offers buy, sell, and total volumes. Because the total volume is so easy to calculate, the developer of such an indicator may choose to store the buy and sell volumes only and make the total volume available as a calculated property.
Tip: Make your indicators more efficient by limiting the amount of information you store in files.
The third step in the runtime cycle is, therefore, to run the calculations procedure of each data dependency and add the calculated properties to the JSON object structure in memory, so that all properties are available for the next steps in the cycle.
4. Products structure
Steps 1, 2, and 3 are run for each one of the data dependencies, and the results are put inside an object named products. In this way, all data dependencies may be easily accessed with the following kind of syntax:
products.pluralVariableName[index].property For example:
products.bollingerBands[10].deviation Each product remains an array of JSON objects with named variables.
5. Data building
The fifth step in the runtime cycle is running the indicator’s data building procedure. This is where the developer codes the logic to go over the input datasets and perform the calculations required to produce the indicator’s output.
Indicators may have multiple products, thus, the data building procedure corresponding to each product is run sequentially, in the order products have been defined around the indicator bot node.
As transformations for a given product are finished, the results are appended to the products object and made available immediately for the rest of the products in the queue to use as input, in addition to the original data dependencies. Data dependencies are the same for all products, as data dependencies are defined at the level of processes.
6. Writing the output
The final step is turning the object-formatted records back into a minimized array format, and saving the corresponding files in the dataset.
Appendix: Building 2 Interdependent Indicators
Let's say you built indicator1 with product A that depends on Indicator Candles Volumens, product Candles, and you are building indicator2, product B that depends on indicator1 product A.
When you wrote the code for Indicator 1 Product A, you were using the syntax:
variable property1 =
record.current.open Why? Because Indicator 1 depends on Candles Volumens, product Candles, and each candle has a property called open, and since product Candles is the only data dependency, it is then also the main dependency and then it will be served to you inside the record object for your convenience.
You could also have done:
variable property1 =
products.candles[index] but that is more difficult because you don't really know which index you are at at any given moment in time. It is best and easier to use the record object. The array is for very advanced users. Then you want to write the code for indicator 2, product B, you already declared Indicator 1, Product A as the only data dependency.
Since Indicator1 product A is the only data dependency of indicator 2, then it is also the main dependency, which means that for building indicator 2, the Data Building Process will iterate over every element of Indicator 1 in this case (not candles because that is not a dependency at all).
In the same way that you did before, you will write code like this:
variable
propertyX = record.current.property1 (that means the property 1 of Product A of Indicator 1) Data Mining — TOC
You just read page 19 in the topic.
3. Data Mining - Dataset Types
4. Data Mining - Datasets Type Conversions
5. Data Mining - Indicator Frameworks
6. Data Mining - Data Mine Hierarchy
7. Data Mining - Managing Execution
8. Data Mining - Data Dependencies
9. Data Mining - Status Reports
10. Data Mining - Status Dependencies
14. Data Mining - Fetching Data From APIs
15. Data Mining - Consuming Data From Strategies
16. Data Mining - Indicator Bots
17. Data Mining - Indicator Bots - Process Definition
18. Data Mining - Indicator Bots - Product Definition
19. Data Mining - Indicator Bots - Data Mining Workflow
20. Data Mining - Indicator Bots - Coding the Indicator's Logic