Available translations
loading results
Data Mining - Data Mine Hierarchy
Summary: Data mines enable the processing of data by consuming datasets others have produced, performing calculations, and outputting a new value-added data product that others may consume. In other words, Data Mines are user created packages of definitions. These definitions usually include Sensor Bot, Indicator Bot and Plotter definitions. The Data Mines shipped with Superalgos are packages contributed by users of the software.
Superalgos puts a powerful data processing and visualization system into your hands. Use it to the fullest and you will see your intelligence increase by orders of magnitude.
Anyone may access raw trades and order book data from exchanges. Anyone may access information in the form of standard indicators and technical studies.
However, Superalgos allows you to perform any sort of processing over existing information across the whole span of the market, store the results into new datasets, and make them available to trading strategies. Moreover, it allows you to produce elaborate graphical representations of data, which is instantly browsable over the charts.
Data mines’ functionality is highly flexible and works under the same paradigm used for building strategies. They provide a visual interface and a framework that leads you through the process of creating new bots, with minimal coding skills required.
The vast majority of the complexity of processing data is handled internally by the bots, and you don’t need to deal with it or even understand it. The system isolates complexity and only exposes you to the crucial elements that impact what you wish to accomplish.
This section of the documentation provides you with the technical understanding of how data is handled by the system. The core of the functionality this section unveils covers—in essence—the following points:
The capacity to define any number of calculation processes that may feed on each other’s outputs, resulting in a wealth of information that may be consumed by trading strategies.
The properties and characteristics of datasets resulting from the execution of such processes—how and where they are stored, in what format, how they are consumed internally, and potentially, by other systems.
The capacity to visualize the data over the charts and browse it with all the features built in to the charting system.
How Bots Use Data Mines
It is important to undestand that a Data Mine is a package of definitions. When users run a Bot, they do it inside a Task. This task needs to know how to run the bot. In order to find this out, the bot within the task must have a reference to a Process Definition exisiting at a Data Mine. When the bot is launched inside the task, at that moment, it receives all the information regarding the task that is being used to launch the bot, while at the same time it also receives the information regarding the Bot's definition from the Data Mine.
Important: Using both sides of this information (Task data and Bot definition) the Bot is able to do it's job.
Who Creates Data Mines?
Data Mines are created by power users and contributed to the project (the ones shipping with the software). Users might create Data Mines for their personal use and never share them. However, if they share them, they receive SA Tokens for that, and also for adapting and mantaining them, if necessary, for each new release.
Supported Bots
There are three types of bots that can hold their definitions at a Data Mine.
- Sensor Bots: used to fetch data from some Internet source by coding a specialized algorithm.
- API Data Fetcher Bots: used to fetch data from an Internet API without needing to code.
- Indicator Bots: used to process data.
Data Mine
Data-Mining->Node->Data Mine->Definition
A data mine is a hierarchy that contains definitions of sensor and indicators bots, and their plotters. These definitions are consumed from the actual code of such bots.
Data-Mining->Node->Data Mine->Content
In a way, data mines are catalogs of bots that anyone may create. This hierarchy provides all the tools required to create sensors, indicators, and plotters, with minimal coding required.
Because the hierarchy provides definitions, it does not intervene in the execution of bots. Instead, the definitions in data mines are instantiated from the network hierarchy by bot instances. Bot instances and process instances are the ones running the code according to the definitions in the corresponding data mine.
Sensor Bot
Data-Mining->Node->Sensor Bot->Definition
A sensor bot extracts raw data from external sources where a certain specific extracting algorithm needs to be coded, and stores it in a dataset that other bots may consume.
Indicator Bot
Data-Mining->Node->Indicator Bot->Definition
An indicator bot is an algorithm that processes information that other bots have generated, and produces elaborate datasets for others to consume.
Plotter
Foundations->Node->Plotter->Definition
A plotter is a device used to produce a graphic representation of datasets generated by bots.
Interdependency of Data Mines
- Data Dependency: This means that a Bot uses as an input the data produced by another bot.
- Status Dependency: This means that a Bot uses the status of another Bot as parameters for it's own processing.
Tip: The most common use case is when a Bot needs the date on which a market starts or ends. This information is automatically fetched from the Status Report of the bot that produces the raw data because it knows which data is the oldest and the newest at all times.
- Execution Dependency: This means that a Bot depends on other Bot execution to finish in order for it to start it's own execution.
Data Mining — TOC
You just read page 6 in the topic.
3. Data Mining - Dataset Types
4. Data Mining - Datasets Type Conversions
5. Data Mining - Indicator Frameworks
6. Data Mining - Data Mine Hierarchy
7. Data Mining - Managing Execution
8. Data Mining - Data Dependencies
9. Data Mining - Status Reports
10. Data Mining - Status Dependencies
14. Data Mining - Fetching Data From APIs
15. Data Mining - Consuming Data From Strategies
16. Data Mining - Indicator Bots
17. Data Mining - Indicator Bots - Process Definition
18. Data Mining - Indicator Bots - Product Definition
19. Data Mining - Indicator Bots - Data Mining Workflow
20. Data Mining - Indicator Bots - Coding the Indicator's Logic