Available translations

Data Mining - Indicator Bots - Process Definition

foundations.png
Summary: The process definition is the compilation of configurations, dependencies, and references that make up a process.
Foundations->Node->Process Definition->Definition
process-definition.png
The process definition node groups all definitions required for a process to function. The Multi-Time-Frame-Market Process deals with time frames of one hour and above. The Multi-Time-Frame-Daily Process deals with time frames of 45 minutes and below.
Foundations->Node->Process Definition->Content
As hinted above, most bots—in particular indicators—have two different processes. The reason is that different data structures need to be handled in different manners. The Multi-Time-Frame-Daily process handles daily files, while the Multi-Time-Frame-Market process handles market files.
The Multi-Time-Frame-Market process deals with time frames of one hour and above. Because these time frames produce relatively small numbers of records, the process builds one single file per time frame spanning the whole market history—hence the name Multi-Time-Frame-Market.
On the other hand, the Multi-Time-Frame-Daily process deals with time frames below one hour. These time frames produce huge numbers of records, therefore, the corresponding data must be fragmented in multiple files. The Multi-Time-Frame-Daily process builds one file per day for each time frame—hence the name Multi-Time-Frame-Daily.
Properties
  • codeName is the name of the process as used within the code of the system.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Process Output
Foundations->Node->Process Output->Definition
process-output.png
The process output groups the definitions of which datasets are impacted by the process, that is, which datasets the process builds or takes a part in building.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Output Dataset Folder
Foundations->Node->Output Dataset Folder->Definition
output-dataset-folder.png
An output dataset folder is an organizational device used to create arrangements of output datasets, particularly useful when the bot has many products.
Foundations->Node->Output Dataset Folder->Content
In cases in which a single bot has many different products, output dataset folders may help organize the outputs referencing each product, making their management easier. Folders may be nested like folders in the file system.
The use of output dataset folders is optional, as product definitions may also exist outside of folders.
Output Dataset
Foundations->Node->Output Dataset->Definition
output-dataset.png
The output dataset is a reference to a dataset definition. By establishing such reference, the process acquires the definitions as of how the dataset is to be constructed.
Foundations->Node->Output Dataset->Content
There are other effects of establishing a reference from the output dataset to a product dataset definition. Upon execution, every time a process finishes a processing cycle, it triggers an event that may be consumed by other entities. This event indicates that the datasets impacted by the process have been updated.
An example of other entities that may be listening to such events is that of plotters. Plotters read datasets and create graphical representations of this data over the charts. Charts are constantly updating the information in the form of candles and indicators in realtime, synchronized with the data being extracted from the exchange by the sensor bot. That kind of automatism is possible thanks to the events that processes trigger every time an execution cycle is finished, signaling to everyone listening that new data is available on each of the impacted datasets.
Process Dependencies
Foundations->Node->Process Dependencies->Definition
process-dependencies.png
Process dependencies are references to various data structures on which the process depends to function.
Foundations->Node->Process Dependencies->Content
While processes run autonomously, most processes participate in a value-adding chain by which a process produces a data product that other processes may consume as an input to be processed further. This means that bots—while autonomous in their particular jobs—do depend both on other bots and on the data other bots produce.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Status Dependency
Foundations->Node->Status Dependency->Definition
status-dependency.png
Status dependencies are references to a status report that define which process the process establishing the reference depends on.
Foundations->Node->Status Dependency->Content
The reference is established to acquire the information relative to what the target process is doing. For example, by reading a status report a process may learn when was the last time the referenced process ran, and what was the last file processed.
The status report referenced may belong to the same process— which is called a self-reference. In such a case, the process is learning what it did the last time it ran. Also, the status report referenced may belong to another process—another bot. In that case, the dependency may be of the Market Starting Point or Market Ending Point types.
  • Self Reference is mandatory, as a process needs to read its own status report every time it wakes up.
  • The market starting point is a status dependency existing on Multi-Time-Frame-Daily processes so that the process establishing the reference learns the datetime of the start of the market. Usually, the reference is established with the sensor's Historic-OHLCVs process status report. Multi-Time-Frame-Market processes do not have this type of status dependency as the date of the start of the market is implied in their dataset (a single file with all market data).
Data Dependency
Foundations->Node->Data Dependency->Definition
data-dependency.png
Data dependencies are references established with dataset definitions of other bots, determining which datasets the process establishing the reference uses as input.
Foundations->Node->Data Dependency->Content
Most bots consume data other bots have produced. Because bots need the data as input for their calculations, processes establish a data dependency with the dataset definitions of other bots. The reference provides the process with all the information needed to decode the dataset, enabling it to perform the required calculations.
Data Mine Data Dependencies
Foundations->Node->Data Mine Data Dependencies->Definition
data-mine-data-dependencies.png
Data mine data dependencies are references established with entire data mines to facilitate establishing data dependencies with multiple datasets in the given data mine.
Foundations->Node->Data Mine Data Dependencies->Content
The node may be used as an organizational device, simply to arrange bot data dependencies. However, the smart use of the node involves automating the deployment of multiple data dependencies and their references.
Bot Data Dependencies
Foundations->Node->Bot Data Dependencies->Definition
bot-data-dependencies.png
A bot data dependencies node is an organizational device used to arrange data dependencies corresponding to a specific bot.
Foundations->Node->Bot Data Dependencies->Content
The device exists as an offspring of a data mine data dependencies node, and does not require a reference to a bot in the given data mine.
Data Dependency Folder
Foundations->Node->Data Dependency Folder->Definition
data-dependency-folder.png
A data dependency folder node is an organizational device used to map the arrangement of product definition folders of a given bot.
Foundations->Node->Data Dependency Folder->Content
The use of product data dependency folders is optional, as data dependencies may also exist outside of folders.
Status Report
Foundations->Node->Status Report->Definition
status-report.png
A status report serves as a temporal annotation that bots read every time they run to know what was done in the previous cycle and what the state of affairs is at present. They are dynamic and change constantly, with updates after every cycle of the associated process.
Foundations->Node->Status Report->Content
Bots do not run continuously. Instead, they run in cycles. A cycle usually lasts until there is no more data to process, and once they finish, they shut down until the next cycle is due. A status report is a file every bot writes at the end of each cycle with information about the last run, including the datetime of the run and the last record processed.
A status report may be consumed by the same bot producing it, or by other bots.
Execution Started Event
Foundations->Node->Execution Started Event->Definition
execution-started-event.png
The execution started event is the event that triggers the execution of a process. It usually references the execution finished event of another process on which the process depends on.
Foundations->Node->Execution Started Event->Content
These references determine when a process is due for another run. By listening to the execution finished event of the process it depends on, it may wake up just in time to process the new batch of data the dependency has just delivered.
Bots form a sort of multi-branched execution sequence with an indeterminate number of dependencies. Every time the bot further down the tree of dependencies finishes a cycle, it triggers the execution of multiple bots listening to its execution finished event.
In the context of a trading process instance running a trading session on the network hierarchy, the execution started event may be used to force the trading process to run only after the last indicator bot dependency finishes its job. This guarantees that all dependencies are up to date and that the trading bot will evaluate the information corresponding to the same candles for all indicators used by the trading system.
Not setting up this event on a trading session may result in eventual data inconsistencies, as—in theory—the trading bot may run with some indicators up to date and some slightly delayed.
Execution Finished Event
Foundations->Node->Execution Finished Event->Definition
execution-finished-event.png
The execution finished event is the event that processes trigger once they have finished an execution cycle. The event is broadcasted to whoever wants to listen, so that other bots may know when the process has finished its execution cycle.
Foundations->Node->Execution Finished Event->Content
The execution finished event is responsible for triggering the execution of every process that depends on the data a bot produces. If bot Alice depends on bot Bob, Alice listens to the execution finished event of Bob so that it may start a new execution cycle as soon as Bob finishes its cycle. Alice listens to Bob's execution finished event by establishing a reference from its execution started event.
Previous
Data Mining - Indicator Bots
Next
Data Mining - Indicator Bots - Product Definition