Available translations

Data Mining - Indicator Bots - Product Definition

foundations.png
Summary: The product definition is the compilation of configurations, references, and JavaScript code that make up the data product.
Foundations->Node->Product Definition->Definition
product-definition.png
The product definition node holds all definitions that make up a product, including how the product is calculated and the datasets the product is comprised of.
Foundations->Node->Product Definition->Content
A bot may feature multiple products, thus, there may be multiple product definitions. Three main aspects need to be defined in terms of products: the characteristics of the dataset, the composition of each record, and the way records are calculated.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Dataset Definition
Foundations->Node->Dataset Definition->Definition
dataset-definition.png
A data set is a collection of files stored in the designated location, as per the definitions on the referenced data product in the corresponding data mine.
Foundations->Node->Dataset Definition->Content
A good part of what makes datasets easy to consume by other bots is the fact that they are standardized in terms of their structure. It is that structure that is specified in the dataset definitions.
Record Definition
Foundations->Node->Record Definition->Definition
record-definition.png
The record definition is where indicators define how many properties, what properties and in which order those properties will be made available in each record.
Foundations->Node->Record Definition->Content
Bots store data in the form of arrays of records, in a minimized plain text file and the standard JSON format, although not as objects with named properties, but as arrays.
The choice of plain text for storage of large volumes of information has benefits in terms of not requiring any sophisticated technology to store and serve the data (i.e.: no database is required).
Best practice indicates that there needs to be a balance between the amount of data that is stored and calculations that may be performed at runtime.
Storing objects with named properties in the JSON format would be inappropriate as labels would repeat over and over, creating a ridiculous amount of unnecessary information. However, storing arrays in the JSON standard facilitates the handling of files in the context of JavaScript and Node.JS.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Record Property
Foundations->Node->Record Property->Definition
record-property.png
A record property is a piece of information that may be stored as part of the dataset or made available as a calculated property.
Foundations->Node->Record Property->Content
Foundations->Node->Record Definition->Record Properties
What are the Record Properties used for?
In general this is the mechanism to define what information is going to be available to the consumers of data of this product.
When properties don't have a Formula, they define the format of the file that stores this data on disk. These record properties defines, the data type (string, number, etc), and in which order the data is saved. At the file there are no headers, just the data, because the header information is available here, for any process writing or reading the data from disk.
The formulas nodes there are used to expand the record definition adding fields that are not actually going to be saved on disk, to minimize space used, and they are going to be calculated at runtime. This allows processes that consume this data to have ell the fields available at runtime even though some were not actually saved on disk.
Records may have as many properties as required.
Record Formula
Foundations->Node->Record Formula->Definition
record-formula.png
In the context of a record definition, formulas are used to assign a value to the property, usually in the form of a variable declared in the calculation or data building procedures.
Calculations Procedure
Foundations->Node->Calculations Procedure->Definition
calculations-procedure.png
The calculations procedure processes information to be fed to calculated properties. Calculated properties are not stored in the data set; instead, they are calculated in real-time as the data is consumed by others.
Foundations->Node->Calculations Procedure->Content
Bots store data in the minimized array format as a strategy to build the lightest possible datasets. For the same reason, a bot may choose not to store easily calculable information.
For example, imagine an indicator that stores buy and sell volumes... does it need to store the total volume? Given that it is so easy to calculate, the answer is probably not. It is up to the developer to decide, but it is certainly not required.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Procedure Initialization
Foundations->Node->Procedure Initialization->Definition
procedure-initialization.png
The procedure initialization is used to initialize variables used by the procedure loop.
Procedure Loop
Foundations->Node->Procedure Loop->Definition
procedure-loop.png
The procedure loop holds the JavaScript code that reads an input, processes the information, and generates an output recursively.
Procedure Javascript Code
Foundations->Node->Procedure Javascript Code->Definition
procedure-javascript-code.png
The Procedure JavaScript code node may hold any snippet of valid JavaScript.
Data Building Procedure
Foundations->Node->Data Building Procedure->Definition
data-building-procedure.png
The data building procedure processes information to be fed to properties that are stored in the output dataset.
Foundations->Node->Data Building Procedure->Content
This procedure holds the core logic of the bot, as it features the code that performs the calculations required to produce the new data set. The code is split between a procedure initialization and a procedure loop.
Foundations->Concept->Reusable Snippets->Note for Hierarchy Tables
Previous
Data Mining - Indicator Bots - Process Definition
Next
Data Mining - Indicator Bots - Data Mining Workflow