Time series implementation details

These implementation details describe the stages and processing that are specific to an AutoAI time series experiment.

Implementation details

Refer to these implementation and configuration details for your time series experiment.

Time series stages for processing an experiment.
Time series optimizing metrics for tuning your pipelines.
Time series algorithms for building the pipelines.
Supported date and time formats.

Time series stages

An AutoAI time series experiment includes these stages when an experiment runs:

Initialization
Pipeline selection
Model evaluation
Final pipeline generation
Backtest

Stage 1: Initialization

The initialization stage processes the training data, in this sequence:

Load the data
Split the data set L into training data T and holdout data H
Set the validation, timestamp column handling, and lookback window generation. Notes:
- The training data (T) is equal to the data set (L) minus the holdout (H). When you configure the experiment, you can adjust the size of the holdout data. By default, the size of the holdout data is 20 steps.
- You can optionally specify the timestamp column.
- By default, a lookback window is generated automatically by detecting the seasonal period by using signal processing method. However, if you have an idea of an appropriate lookback window, you can specify the value directly.

Stage 2: Pipeline selection

The pipeline selection step uses an efficient method called T-Daub (Time Series Data Allocation Using Upper Bounds). The method selects pipelines by allocating more training data to the most promising pipelines, while allocating less training data to unpromising pipelines. In this way, not all pipelines see the complete set of data, and the selection process is typically faster. The following steps describe the process overview:

All pipelines are sequentially allocated several small subsets of training data. The latest data is allocated first.
Each pipeline is trained on every allocated subset of training data and evaluated with testing data (holdout data).
A linear regression model is applied to each pipeline by using the data set described in the previous step.
The accuracy score of the pipeline is projected on the entire training data set. This method results in a data set containing the accuracy and size of allocated data for each pipeline.
The best pipeline is selected according to the projected accuracy and allotted rank 1.
More data is allocated to the best pipeline. Then, the projected accuracy is updated for the other pipelines.
The prior two steps are repeated until the top N pipelines are trained on all the data.

Stage 3: Model evaluation

In this step, the winning pipelines N are retrained on the entire training data set T. Further, they are evaluated with the holdout data H.

Stage 4: Final pipeline generation

In this step, the winning pipelines are retrained on the entire data set (L) and generated as the final pipelines.

As the retraining of each pipeline completes, the pipeline is posted to the leaderboard. You can select to inspect the pipeline details or save the pipeline as a model.

Stage 5: Backtest

In the final step, the winning pipelines are retrained and evaluated by using the backtest method. The following steps describe the backtest method:

The training data length is determined based on the number of backtests, gap length, and holdout size. To learn more about these parameters, see Building a time series experiment.
Starting from the oldest data, the experiment is trained by using the training data.
Further, the experiment is evaluated on the first validation data set. If the gap length is non-zero, any data in the gap is skipped over.
The training data window is advanced by increasing the holdout size and gap length to form a new training set.
A fresh experiment is trained with this new data and evaluated with the next validation data set.
The prior two steps are repeated for the remaining backtesting periods.

Time series optimization metrics

Accept the default metric, or choose a metric to optimize for your experiment.

Metric	Description
Symmetric Mean Absolute Percentage Error (SMAPE)	At each fitted point, the absolute difference between actual value and predicted value is divided by half the sum of absolute actual value and predicted value. Then, the average is calculated for all such values across all the fitted points.
Mean Absolute Error (MAE)	Average of absolute differences between the actual values and predicted values.
Root Mean Squared Error (RMSE)	Square root of the mean of the squared differences between the actual values and predicted values.
R²	Measure of how the model performance compares to the baseline model, or mean model. The R² must be equal or less than 1. Negative R² value means that the model under consideration is worse than the mean model. Zero R² value means that the model under consideration is as good or bad as the mean model. Positive R² value means that the model under consideration is better than the mean model.

Reviewing the metrics for an experiment

When you view the results for a time series experiment, you see the values for metrics used to train the experiment in the pipeline leaderboard:

Reviewing experiment results

You can see that the accuracy measures for time-series experiments may vary widely, depending on the experiment data evaluated.

Validation is the score calculated on training data.
Holdout is the score calculated on the reserved holdout data.
Backtest is the mean score from all backtests scores.

Time series algorithms

These algorithms are available for your time series experiment. You can use the algorithms that are selected by default, or you can configure your experiment to include or exclude specific algorithms.

Algorithm	Description
ARIMA	Autoregressive Integrated Moving Average (ARIMA) model is a typical time series model, which can transform non-stationary data to stationary data through differencing, and then forecast the next value by using the past values, including the lagged values and lagged forecast errors
BATS	The BATS algorithm combines Box-Cox Transformation, ARMA residuals, Trend, and Seasonality factors to forecast future values.
Ensembler	Ensembler combines multiple forecast methods to overcome accuracy of simple prediction and to avoid possible overfit.
Holt-Winters	Uses triple exponential smoothing to forecast data points in a series, if the series is repetitive over time (seasonal). Two types of Holt-Winters models are provided: additive Holt-Winters, and multiplicative Holt-Winters
Random Forest	Tree-based regression model where each tree in the ensemble is built from a sample that is drawn with replacement (for example, a bootstrap sample) from the training set.
Support Vector Machine (SVM)	SVMs are a type of machine learning models that can be used for both regression and classification. SVMs use a hyperplane to divide the data into separate classes.
Linear regression	Builds a linear relationship between time series variable and the date/time or time index with residuals that follow the AR process.

Supported date and time formats

The date/time formats supported in time series experiments are based on the definitions that are provided by dateutil.

Supported date formats are:

Common:

    YYYY
    YYYY-MM, YYYY/MM, or YYYYMM
    YYYY-MM-DD or YYYYMMDD
    mm/dd/yyyy
    mm-dd-yyyy
    JAN YYYY

Uncommon:

    YYYY-Www or YYYYWww - ISO week (day defaults to 0)
    YYYY-Www-D or YYYYWwwD - ISO week and day

Numberng for the ISO week and day values follows the same logic as datetime.date.isocalendar().

Supported time formats are:

    hh
    hh:mm or hhmm
    hh:mm:ss or hhmmss
    hh:mm:ss.ssssss (Up to 6 sub-second digits)
    dd-MMM 
    yyyy/mm

Notes:

Midnight can be represented as 00:00 or 24:00. The decimal separator can be either a period or a comma.
Dates can be submitted as strings, with double quotation marks, such as "1958-01-16".

Supporting features

Supporting features, also known as exogenous features, are input features that can influence the prediction target. You can use supporting features to include additional columns from your data set to improve the prediction and increase your model’s accuracy. For example, in a time series experiment to predict prices over time, a supporting feature might be data on sales and promotions. Or, in a model that forecasts energy consumption, including daily temperature makes the forecast more accurate.

Algorithms and pipelines that use Supporting features

Only a subset of algorithms allow supporting features. For example, Holt-winters and BATS do not support the use of supporting features. Algorithms that do not support supporting features ignore your selection for supporting features when you run the experiment.

Some algorithms use supporting features for certain variations of the algorithm, but not for others. For example, you can generate two different pipelines with the Random Forest algorithm, RandomForestRegressor and ExogenousRandomForestRegressor. The ExogenousRandomForestRegressor variation provides support for supporting features, whereas RandomForestRegressor does not.

This table details whether an algorithm provides support for Supporting features in a time series experiment:

Algorithm	Pipeline	Provide support for Supporting features
Random forest	RandomForestRegressor	No
Random forest	ExogenousRandomForestRegressor	Yes
SVM	SVM	No
SVM	ExogenousSVM	Yes
Ensembler	LocalizedFlattenEnsembler	Yes
Ensembler	DifferenceFlattenEnsembler	No
Ensembler	FlattenEnsembler	No
Ensembler	ExogenousLocalizedFlattenEnsembler	Yes
Ensembler	ExogenousDifferenceFlattenEnsembler	Yes
Ensembler	ExogenousFlattenEnsembler	Yes
Regression	MT2RForecaster	No
Regression	ExogenousMT2RForecaster	Yes
Holt-winters	HoltWinterAdditive	No
Holt-winters	HoltWinterMultiplicative	No
BATS	BATS	No
ARIMA	ARIMA	No
ARIMA	ARIMAX	Yes
ARIMA	ARIMAX_RSAR	Yes
ARIMA	ARIMAX_PALR	Yes
ARIMA	ARIMAX_RAR	Yes
ARIMA	ARIMAX_DMLR	Yes

Learn more

Scoring a time series model

Parent topic: Building a time series experiment