February 14th 2022, Release Note: WFM UK | Labour Productivity: NextGen Forecasting

Summary

Release date: February 14th 2022 - April 25th 2022

NextGen Forecast Engine Activation

• Enabled by Default? - No
• Set up by customer Admin? - No
• Enable via Support ticket? - No
• Affects configuration or data? - Yes

What's Changing?

The team at Fourth is aware of how important accurate forecasts are to any business and have been investing in developing a next-generation system to provide the most accurate possible forecasts for our customers.

The NextGen Forecasting Engine is using state-of-the-art machine learning algorithms to accurately and quickly learn trends and patterns specific from the historical sales for each location to produce a location-specific forecast, down to 15-minute timeslots across a trading day.

The NextGen Forecast Engine replaces the legacy forecasting system and generates forecasts that are fed to the existing Labour Productivity dashboards that Fourth customers and users are familiar with.

Fourth has established a professional Data Science function and is investing in modern data infrastructure and state-of-the-art machine learning models. A team of data scientists, machine learning engineers and data engineers look after the new NextGen Forecast Engine and are constantly working to optimise, grow and sharpen the forecast accuracy.

 Between February 14th and April 25th 2022, the NextGen Forecast Engine will be activated across all Labour Productivity and Advanced Schedules customers who use Fourth's system forecast. We will be in touch with admin users to notify them of the exact date that each organisation will transition.

Reason for the Change?

Having an accurate estimate of future sales allows users to plan the labour, inventory, and intra-day workload in the most efficient way. Forecasting has a meaningful impact on profitability – labour management and inventory costs, limiting opportunity costs.

Powered by machine learning, Fourth's NextGen Forecast Engine harnesses cutting edge technology to empower users' in making better-informed spending plans and, ultimately, growing the bottom line.

Customers Affected

All Labour Productivity and Advanced Schedules customers using Fourth's system forecast.

Release Note Info/Steps

NextGen Forecasting Modelling Algorithm

Advances in the field of machine learning over the past few years have enabled businesses to leverage the data they’ve accumulated in making better-informed decisions about the future. 

The NextGen Forecast Engine is powered by machine learning, a branch of artificial intelligence where data sits at the centre of the solution. We are leveraging the latest developments and the use of algorithms that are powerful in automatically learning historical trends and patterns. These algorithms are routinely outperforming expert rules as well as simple time-series algorithms. 

The algorithms consider multiple years of data to detect and understand the most predictive factors and their importance for future sales. Multiple data points are created, and outliers and missing data are automatically handled by the engine.

A model is trained for each individual location, allowing the algorithms to be location-specific and acknowledge the differences between types of locations. A competition of multiple algorithms and machine learning models is run, and the best performing model is selected to produce the forecasts available in the system.

Each day, the engine generates forecasts for eight weeks in the future – from the current day +1 (tomorrow) to the current day +56 (56 days from today). These 56 forecasts are referred to as ‘offsets’, with the current 'day+1' being “offset 1”, the next one being “offset 2” and so forth until “offset 42”. 

The NextGen Forecast Process

 Sales Uploaded 

The first step in being able to generate forecasts is for sales data to be uploaded to the designated space as agreed with Labour Productivity upon onboarding. Fourth's batch processing pipelines work with the latest available data each morning. Fresh data yields more accurate forecasts.

 Data Processed and Forecasts Generated by Fourth 

A series of pipelines that get the required date are instantiated and run through a range of processing steps, each of which transforms and enriches pure sales data into a feature space including dozens of dimensions. A trained model then generates forecasts based on those features. 

 Forecasts Made Available in Labour Productivity

Once the forecasts have been generated, they are fed to the Labour Productivity dashboards where users can start extracting value from them. 

 Monitoring - Automated Alerting and Daily Human Monitoring

The team at Fourth have set up several monitoring mechanisms, including alerts that trigger when strange numbers appear, as well as having a human or two keep an eye on metrics and charts daily.

Data Features

All historical sales data for each location is considered. The algorithm automatically detects the most important patterns (be they weekly, seasonal, daily trends) which are most predictive of future sales. Multi-year history of sales data is also considered.

Exceptionally high or low sales data, so-called 'outliers' are being handled by the engine to understand the impacts.

The system uses a range of intrinsic and external data to generate the feature space that the models are trained and predicted on. These include:

• Intrinsic: historic lookups, historic error values, rolling stats, etc. 
• External: public holidays data, school holidays, Covid indicators

Feature generation is the primary focus for improving the forecasts, so Fourth constantly experiment with adding/changing features to improve performance.

 While we are constantly improving the algorithms and data sources, the accuracy of the algorithm can be impacted when external factors influencing the forecast haven’t been available to the engine, or in a situation of highly volatile sales, when it is difficult to detect the signal from the noise.

Monitoring of NextGen Forecasting 

The forecasting report most closely looked at is generated once the forecasts have been fed to Labour Productivity as a final step of the pipeline. The report extracts key metrics on forecasts generated for upcoming days, as well as the performance of forecasts over past days. It also highlights outliers amongst the locations and/or offsets.

Fourth uses the forecasting report as a heartbeat for the whole engine, as it holds, in a single interface, measures of code execution, freshness, completeness and sanity of both incoming sales data and outgoing forecasts.

Measuring How Well the Forecasting Engine Performs 

Measuring the performance of any predictive system requires a range of metrics. For the NextGen Forecast Engine, Fourth decided to rely on the WAPE (Weighted Average Percentage Error) metric. 

See the Appendix for details about what WAPE is, and a few case studies. 

Future improvements 

Fourth is constantly working on the algorithms and adding new data sources to improve the accuracy of the forecasting engine even further. First-hand experience from businesses is critical for building a highly accurate forecasting engine. Any feedback (such as cultural and sporting events, road closures, etc.) relevant for specific locations is welcomed and will be reviewed and considered.

FAQ

 What’s the difference between the old and the new forecasting systems?

The legacy forecasting system was set up about seven years ago, with the same goals as the new one – providing information to enable better-informed planning for Fourth customers. In terms of implementation, however, it was much more straightforward – simply stacking heuristics in a test ANOVA-style solution. 

In contrast, the NextGen Forecast Engine allows for much more sophistication via machine learning. 

 How does the NextGen forecast compare with the old one in terms of forecast accuracy? 

As an example, when conducting analysis over December 2021 the WAPE when testing on live client locations using the NextGen Forecast Engine was 17.39% compared with 22.86% from the current system forecast - this represents close to a 25% improvement in forecast accuracy! 

 Why are my forecasts bad? 

If a forecast value looks out of place, it doesn’t necessarily mean something’s broken, but it’s most likely a problem with the input data. The first thing to check would be the most recently uploaded sales data.  

 How can I help improve my forecasts? 

The team is constantly looking at improving the features that models operate on. One crucial feature category that would almost certainly help with forecasts is structured input from customers. For example, upcoming events or other notable information being flagged by managers of locations.

 Can I revert to/stay on the previous system? 

Yes, for now. While the legacy system will not be immediately shut down, Fourth will cease maintaining it soon and all customers will eventually be migrated to the NextGen Forecast Engine. Most importantly, the new engine allows for accuracy improvements, quality upgrades and scale-up. 

 Why are my forecasts not updated frequently/on time? 

Fourth's batch processing pipelines run on the most up to date data each morning. If the same data from the previous day is seen, the predictions will hardly be different than those from yesterday. For example, if updated sales data is not received in a timely manner.

Appendix

 What is WAPE? 

The Weighted Average Percentage Error (WAPE) is a metric of the accuracy of a predictive model that is particularly well suited for the hospitality industry. To understand WAPE, let’s first see MAPE, or the Mean Absolute Percentage Error.

Notably, both percentage error metrics mean that the lower the metric, the better the forecasts.

MAPE = the percentage error of the forecast in relation to the actual values: 

The table above shows that the MAPE, while not wrong, is not particularly useful for a business with widely differing sales across locations – just a single USD of error (on B) significantly inflated the overall metric. 

On the other hand, WAPE = the percentage error, weighted by the actual value 

The table above shows how WAPE is a stronger metric across a diverse business, as it not only calculates the error, but also the significance of the error in the wider context, i.e. the whole business.

 Why WAPE? 

The above tables show how WAPE is a well-rounded metric for a business with multiple locations (note – for a single location, WAPE and MAPE are equal). Still, we sometimes also employ other metrics to view the data from different angles, like: 

• MAPE, which allows us to weigh each location equally 
• Mean Absolute Error (MAE), which looks at the absolute value (not percentage) error 
• Mean Squared Error (MSE), which squares the value (not percentage) error, making it more sensitive to outliers

 Examples

Given the above tables as templates, let’s look at a few case studies on what the WAPE would look like in various scenarios.

Normal sales, normal forecasts: 

Conclusion – the better the forecasts, the lower the WAPE. 

Sales normal, forecasts approaching £0 (perhaps due to data processing error): 

Conclusion – when forecasts approach £0, WAPE approaches 100%. 

Sales approaching £0 (perhaps due to closure), forecasts normal: 

Conclusion – when sales approach £0, WAPE approaches infinity.