This writeup describes the approach and steps taken to integrate SAP (IBP) with the Google Cloud Platform.
To establish a global BI platform on GCP to bring supply chain KPI visibility to one platform and phase out APO.
- The IT team has recently moved to cloud-based supply chain analytics by adopting the SAP IBP solution
- For the Global BI project, the BI platform that is built aids in generating reports on demand forecast, statistical forecast, lag, and historical IBP data to prepare the KPI in Tableau
- The BI platform should support future requirements of Big Data, Machine Learning, IoT, and other solutions
- Real-time Integration of GCP with IBP for supply chain analytics and visualization
- Phasing out legacy applications such as APO and adopting the new methodology for supply chain analytics
SpringML worked with the Client to understand the challenges. The experts at SpringML proposed the tool and method selection for IBP to GCP integration. SpringML designed and implemented a process framework and approach for the Integration of massive data in real-time utilizing the IBP API.
- Choosing the right tool (IBP API as opposed to CPIDS) for massive integration of millions of records
- Capacity building and assessment for network and SAP IBP
- Massive reusability (> 30%) across key figures to be implemented
- Master and Transaction data
- Real-time integration with high performance
- Step-by-step guidance through the Integration and Orchestration
- The Comprehensive data loading sequence
- Data integration workshops
- Manage testing cycles
- Accelerated Data Validation
- Certificate-based secure data communication
The client realized several benefits from this SpringML-delivered solution, including:
- Standard GCP functionality across the business
- Improved reporting globally
- Reduced the ongoing platform costs and infrastructure investment
- Saves considerable time versus manual processes
- Data needed by the user is available in the new system in a complete, accurate, and real-time manner
- Bespoke design architecture tremendously reduced build time and time to realize objectives
What is IBP
SAP Integrated Business Planning for Supply Chain (SAP IBP) is powered by SAP HANA in-memory technology. This cloud-based intelligent solution combines sales and operations planning (S&OP), forecasting and demand, supply, demand-driven replenishment, and inventory planning. Take the advantage of powerful supply chain analytics, what-if simulations, alerts, and more to stay ahead of change while improving responsiveness.
For more information on IBP, refer to the link SAP IBP.
IBP and GCP Integration
IBP exposes an OData API to source data from its system.
What is OData API
OData is an API implemented as a REST API and the technical protocol is Open Data Protocol (OData). This means that you can utilize standard HTTP methods (for example, the GET method) to call the API.
For more information on OData, view the link OData-API.
Designing a Pipeline in Google Cloud
Google Cloud will send API requests to IBP using the exposed endpoint to receive data in JSON format. Once we receive the payload, it will be stored in Google Storage. Later, it will be processed or mapped and stored in the BigQuery tables.
We are going to leverage the Google cloud platform services like Composer, Data fusion, GCS, BigQuery to integrate SAP IBP.
Cloud composer will initialize the process by deploying and starting a data fusion pipeline. The fusion pipeline receives the data from IBP Odata API in JSON format. The JSON attributes are flattened, mapped with the respective columns, and stored in the GCS buckets or BigQuery tables by applying the required transformations.
Composer is used for the orchestration of pipelines. We create DAG’s in the composer to generate fusion templates dynamically. This takes care of deploying and starting templates related to each key figure and BigQuey/GCS loads.
Data fusion Pipeline
A separate data fusion pipeline is developed for every individual key figure in IBP. These pipelines are scheduled using cloud composer.
For more information about Key Figure, refer to the IBP Key Figure link.
Pipeline Components :
HTTP Plugin :
- Configure HTTP plugin to fetch data from IBP Odata API by sending API requests to the GCP
- Use batch paginations to fetch data in 5k or 10k records in a batch
- Data will be in JSON format
- We can leverage wrangler, Java scripts, and few other plugin options to apply required transformations, formatting while creating standard output schema on top of data
- Creating or Deriving a new column
- Flattening JSON attributes
GCS/ BigQuery :
Store data in GCS buckets or BigQuery.
Whenever we are getting an error, the system will collect the error messages. These messages are stored in the BigQuery table for further analysis and tracking.
Data fusion leverages data proc clusters to run all the pipelines. By default, we use the Spark framework to run our pipeline. Data fusion dynamically creates data proc clusters with the default configuration and kills the clusters once the job is completed. We have an option to configure existing data proc clusters to run fusion jobs.
Autoscaling Policy: We can utilize autoscaling policy in data proc by setting the auto-scaling policy that helps us to scale the cluster capacity as per the requirement. This can be applied only if we are going to use the existing Data Proc cluster.
We create a raw table that includes all the records for a particular key figure. These raw tables are used to identify unique records. We join the multiple raw tables to create a refined table.
IBP_Raw_tables acts as a staging layer, where it holds the data for each key figure in a separate table. It is more like replicating on a table-by-table basis.
We can leverage materialized views in BigQuery. Materialized views are pre-computed views that periodically cache the results of a query to provide customers increased efficiency. BigQuery uses pre-computed results from materialized views and, whenever possible, reads only the delta changes from the base table to compute up-to-date results quickly. Materialized views can be queried directly or can be used by the BigQuery optimizer to process queries to the base table. We can create materialized views that join multiple key figure tables, and create a final version of the refined base table.
The data in the refined table can be sourced by the business users using a visualization tool for further analysis.
The integration with Google Cloud Platform helps SAP IBP to leverage the GCP infrastructure for better performance and analysis.