In order to get out-of-sync data aligned again, it is possible to implement a sync service which compares the data in the ETL server against the data in SAP. If differences are found, the ETL key of changed/created/deleted objects will be stored into the ETL log table. With the next getList / getData cycle, the different data will be aligned properly.
Note: the request- response examples are formatted with field names in uppercase, however the webservice could also be customized to use lower case or camelcase formatted names.
Please refer also the ETL concept page.
The function module /EPO1/ETL_SYNC is configured to be accessible via webservice (see EPO customizing).
Import parameters:
For a small set of object, the sync-service may be invoked by one single call - containing the complete list of ETL-key/timestamp combinations. However, for a large data set, the data transfer might be split into several webservice calls, transferring a smaller block of data to SAP.
The data processing will be started after receiving the ivLastBlock - flag set to 'X' in the background.
Depending on the ETL object (see table /EPO1/ETLOBJ_IN), a special function module will be called, which collects all the available information from SAP (containing the ETL-key and the timestamp of the last change).
Afterwards, both collected data (from the ETL-server and from SAP) are compared. Differences will be written into the /EPO1/ETLLOG - table, such that the differences can be equalized during the next getList/getData processing.
Please note, that a 'first data preparation' is also possible, when the ETL-server has no data for a given ETL-object/prefix; just send an empty itObjectList, resulting in a completely filled /EPO1/ETLLOG - table.
Please note, that for performance reasons, the SYNC service should not be called too often.
Example: start the sync data transfer for assets (URL: .../epo1soa/jsonhandler/epoetl/sync/ETLAsset/1000)
{
"itObjectList": [
{
"etlKey": "10000000000000020000",
"timestamp": "20240101120000"
},
{
"etlKey": "10000000000000020001",
"timestamp": "20240101120100"
},
...
],
"ivFirstBlock":"X" // indicating the begin of the data transfer
}
continuing the data transfer
{
"itObjectList": [
{
"etlKey": "10000000000000020050",
"timestamp": "20240101120130"
},
{
"etlKey": "10000000000000020051",
"timestamp": "20240101120200"
},
...
]
}
ending the data transfer
{
"itObjectList": [
{
"etlKey": "10000000000000020098",
"timestamp": "20231011150248"
},
{
"etlKey": "10000000000000020099",
"timestamp": "20231011150125"
},
...
],
"ivLastBlock":"X" // indicating the end of the data transfer
}
complete data transmission in one single request (for small data sets):
{
"itObjectList": [
{
"etlKey": "10000000000000020001",
"timestamp": "20240101120000"
},
{
"etlKey": "10000000000000020002",
"timestamp": "20240101120100"
},
{
"etlKey": "10000000000000020003",
"timestamp": "20240101120110"
}
],
"ivFirstBlock":"X" // indicating the begin of the data transfer
"ivLastBlock":"X" // indicating the end of the data transfer
}
Instead of appending the ETL object and the prefix to the URL, they could also be passed as request - fields. Please note, that the URL path - values have a higher priority than the request-values.
Original URL:
https://server.path.or.name/epo1soa/jsonhandler/epoetl/sync/ETLKostl/1000
Alternative URL:
https://server.path.or.name/epo1soa/jsonhandler/epoetl/sync
Example: start the sync data transfer for assets with ETL-object- and prefix-values in the reqeust
{
"ivEtlObject": "ETLAsset",
"ivPrefix: "1000",
"itObjectList": [
{
"etlKey": "10000000000000020000",
"timestamp": "20240101120000"
},
{
"etlKey": "10000000000000020001",
"timestamp": "20240101120100"
},
...
],
"ivFirstBlock":"X" // indicating the begin of the data transfer
}
The current SYNC-status (idle, data collection, data processing), timestamp and error message of the last SYNC-run and a small statistic of created/changed/deleted objects is stored in table /EPO1/ETLSYNC_ST and is returned as part of the getList response.
Status 'DATA COLLECTING'
After starting the data transfer with 'FIRST_BLOCK', the combination ETL-object/prefix is marked as 'collecting' in the table /EPO1/ETLSYNC_ST.
Status 'DATA PROCESSING'
After finalizing the data transfer with 'LAST BLOCK', the combination ETL-object/prefix is marked as 'processing' in the table /EPO1/ETLSYNC_ST. Now, the data processing starts (and might run a while, depending on the number of records to be compared).
Status '' (empty / no status)
There is currently no data collecting or data processing active; if a sync has been invoked, it has finished and the data is ready to be processed with getList /getData.
Example - response for getList (URL: …/epo1soa/jsonhandler/epoetl/getList/ETLAsset/1000) after starting the sync with 'FIRST_BLOCK':
{
"epo1ok": "X",
"epo1exp": {
"etlQty": 1234,
"hasMoreResults": false,
"syncTimestamp": "20240201100018",
"syncStatistic": {
"syncTimestamp": "20240214100018",
"status": "DATA COLLECTING" // still waiting for the flag LAST_BLOCK
},
"etlKeyT": [
"10000000000000020001",
"10000000000000020002"
...
]
}
}
After receiving the LAST_BLOCK, the data processing (comparision of external / internal data) will start in a background process
{
"epo1ok": "X",
"epo1exp": {
"etlQty": 1234,
"hasMoreResults": false,
"syncTimestamp": "20240214100018",
"syncStatistic": {
"syncTimestamp": "20240214100018",
"status": "DATA PROCESSING" // processing data, filling the ETL log table
},
"etlKeyT": [
"10000000000000020001",
"10000000000000020002"
...
]
}
}
After finishing the data processing, the STATUS will be empty / missing; now, the data in the ETL log table will be complete:
{
"epo1ok": "X",
"epo1exp": {
"etlQty": 1234,
"hasMoreResults": false,
"syncTimestamp": "20240214100018",
"syncStatistic": {
"syncTimestamp": "20240214100018",
"createdObjects": 21,
"updatedObjects: 2,
"deletedObjects: 3
},
"etlKeyT": [
"10000000000000020001",
"10000000000000020002",
...
]
}
}
CREATED_OBECTS:
number of objects, which are present in SAP but has not been supplied by the SYNC - data
UPDATED_OBJECTS:
number of objects, which has been updated in SAP after the timestamp provided by the SYNC - data
DELETED_OBJECTS:
number of objects, which are present in the SYNC - data, but are missing or marked as deleted in SAP