xGen Data Sync is a data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use.

The Data Sync process is designed to simplify the source extraction process and reduce the time and effort required to move data from source system to the Data Lake and Data Warehouse. By automating the pipeline creation process, users can avoid manual errors and inconsistencies and ensure that their data is accurately and consistently processed and stored.

xGen Data Sync offers automatic metadata generation and table structure creation to ensure high-quality data and minimize errors. When creating a Data Sync, xGen automatically generates metadata based on the source object, and maps it to the destination Data Warehouse. This metadata includes information on the data types, lengths, and formats of the source field. Using this metadata, xGen then automatically creates table structures in the destination Data Warehouse with the correct data types and fields. This process ensures that the data is correctly formatted and avoids data quality issues that can arise from manual table creation or incorrect data types.

xGen supports a variety of data load modes as well:

1. Incremental – Hist + Active

During the initial data load, all records in the staging table are inserted into the base object as new records. Once the initial data load has occurred for a base object, any subsequent load processes are called incremental loads because only new or updated data is loaded into the base object.

2. Full Refresh – Overwrite

Overwrite sync mode is a powerful feature for data replication, particularly when the entire dataset needs to be refreshed. This mode is distinct from other sync strategies, as it completely replaces the data in the destination with the latest data from the source during each sync operation.

3. Incremental – Append

Incremental Append mode i.e. syncing only replicate new or modified data. This prevents re-fetching data that you have already replicated from a source.

4. Full Refresh – Append

In this variant, new syncs will take all data from the sync and append it to the destination table. Therefore, if syncing similar information multiple times, every sync will create duplicates of already existing data.

For the source connectors which do not have the option for incremental data load mode, only two data load modes will be available (Full refresh – Overwrite and Full refresh – Append)

Create Data Sync #

1. Setting up a new Data Sync in xGen is a quick and simple process! When viewing the xGen UI, you’ll see the main navigation bar on the left side of your screen. Click the Spaces tab to bring up a list of all available spaces and enter the space of choice.

2. Once inside the space, you’ll see the Data Sync button on the main navigation bar on the left side of your screen. Click the Data Sync button and click on + Add.

3. You can choose the Source connector from the Select Source drop down and also the Destination connector from the Select Destination drop down. In the Sync name field, you can enter a name of your choosing to help you identify this instance of the Data Sync.

4. Once you’ve filled out all the required fields, select the entities that are to be moved from the source connector to the destination. Select the appropriate extraction mode for the entities. In case of extraction mode being Incremental, you will also need to select ‘Cursor Field’ for the same.

5. Confirm or select primary key columns for the entities to be extracted. This action needs to performed only when primary key columns are not fetched from the source data.

6. Click on Settings for additional options such as table prefix, destination schema or transformation.
Note: When a prefix is added it will precede the original table name. (Example; Table ‘category’ will be created as ‘prefix_category’).

7. Once you have entered all the required fields, click the + Create button and xGen will create a Data Sync.

Edit Data Sync #

1. Navigate to Data Sync and select the desired sync form the list to edit.

2. Click on Sync Details button and edit the desired parameter.

3. Once you’ve edited all the required fields, click the Save button. When you edit the data sync and changes are saved, it will run automatically and extract the data from the source accordingly.

Delete Data Sync #

1. Navigate to Data Sync and select the desired sync form the list to delete.

2. Click on Sync Details button and click the Delete button.

Before deleting any data sync, you have to make sure that any objects in xGen are not dependent on the selected data sync. If yes, then you need to delete the dependent items first and then can proceed with deleting the data sync.

Schema Change in Data Sync #

1. Navigate to Data Sync and select the desired sync form the list where Scheme Updates Detected icon appears in yellow colour.

2. Click on Sync Details button and click on the Edit button.

3. Click on Refresh Schema tab and click on Apply button. Also, due to changes in the source data configuration, you will have an option to reset the entire data load or not. A reset will delete data in the destination of the affected streams and then re-sync that entire data.

4. Once the schema changes have been applied, click on Save.

When schema is updated in the data sync, it will run automatically and extract the data from the source accordingly.

Data Sync Home Screen Buttons #

When you navigate to Data Sync home screen, it will be visible in a Card View, which can also be changed to List View. You will have have a few action buttons as explained below,

1. Schema Changes Detected (Yellow) – If there are any schema changes in the source data, you will be notified via this icon.

2. Sync Data Flow (Green) – Runs the data sync.

3. Job Logs (Purple) – Shows the job logs for previously run data sync as well as the current running data sync, you can also view job logs here.

4. Show Pipeline (Blue) – Shows if the data sync has been scheduled to run via Data Pipeline.

5. Show Sync Lineage (Blue) – Shows the lineage of the data sync connection in xGen.

6. Show Sync Details (Blue) – Allows you to edit the data sync.

7. Delete Connection (Red) – Allows you to delete the data sync.