DynamoDB does not use fixed table schemas. The discovery phase involves three key steps:
The connector performs a scan of the table, initially limiting the scan to 1,000 items. If the data exceeds 1MB, it will initiate additional scans with the same limit until at least 1,000 items are retrieved.
Once the items are retrieved, the connector aggregates all unique top-level attributes found across the items. It assumes that identical attributes across items have the same type, with possible nested attribute values.
For every attribute discovered in the items, the connector identifies its type by using the AttributeValue.type() method. Based on the type received, the attribute is mapped to an appropriate type supported by xGen in the schema.
| Feature | Supported |
|---|---|
| Full Refresh Sync | Yes |
| Incremental – Append Sync | Yes |
| Replicate Incremental Deletes | No |
| Namespaces | No |
This sync works as a typical full refresh sync, replicating all data without any incremental checks.
The cursor field must be a top-level attribute and cannot be nested. It should always have a value and must be either a string or integer type. If these conditions are not met, the results of the incremental sync might be inconsistent and rely on DynamoDB’s comparison algorithm.
This section provides a detailed guide on how to configure the connector to connect with DynamoDB.
To use User-based Access, provide the access_key_id and secret_access_key in the configuration. For Role-based Access, omit both parameters. In this case, the connector will use DefaultCredentialsProvider, which will utilize the AWS role executing the container workload.