To set up AWS S3 as data lake, you need to follow below steps
1. Navigate to Settings.
2. Click on Destination
3. Click on Setup Data Lake.
4. Insert all the data lake specific credentials, along with a name and description for the connection.
5. Click on Validate and then Create to save the data lake connection.
| S3 Key ID | Access key ID to access S3 bucket |
| S3 Access Key | Corresponding secret access key ID |
| S3 Bucket Name | Name of the S3 bucket (Keep default) |
| S3 Bucket Path | Directory under the S3 bucket where the data will be written |
| S3 Bucket Region | Region of the S3 bucket |
| Output Format | Format of data output |
| Flattening | Whether the input JSON data should be normalized in the output CSV |
| Compression | Whether the output files should be compressed |
| Endpoint | S3 endpoint URL |
| S3 Path Format | Format string on how data will be organized inside the S3 bucket |
| S3 Filename pattern | Allows to set the file name format for S3 staging files |
To set up RedShift as data warehouse, you need to follow below steps
1. Navigate to Settings.
2. Click on Destination
3. Click on Setup Data Warehouse.
4. Insert all the data warehouse specific credentials, along with a name and description for the connection.
5. Click on Validate and then Create to save the data warehouse connection.
| Host | Host endpoint of Redshift cluster |
| Port | Port of database |
| Username | Username to access database |
| Password | Password of corresponding username |
| Database | Name of database |
| Default Schema | Default schema where the loaded tables are stored |
| JDBC URL Params | Additional properties to pass JDBC URL string when connecting to database |
| Uploading Method | The way data will be uploaded to Redshift via AWS S3 |
| S3 Bucket Name | S3 bucket name where data will be stored |
| S3 Bucket Path | S3 bucket path where data will be stored |
| S3 Bucket Region | Region of S3 bucket |
| S3 Access Key ID | Access key ID |
| S3 Secret Access Key | Secret access key |
| Purge Staging Files and Tables | Whether to delete staging files from S3 after completing the sync |
| Destination V2 Raw Table Schema | Schema to write raw tables into |
| Enable Loading data incrementally to final table | |
| Disable Final Tables | Disbale / Enable writing final tables |
| SSH Tunnel Method | Whether to initiate SSH tunnel before connecting to database |