Skip to main content
Skip to main content
Edit this page

Amazon DocumentDB source setup guide

Supported DocumentDB versions​

ClickPipes supports DocumentDB version 5.0.

Configure change stream log retention​

By default, Amazon DocumentDB has a 3-hour change stream log retention period, initial load may take much longer depending on existing data volume in your DocumentDB. We recommend setting the change stream log retention to 72 hours or longer to ensure that it is not truncated before the initial snapshot is completed.

Update change stream log retention via AWS Console​

  1. Click Parameter groups in the left panel, find the parameter group used by your DocumentDB cluster (if you are using the default parameter group, you will need to create a new parameter group first in order to modify it).
  1. Search for change_stream_log_retention_duration, select and edit it to 259200 (72 hours)
  1. Click Apply Changes to apply the modified parameter group to your DocumentDB cluster immediately. You should see the status of the parameter group transition to applying, and then to in-sync when the change is applied.

Update change stream log retention via AWS CLI​

To check the current change stream log retention period via AWS CLI:

aws docdb describe-db-cluster-parameters --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --query "Parameters[?ParameterName=='change_stream_log_retention_duration'].{Name:ParameterName,Value:ParameterValue}"

To set the change stream log retention period to 72 hours via AWS CLI:

aws docdb modify-db-cluster-parameter-group --db-cluster-parameter-group-name <PARAMETER_GROUP_NAME> --parameters "ParameterName=change_stream_log_retention_duration,ParameterValue=259200,ApplyMethod=immediate"

Configure a database user​

Connect to your DocumentDB cluster as an admin user and execute the following command to create a database user for MongoDB CDC ClickPipes:

db.getSiblingDB("admin").createUser({
    user: "clickpipes_user",
    pwd: "some_secure_password",
    roles: ["readAnyDatabase", "clusterMonitor"],
})
Note

Make sure to replace clickpipes_user and some_secure_password with your desired username and password.

What's next?​

You can now create your ClickPipe and start ingesting data from your DocumentDB instance into ClickHouse Cloud. Make sure to note down the connection details you used while setting up your DocumentDB cluster as you will need them during the ClickPipe creation process.