Skip to content

Commit

Permalink
DOC-4674 Addressing feedback from Georgiana, Yaron, and Cornel
Browse files Browse the repository at this point in the history
  • Loading branch information
cmilesb committed Jan 8, 2025
1 parent 79fee3a commit 40891de
Show file tree
Hide file tree
Showing 6 changed files with 47 additions and 16 deletions.
6 changes: 5 additions & 1 deletion content/operate/rc/databases/rdi/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,18 +41,22 @@ RDI encrypts all network connections with TLS. The pipeline will process data fr

Before you can create a data pipeline, you must have:

- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS)
- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). This will be the target database.
- One supported source database, also hosted on AWS and connected to [AWS PrivateLink](https://aws.amazon.com/privatelink/):
- MySQL
- Oracle
- SQL Server
- PostgreSQL
- mariaDB
- Amazon Aurora

{{< note >}}
Please be aware of the following limitations:

- The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
- The target database must use multi-zone [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}).
- The target database can use TLS, but can not use mutual TLS.
- The target database cannot be in the same subscription as another database that has a data pipeline.
- Source databases must also be hosted on AWS.
- One source database can only be synced to one target database.
{{< /note >}}
Expand Down
6 changes: 3 additions & 3 deletions content/operate/rc/databases/rdi/define.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ After you have [prepared your source database]({{<relref "/operate/rc/databases/
1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) in the **Source database secrets ARN** field.
1. Select **Start pipeline setup**.
{{<image filename="images/rc/rdi/rdi-start-pipeline-setup.png" alt="The start pipeline setup button." width=200px >}}
1. Redis Cloud will attempt to connect to PrivateLink. Accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).
1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).

If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{<relref "/operate/rc/databases/rdi/setup#set-up-connectivity">}}) for more info.

Expand All @@ -49,13 +49,13 @@ When your pipeline is provisioned, select **Complete setup**. You will then [def

## Define data pipeline

After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas and columns that you want to import and synchronize with your primary database.
After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database.

### Configure a new pipeline

1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Pipeline definition** section.
{{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}}
1. For the **Configure a new pipeline** option, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON**.
1. For the **Configure a new pipeline** option, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON.
{{<image filename="images/rc/rdi/rdi-configure-new-pipeline.png" alt="The Pipeline definition screen. Configure a new pipeline is selected." width=75% >}}
Select **Continue**.
{{<image filename="images/rc/rdi/rdi-continue-button.png" alt="The continue button." width=150px >}}
Expand Down
39 changes: 32 additions & 7 deletions content/operate/rc/databases/rdi/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ weight: 1
{{<image filename="images/rc/rdi/rdi-select-source-db.png" alt="The select source database type list." width=500px >}}
1. If you know the size of your source database, enter it into the **Source dataset size** field.
{{<image filename="images/rc/rdi/rdi-source-dataset-size.png" alt="Enter the amount of source data you plan to ingest." width=400px >}}
1. Under **Setup connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it.

{{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The select source database type list." width=80% >}}

The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`.

## Prepare source database

Expand All @@ -41,24 +46,44 @@ See the [RDI architecture overview]({{< relref "/integrate/redis-data-integratio

You need to share your source database credentials and certificates in an Amazon secret with Redis Cloud so that the pipeline can connect to your database.

To do this, you need to:
1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions.
1. [Create a secret](#create-database-credentials-secret) containing the source database credentials encrypted using that key.

### Create encryption key

In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) with the following settings:

1. In **Step 1 - Configure key**:
- **Key type**: Select **Symmetric**.
- **Key usage**: Select **Encrypt and decrypt**.
- Under **Advanced options**, set the following:
- **Key material origin**: Select **KMS - recommended**.
- **Regionality**: Select **Single-Region key**.
1. In **Step 2 - Add labels**, add an alias and description for the key.
1. In **Step 3 - Define key administrative permissions**, under **Key deletion**, select **Allow key administrators to delete this key**.
1. In **Step 4 - Define key usage permissions**, under **Other AWS accounts**, select **Add another AWS account**. Enter the AWS account ID for the Redis Cloud cluster that you saved earlier.

### Create database credentials secret

In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:

- **Key/value pairs**: Enter the following key/value pairs.

- `username`: Database username
- `password`: Database password
- `server_certificate`: Server certificate in PEM format *(TLS only)*
- `client_certificate`: [X.509 client certificate](https://en.wikipedia.org/wiki/X.509) or chain in PEM format *(mTLS only)*
- `client_certificate_key`: Key for the client certificate or chain in PEM format *(mTLS only)*
- `client_certificate_passphrase`: Passphrase or password for the client certificate or chain in PEM format *(mTLS only)*
- `trust_certificate`: Server certificate in PEM format *(TLS only)*
- `client_public_key`: [X.509 client certificate](https://en.wikipedia.org/wiki/X.509) or chain in PEM format *(mTLS only)*
- `client_private_key`: Key for the client certificate or chain in PEM format *(mTLS only)*
- `client_private_key_passphrase`: Passphrase or password for the client certificate or chain in PEM format *(mTLS only)*

{{<note>}}
If your source database has TLS or mTLS enabled, we recommend that you enter the `server_certificate`, `client_certificate`, and `client_certificate_key` into the secret editor using the **Key/Value** input method instead of the **JSON** input method. Pasting directly into the JSON editor may cause an error.
If your source database has TLS or mTLS enabled, we recommend that you enter the `trust_certificate`, `client_public_key`, and `client_private_key` into the secret editor using the **Key/Value** input method instead of the **JSON** input method. Pasting directly into the JSON editor may cause an error.
{{</note>}}

- **Encryption key**: Select a self-managed encryption key from the list of keys, or select **Add a new key** to [create one](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html).
- **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.

- **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `<AWS ACCOUNT ID>` with the provided AWS Account ID.
- **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `<AWS ACCOUNT ID>` with the AWS account ID for the Redis Cloud cluster that you saved earlier.

```json
{
Expand Down
12 changes: 7 additions & 5 deletions content/operate/rc/databases/rdi/view-edit.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,21 @@
---
Title: View and edit data pipeline
Title: View data pipeline
alwaysopen: false
categories:
- docs
- operate
- rc
description: Edit and observe your data pipeline.
description: Observe and stop your data pipeline.
hideListLinks: true
weight: 3
---

Use the **Data pipeline** tab in your database to view and edit your data pipeline.
Use the **Data pipeline** tab in your database to view your data pipeline.

The **Data pipeline** tab gives an overview of your data pipeline and lets you view your data stream metrics.

{{<image filename="images/rc/rdi/rdi-status-metrics-tables.png" alt="The select source database type list." width=80% >}}

The **Status** table shows statistics for the whole data pipeline:
- **Status**: The status of the data pipeline. Possible statuses include:
| Status | Description |
Expand All @@ -39,7 +41,7 @@ The **Data stream metrics** table shows the following metrics for each data stre
| **Filtered** | Number of records from the source table that were filtered from being inserted into the target database. |
| **Rejected** | Number of records from the source table that could not be parsed or inserted into the target database. |


<!-- DOUBLE CHECK THIS BEFORE PHASE 2
## Edit data pipeline
To change the data you want to ingest from the data pipeline:
Expand Down Expand Up @@ -106,7 +108,7 @@ To reset the data pipeline and restart the ingest process:
1. Select **Reset data pipeline**.
At this point, the data pipeline will re-ingest data from the source database to your target Redis database.
At this point, the data pipeline will re-ingest data from the source database to your target Redis database. -->

## Stop and restart data pipeline

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 40891de

Please sign in to comment.