RDS#

Create Private Connection for RDS#

Placeholder

Prerequisites#

  • Obtain the endpoint name and port of an existing RDS database cluster that is owned by the user and will be accessed in Flink jobs. If you intend to use the RDS as a sink, ensure that you select endpoints with Writer type instead of Reader type. When you create a database, make sure that the Settings section specifies the DB cluster identifier field and not the DB instance field. Otherwise, all the steps will not work. The difference depends on the AWS region you choose to create a database in.

  • Download AWS official template sample file CrossAccountRDSAccess.yml, which provides support for cross-account RDS access.

  • Enable binlog format on AWS (Head to the Enabling Binary Log section to learn how to do this.)

  • Make sure that the Security Group for the RDS is properly set up. Follow the instructions in this document.

Using the console#

To access an RDS database owned by a user from Flink jobs running in a VVC workspace, it is necessary to establish an RDS connection via PrivateLink and NLB (Network Load Balancer).

Here are the steps to accomplish this:

Step 1: Create stack#

  1. Open the CloudFormation console at https://console.aws.amazon.com/cloudformation/

  2. Choose Create stack with new resources (standard).

  3. In the Prepare template field, choose Template is ready.

  4. In the Template source field, choose Upload a template file, then upload CrossAccountRDSAccess.yml file.

  5. Click Next.

Step 2: Specify stack details#

  1. In the Stack name field, provide a name for your stack.

  2. In the Enter Lambda Function Name field, use the default Lambda function name or create a new one.

  3. In the RDS Type drop-down menu, choose db-cluster

  4. In the Enter (Amazon Aurora/Amazon RDS/Amazon RDS Proxy) Endpoint field, type the endpoint name of a database

    • To check the endpoint of your database, type RDS in the search bar at the top of the AWS Console page and click on RDS in the drop-down menu.

    • In the side navigation pane, click Databases.

    • Click on your DB identifier.

    • Copy the endpoint name and port of the Write instance type from the Connectivity & security tab.

  5. Paste/type the endpoint name and port of the Write instance type in the Enter (Amazon Aurora/Amazon RDS/Amazon RDS Proxy) Endpoint and Enter RDS Port fields.

  6. In the Select VPC from the list field, choose your VPC from the drop-down menu. Make sure that you followed the instructions from this document before selecting your VPC.

  7. In the Select Subnets field, choose your subnet with a private connection from the drop-down menu. Make sure that you followed the instructions from this document before selecting your subnet.

  8. In the Select Security Group for Lambda field, choose your group that can communicate with RDS and Lambda between each other from the drop-down menu.

  9. In the Provide Role ARN of target Account/VPC field, type arn:aws:iam::794031221915:role/pyxis.

Step 3: Configure stack options#

  1. Use the default stack configuration or update it as you wish.

  2. Click Next.

Step 4: Review stack#

  1. Verify the information that you have provided.

  2. Check a box “I acknowledge that AWS CloudFormation might create IAM resources with custom names”.

  3. When you are done choose Submit.

  4. Wait until the stack is created.

Step 5: Configure the Security Group#

To ensure proper connectivity between your application and the RDS instance, you need to configure the security group associated with the RDS instance to allow incoming traffic to the RDS port (e.g., 3306 for MySQL).

  1. Sign in to the AWS Management Console and navigate to the Amazon RDS console at https://console.aws.amazon.com/rds/.

  2. In the side navigation pane, click Security groups >> Create subnet

  3. Select your security group ID

  4. Click Edit Inbound rules >> Add rule

  5. In the Type column, make sure you set Custom TCP value, double check if the port range is correct, and add you VPC address in the Source column.

  6. Click on Save rules to apply the changes.

Step 6: Enable Cross-zone load balancing for the NLB#

  1. On your Stacks page, click the Resources tab.

  2. Find rNetworkLoadBalancer in the Logical ID column and click on its physical ID in the Physical ID column.

  3. Select your network balancer and click on the Attributes tab

  4. To enable cross-zone load balancing, click Edit, and toggle Enable cross-zone load balancing

Step 7: Set up a Target Group IP address in the NLB#

You must call the Lambda function to register a target IP for a Target Group and you can do this by performing a test run for the Lambda function.

  1. On your Stacks page in CloudFormation, find AWS::Lambda::Function type in the Type column, and click on its physical ID in the Physical ID column.

  2. Scroll down and click Test.

  3. In the form, in the Event name provide a name for your test, and hit Save.

  4. Click on Test again to run the test.

Note

Performing a test run for the Lambda function in this stack updates the IP address of RDS primary instance in the NLB. The Lambda function in this stack is responsible for recovering the target group of NLB, otherwise this connection may be broken after database cluster failover. So it’s important to make sure the Lambda function works fine. From now on, the target will be automatically recovered after database cluster failover.

Step 8: Copy Service_Name and Private_Endpoint to a clipboard#

You will need Service_Name and Private_Endpoint in the final step to create a connection in the VVC Console.

Service_Name

  1. To check the service name, type CloudFormation in the search bar at the top of the AWS Console page and click on CloudFormation in the drop-down menu. You can also go directly to CloudFormation.

  2. In the side navigation pane, click Stacks

  3. Click on your stack name.

  4. Click on the Resources tab.

  5. Find rVPCEndpointService in the Logical ID column, remember its physical ID, and click on its physical ID.

  6. Find that physical ID in the list of Endpoint services.

  7. In the Details tab, find Service name, and copy its value.

Private_Endpoint

  1. To check the endpoint of your database, type RDS in the search bar at the top of the AWS Console page and click on RDS in the drop-down menu. You can also go directly to RDS.

  2. In the side navigation pane, click Databases

  3. Click on your DB identifier* and

  4. Copy the endpoint name and port of the Write instance type.

Using AWS CLI#

To access an RDS database owned by a user from Flink jobs running in a VVC workspace, it is necessary to establish an RDS connection via PrivateLink and NLB (Network Load Balancer).

Here are the steps to achieve this:

  1. Create new stack based on AWS official sample file CrossAccountRDSAccess.yml, please prepare following environment variables before executing:

    1. STACK_NAME: a meaning name to identify this stack.

    2. TEMPLATE_FILE_PATH: path of template file downloaded before.

    3. VPC_ID: target VPC in which NLB will be deployed.

    4. SUBNET_IDS: subnet IDs with which NLB and Lambda function will be associated, space-separated for multiple subnets, choose that can access both the specified RDS database cluster and public network.

    5. SECURITY_GROUP_IDS: security group IDs with which Lambda function will be associated, space-separated for multiple security groups, choose that can access both the specified RDS database cluster and public network.

    6. ALLOWED_ROLE_ARN: the ARN of a role who can establish connection with the to-be-created endpoint service: VVC_ROLE_ARN (Tips for developer: VVC_ROLE_ARN must be replaced by the actual VVC Role ARN which can be retrieved from pyxis OpenAPI.).

    7. RDS_PORT: the endpoint port of a database cluster fetched above.

    8. RDS_CLUSTER_ENDPOINT: the endpoint name of a database cluster fetched above.

    9. aws cloudformation create-stack \
    --stack-name ${STACK_NAME} \
    --template-body file://${TEMPLATE_FILE_PATH} \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters ParameterKey=pVPCid,ParameterValue=${VPC_ID} \
                 ParameterKey=pSubnetsGroup,ParameterValue=${SUBNET_IDS} \
                 ParameterKey=pSecurityGroup,ParameterValue=${SECURITY_GROUP_IDS} \
                 ParameterKey=pTargetRoleARN,ParameterValue=${ALLOWED_ROLE_ARN} \
                 ParameterKey=pRDSPOrt,ParameterValue=${RDS_PORT} \
                 ParameterKey=pDBEndPoint,ParameterValue=${RDS_CLUSTER_ENDPOINT}

  2. Enable Cross-zone load balancing for NLB

    1. NLB_ARN=$(aws elbv2 describe-load-balancers \
   --names NLB-RDS-${STACK_NAME} \
   --query LoadBalancers[0].LoadBalancerArn \
   --output text)
aws elbv2 modify-load-balancer-attributes \
   --load-balancer-arn ${NLB_ARN} \
   --attributes Key=load_balancing.cross_zone.enabled,Value=true

  3. Trigger a test run for the Lambda function

    1. aws lambda invoke --function-name NLBAutoUpdate-${STACK_NAME} /tmp/outputfile

    2. Execute command above to perform a test run for the Lambda function in this stack to update the IP address of RDS primary instance to the NLB. Then wait until there’s a healthy target, which can be shown by the following commands:

    3. NLB_ARN=$(aws elbv2 describe-load-balancers \
     --names NLB-RDS-${STACK_NAME} \
     --query LoadBalancers[0].LoadBalancerArn \
     --output text)
aws elbv2 modify-load-balancer-attributes \
     --load-balancer-arn ${NLB_ARN} \
     --attributes Key=load_balancing.cross_zone.enabled,Value=true

    b) The Lambda function in this stack is responsible for recovering the target group of NLB, otherwise this connection may be broken after database cluster failover. So it’s important to make sure the Lambda function works fine. From now on, the target will be automatically recovered after database cluster failover. b) At last, copy the name of the endpoint service in this stack, it will be used in the next steps. c) .. code-block:

    aws ec2 describe-vpc-endpoint-service-configurations | yq '.ServiceConfigurations[] | select(.NetworkLoadBalancerArns[]|test("'${STACK_NAME}'")) | .ServiceName' | sed s/\"//g

Create private connection in VVC Console#

To create a private connection in VVC, follow these steps:

  1. Navigate to the VVC Console.

  2. In the left-hand navigation pane, click on Security and then select Private Connection tab. Click on the Create Connection button to start the process.

  3. Select AWS MySQL from the Connection System field and provide a name for the connection.

  4. Add the Service_Name and Private_Endpoint values.

Placeholder

Important

The Private_Endpoint value should not include a port address.