Kinesis#

Create Private Connection for Kinesis#

Prerequisites#

  • Get the ARNs of kinesis streams which are owned by the user and will be used in Flink jobs.

Step 1: Set up OpenID Connect#

For enhanced security, we highly advise against bundling or distributing long-term AWS credentials with your applications. Instead, it is considered best practice to build your application using temporary AWS security credentials obtained through an OIDC (OpenID Connect) identity provider.

Here are the steps to accomplish this:

  • Open the IAM console at https://console.aws.amazon.com/iam/

  • In the navigation pane, choose Identity providers, and then choose Add provider.

  • For Provider type, choose OpenID Connect.

  • For Provider URL, type the URL of the IdP: https://VVC_OIDC_ID and then choose Get thumbprint to verify the server certificate of your identity provider.

Note

OIDC Provider URLs for different regions:

  • For Audience, type the client ID issued by the Identity provider for your app: sts.amazonaws.com

  • Verify the information that you have provided. When you are done choose Add provider.

Step 2: Assign an IAM role to the identity provider#

To enable your Flink jobs running in VVC workspace to access the Kinesis streams associated with your account, you need to assign an IAM role to your identity provider, which grants external user identities managed by your identity provider the necessary permissions to access AWS resources.

It’s important to keep in mind that a single user can only have one role assigned to their identity provider in a single VVC workspace. However, you can associate the role with additional policies or modify existing policies to grant additional permissions.

Here are the steps to create a new role with the necessary permissions to access Kinesis streams:

  1. Open the IAM console at https://console.aws.amazon.com/iam/

  2. In the navigation pane, choose Roles, and then choose Create role.

  3. Select trusted entity

    1. For Trusted entity type, choose Web Identity.

    2. For Web identity, choose the provider established above.

    3. For Audience, choose “sts.amazonaws.com”, then choose Next.

  4. Add permissions

    1. If there’s no fitable existing policy, choose Create policy to create a new policy to define the read/write permissions for specified kinesis streams. An example policy with read permissions of a specified stream is as following, ARNs of kinesis streams fetched above can be specified in the Resource field:

    2. {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadInputStream",
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:ListShards"
            ],
            "Resource": [
                "arn:aws:kinesis:us-west-1:user-account-id:stream/stream-name"
            ]
        }
    ]
}

    1. Choose policies shown in the permissions policies list, then choose Next.

  5. Name, review, and create

    a) For Role name, type a meaning name to identify this role. a) Verify the information that you have provided. When you are done choose Create role.

Copy the ARN of the newly created role, as it will be required to establish the connection on the VVC portal.

Placeholder