Custom Connectors#
Background Information#
When using Flink SQL to develop drafts, you need to use SQL connectors to connect your source tables, result tables, and dimension tables. Fully managed Flink supports multiple types of common connectors in the cloud. However, compared with the technology stacks that are commonly used for big data, fully managed Flink covers only a small part of connectors. If you want to use more types of connectors, you can customize connectors and upload them for use.
Precautions#
You must develop custom connectors based on the connector standards that are defined by the Apache Flink community. This way, the custom connectors can be recognized and used by fully managed Flink. You must explicitly define connector metadata files and declare factory classes. For more information about how to develop custom connectors, see User-defined Sources & Sinks.
Connectors of the same type can only be uploaded once.
To avoid JAR package dependency conflicts, you also need to pay attention to the following points:
Keep the Flink version that the Flink image and the Pom depend on consistent.
Provided in the dependencies.
Use Shade to package other third-party dependencies. For details about Shade packaging, see Apache Maven Shade Plugin
If a deployment uses a custom connector and the custom connector is updated when the deployment is running, the deployment still uses the original custom connector. If you restart the deployment, the deployment uses the new connector.
Create custom connector#
Log in to Ververica Cloud.
On the Dashboard page, locate the workspace you want to manage, click the title of the workspace or this icon ⋮, and select Open Console.
In the left-side navigation pane of the Console, click Connectors and choose the Create connector tab.
Upload a custom connector JAR file. You can upload a custom connector JAR file in any of the following ways:
Upload File: After clicking Click to select, select your target connector JAR file.
External URL: Enter the external URL address.
Note
The external URL support only accessible HTTP links. You need to make sure that the network between the Flink service and the external HTTP service is connected.
After the upload is complete, click Next. The system will parse the custom connector content you uploaded. If the parsing is successful, you can proceed to the next step. If the parsing fails, please confirm whether the custom connector code you uploaded complies with the Flink community standard.
Click Finish.
Update custom connector#
Log in to Ververica Cloud.
On the Dashboard page, locate the workspace you want to manage, click the title of the workspace or this icon ⋮, and select Open Console.
In the left-side navigation pane of the Console, click Connectors and then Custom connectors.
Click Create Connector.
Upload a custom connector JAR file. You can upload a custom connector JAR file in any of the following ways:
Upload File: After clicking Click to select, select your target connector JAR file.
External URL: Enter the external URL address.
Note
The external URL support only accessible HTTP links. You need to make sure that the network between the Flink service and the external HTTP service is connected.
After the upload is complete, click Next. The system will parse the custom connector content you uploaded. If the parsing is successful, you can proceed to the next step. If the parsing fails, please confirm whether the custom connector code you uploaded complies with the Flink community standard.
Click Finish.
Delete custom connector#
If your custom connector is no longer used, you can delete the custom connector by following the steps below.
Log in to Ververica Cloud.
On the Dashboard page, locate the workspace you want to manage, click the title of the workspace or this icon ⋮, and select Open Console.
In the left-side navigation pane of the Console, click Connectors and then Custom connectors.
Click the Delete icon below the name of the required custom connector.