Release History
Help

YMatrix

Quick Start

Deployment

Data Usage

Manage Clusters

Security

Best Practice

Technical Principles

Advanced Features

Graph Database

Guide

SQL Reference

FAQ

MatrixGate Kafka Encrypted Connection (SSL) Configuration

This document describes how to enable an SSL encrypted connection when MatrixGate connects to Kafka. The SSL configuration features described here apply to MatrixGate v6.8.2 and later.

1 Feature Description

SSL connections and the existing SASL authentication are two independent configuration items. They can be enabled separately or used together.

MatrixGate currently supports one-way TLS: the Kafka connection is encrypted, and MatrixGate verifies the Kafka broker certificate.

After SSL-ca-cert is configured, MatrixGate enables the SSL connection.

Note!

SSL-ca-cert must be configured as the path of the CA certificate that corresponds to the Kafka broker certificate. The certificate file must be readable on the machine where the MatrixGate process runs.

2 Prerequisites

Before configuration, confirm that:

  • The MatrixGate version is v6.8.2 or later.

  • The Kafka broker has an SSL listener address enabled.

  • The MatrixGate node can access the Kafka broker.

  • The CA certificate file is saved on the MatrixGate node and is readable.

3 Configuration Examples

The following examples show only the key settings of the Kafka data source related to SSL and SASL. Adjust the broker addresses, certificate paths, credentials, topic, partition, and job for your environment.

3.1 SSL Only

When only SSL is enabled, you need to configure the Kafka broker addresses and the CA certificate path.

[source]
  source = "kafka"

[source.kafka]
  # Kafka broker addresses. Separate multiple brokers with commas.
  kafka-broker = "broker1:9094,broker2:9094"

  # CA certificate path, used to verify the Kafka broker certificate. Configuring this parameter enables the SSL connection.
  SSL-ca-cert = "/path/to/ca.pem"

  [[source.kafka.topic]]
    topic = "topic_name"
    partition = "*"
    job = "job_name"

3.2 SASL_SSL_PLAIN

When SSL and SASL/PLAIN are enabled together, you need to configure the SSL parameters and the SASL username and password.

[source]
  source = "kafka"

[source.kafka]
  # Kafka broker addresses. Separate multiple brokers with commas.
  kafka-broker = "broker1:9093,broker2:9093"

  # CA certificate path, used to verify the Kafka broker certificate. Configuring this parameter enables the SSL connection.
  SSL-ca-cert = "/path/to/ca.pem"

  # Whether to enable SASL authentication.
  enable-SASL = true

  # Whether to enable the Kafka SASL handshake. Standard Kafka SASL connections usually set this to true.
  enable-SASL-handshake = true

  # SASL mechanism. PLAIN means SASL/PLAIN username and password authentication.
  SASL-mechanism = "PLAIN"

  # SASL/PLAIN username.
  SASL-user = "user"

  # SASL/PLAIN password.
  SASL-password = "password"

  [[source.kafka.topic]]
    topic = "topic_name"
    partition = "*"
    job = "job_name"

3.3 SASL_SSL_SCRAM

When SSL and SASL/SCRAM are enabled together, you need to configure the SSL parameters, the SASL username and password, and the SCRAM hash algorithm.

[source]
  source = "kafka"

[source.kafka]
  # Kafka broker addresses. Separate multiple brokers with commas.
  kafka-broker = "broker1:9093,broker2:9093"

  # CA certificate path, used to verify the Kafka broker certificate. Configuring this parameter enables the SSL connection.
  SSL-ca-cert = "/path/to/ca.pem"

  # Whether to enable SASL authentication.
  enable-SASL = true

  # Whether to enable the Kafka SASL handshake. Standard Kafka SASL connections usually set this to true.
  enable-SASL-handshake = true

  # SASL mechanism. SCRAM means SASL/SCRAM authentication.
  SASL-mechanism = "SCRAM"

  # SASL/SCRAM username.
  SASL-user = "user"

  # SASL/SCRAM password.
  SASL-password = "password"

  # SCRAM hash algorithm. SHA-256 and SHA-512 are supported.
  SCRAM-algorithm = "SHA-512"

  [[source.kafka.topic]]
    topic = "topic_name"
    partition = "*"
    job = "job_name"

3.4 SASL_SSL_GSSAPI

When SSL and SASL/GSSAPI are enabled together, you need to configure the SSL parameters and the SASL/GSSAPI parameters.

GSSAPI authentication uses fast authentication (PAFXFAST) by default. If the connection fails during fast authentication, you can set GSSAPI-disable-PAFXFAST = true in [source.kafka] to disable fast authentication and try again.

3.4.1 Authenticate with a keytab File

[source]
  source = "kafka"

[source.kafka]
  # Kafka broker addresses. Separate multiple brokers with commas.
  kafka-broker = "broker1:9093,broker2:9093"

  # CA certificate path, used to verify the Kafka broker certificate. Configuring this parameter enables the SSL connection.
  SSL-ca-cert = "/path/to/ca.pem"

  # Whether to enable SASL authentication.
  enable-SASL = true

  # Whether to enable the Kafka SASL handshake. Standard Kafka SASL connections usually set this to true.
  enable-SASL-handshake = true

  # SASL mechanism. GSSAPI means Kerberos authentication.
  SASL-mechanism = "GSSAPI"

  # GSSAPI auth method. KEYTAB means keytab-file authentication.
  GSSAPI-auth-type = "KEYTAB"

  # Kerberos configuration file path, usually krb5.conf.
  GSSAPI-kerberos-config-path = "/etc/krb5.conf"

  # keytab file path, used when GSSAPI-auth-type = "KEYTAB".
  GSSAPI-keytab-path = "/etc/security/keytabs/kafka.service.keytab"

  # GSSAPI user or principal.
  GSSAPI-user = "kafka/host.example.com"

  # Kerberos realm.
  GSSAPI-realm = "EXAMPLE.COM"

  # Kerberos service name used by the Kafka broker, usually kafka.
  GSSAPI-service-name = "kafka"

  # Whether to disable GSSAPI fast authentication (PAFXFAST). Default is false; set it to true when the connection fails during fast authentication.
  GSSAPI-disable-PAFXFAST = false

  [[source.kafka.topic]]
    topic = "topic_name"
    partition = "*"
    job = "job_name"

3.4.2 Authenticate with Username and Password

[source]
  source = "kafka"

[source.kafka]
  # Kafka broker addresses. Separate multiple brokers with commas.
  kafka-broker = "broker1:9093,broker2:9093"

  # CA certificate path, used to verify the Kafka broker certificate. Configuring this parameter enables the SSL connection.
  SSL-ca-cert = "/path/to/ca.pem"

  # Whether to enable SASL authentication.
  enable-SASL = true

  # Whether to enable the Kafka SASL handshake. Standard Kafka SASL connections usually set this to true.
  enable-SASL-handshake = true

  # SASL mechanism. GSSAPI means Kerberos authentication.
  SASL-mechanism = "GSSAPI"

  # GSSAPI auth method. USER means username and password authentication.
  GSSAPI-auth-type = "USER"

  # Kerberos configuration file path, usually krb5.conf.
  GSSAPI-kerberos-config-path = "/etc/krb5.conf"

  # GSSAPI user.
  GSSAPI-user = "user"

  # GSSAPI password, used when GSSAPI-auth-type = "USER".
  GSSAPI-password = "password"

  # Kerberos realm.
  GSSAPI-realm = "EXAMPLE.COM"

  # Kerberos service name used by the Kafka broker, usually kafka.
  GSSAPI-service-name = "kafka"

  # Whether to disable GSSAPI fast authentication (PAFXFAST). Default is false; set it to true when the connection fails during fast authentication.
  GSSAPI-disable-PAFXFAST = false

  [[source.kafka.topic]]
    topic = "topic_name"
    partition = "*"
    job = "job_name"
← Previous
Kafka
Next →
Spark