This document provides detailed parameter information for MatrixGate.
Note!
The parameters in this section are those in the configuration file generated before mxgate is started.
| Parameter Name | Default Value | Description |
|---|---|---|
| [database] Category | ||
| --db-database | postgres | Name of the YMatrix database that MatrixGate connects to |
| --db-master-host | Local hostname | Host name of the YMatrix host that MatrixGate connects to |
| --db-master-port | 5432 | Port number of the YMatrix host that MatrixGate connects to |
| --db-user | Current system user name | User name that MatrixGate uses to connect to YMatrix Note! This user must have the permission to create external tables. If you use a non-superuser, grant the permission with the following command: alter user {username} CREATEEXTTABLE; |
| --db-password | Empty | Password of the YMatrix user that MatrixGate uses to connect to |
| --db-max-conn | 10 | Maximum number of connections from MatrixGate to YMatrix |
| [job] Category | ||
| --allow-dynamic | false | When --allow-dynamic=true is specified, the target table to insert into is dynamically adapted based on the content of the POSTed data (the first line). Use this option only when the target table name is not yet determined when MatrixGate starts. If you always insert into a known target table, it is recommended to specify the table name explicitly with --target |
| --delimiter | | | Specifies the character used to separate the columns in each line (row) of the file |
| --error-handling | accurate | How to handle lines with format errors'accurate': Erroneous data is not loaded and an error log is recorded; other data in the batch is not affected'legacy': The entire batch fails |
| --exclude-columns | Empty | The number and order of columns provided by default for data loading must match the table definition. When data loading provides only some columns, use --exclude-columns to mark the column names to exclude; the remaining columns must still follow the order of the table definition. Note: If --use-auto-increment is enabled to skip auto-increment fields, those fields do not need to be listed here. This parameter only needs to mark the other column names to be excluded |
| --format | text | Specifies the data format of the source data: text or csv. text is the fastest but does not support line breaks in character-type values. The csv format applies more widely; character-type columns must be enclosed in double quotes |
| --null-as | Empty string | Specifies the string that represents a null value. The default is an empty string without quotes. If a column in the table has the NOT NULL constraint and the data provides a null value for that column, loading fails. Note: To use \N as the null value, escape the backslash, for example: --null-as \N |
| --time-format | unix-second | Specifies the timestamp unit: unix-second, unix-ms, unix-nano, or raw. \n MatrixGate by default treats the first column of each row as the Unix representation of a timestamp and automatically converts it to the database time format. If the timestamp is not in the first column, or you have already converted it to the database format, use raw so that MatrixGate performs no time-type conversion |
| --upsert-key | Empty | Key names for UPSERT; multiple keys can be specified.A table that requires UPSERT must have a UNIQUE constraint, and all constraint keys must be specified in the parameter |
| --deduplicate-key | Empty | Usage is similar to UPSERT, except that only null values are updated; if the old value is not null, the new value is discarded.Mutually exclusive with --upsert-key; only one can be used |
| --use-auto-increment | true | When the target table contains auto-increment fields, whether to skip assigning values to them during data loading and use the system default auto-increment values instead |
| --target | schemaName.tableName | Specifies the target table name; schemaName can be omitted and defaults to public. Multiple target tables are allowed, used as "--target table1 --target table2 …". When this parameter is not provided, you can specify the --allow-dynamic parameter to allow dynamic adaptation of table names |
| --dml-template | File path of the mapping template that maps JSON fields to tuple columns | |
| [misc] Category | ||
| --log-archive-hours | 72 | MatrixGate log files in the log directory that have not changed for a certain period of time are automatically compressed |
| --log-compress | true | Global switch for automatic log compression |
| --log-dir | /home/mxadmin/gpAdminLogs | Log directory |
| --log-max-archive-files | 0 | Maximum number of compressed log files to keep; when the number is exceeded, the oldest log files are deleted. 0 means no deletion |
| --log-remove-after-days | 0 | Number of days after which compressed log files are automatically deleted. 0 means no deletion |
| --log-rotate-size-mb | 100 | When the current log file exceeds a certain size, it is automatically rotated to a new file and the old file is compressed immediately |
| -v / --verbose | Print detailed verbose logs | |
| -V / --debug | Print detailed debug logs (including verbose and debug logs) | |
| --pprof-port | Port for accessing Pprof information; 0 means disabled |
|
| --no-cleanup | Keep the temporary schema even on normal exit | |
| --grpc-port | Port for accessing gRPC information; 0 means disabled |
|
| [metrics] Category | ||
| --metrics-enable | true | Enable metrics |
| --metrics-sample-interval | 15 (adjusted to 15 from v5.3.2, previously 3) | Metrics sampling interval (seconds). Set it to >0 to enable metrics collection (reduces performance) |
| [source] Category | ||
| --source | http | Data source of MatrixGate; supports http / stdin / kafka / transfer / grpc |
| [source] Category | [HTTP] | Note: This mode is the default data source connection mode in the configuration file |
| --http-port | 8086 | HTTP interface through which users submit data to MatrixGate |
| --max-body-bytes | 4194304 | Maximum size of each HTTP request body |
| --max-concurrency | 40000 | Maximum number of concurrent HTTP connections |
| --request-timeout | 0 | Request timeout; default is 0. When set to a value greater than 0, the request times out after the configured number of milliseconds and returns timeout(408) |
| --disable-keep-alive | false | MatrixGate forcibly closes the connection after each HTTP request |
| --http-debug | false | Output additional diagnostic information for the HTTP source |
| [source] Category | [Transfer] | Note: Migration mode is not the default mode. To use this mode, configure the parameters in this section manually |
| --src-host | IP address of the master of the source database | |
| --src-port | Port number of the master of the source database | |
| --src-user | User name for connecting to the source database (a Superuser is recommended) |
|
| --src-password | Connection password | |
| --src-schema | Schema name of the source table | |
| --src-table | Table name of the source table | |
| --src-sql | SQL used to filter the data to be migrated | |
| --compress | Compression method for transferring data from the Segments of the source database to this database: Empty string “”, meaning no compression, plain-text transfer gzip: Use gzip compression; the gzip Linux command must be installed on the Segment hosts of the source databaselz4: Use lz4 compression; the lz4 Linux command must be installed on the Segment hosts of the source databaselz4 > gzip > no compression is recommended |
|
| --port-base | 9129 | One port is occupied during the transfer |
| --local-ip | Must be an IP address of this machine that the source database can connect to | |
| [source] Category | [Kafka] | Note: This mode is not the default mode. To use this mode, configure the parameters in this section manually; for complete authentication examples, see Kafka Encrypted Connection |
| --kafka-broker | None | Kafka broker addresses; separate multiple brokers with commas |
| --kafka-version | 1_0_0_0 | Kafka version. Optional values: 0_8_2_0, 0_8_2_1, 0_8_2_2, 0_9_0_0, 0_9_0_1, 0_10_0_0, 0_10_0_1, 0_10_1_0, 0_10_1_1, 0_10_2_0, 0_10_2_1, 0_11_0_0, 0_11_0_1, 0_11_0_2, 1_0_0_0, 1_1_0_0, 2_0_0_0 |
| --kafka-lib-log | false | Whether to output logs of the Kafka connection process |
| --kafka-max-checkpoint-gap | 134217728 | Maximum allowed gap (in bytes) between the consumption checkpoint and the consumed offset |
| --kafka-message-count | 0 | Number of messages to consume; 0 means no limit |
| --kafka-params | Empty | Specifies topic, partition, consumer-group, consumer-group-members, and job as a string; can be omitted when [[source.kafka.topic]] is used |
| --enable-sasl | false | Whether to enable SASL authentication |
| --enable-sasl-handshake | false | Whether to enable the Kafka SASL handshake; standard Kafka SASL connections usually set this to true |
| --sasl-mechanism --sasl-user --sasl-password |
Empty | SASL authentication mechanism and credentials |
| --scram-algorithm --scram-authz-id |
Empty | Algorithm and authorization ID used for SCRAM authentication |
| --ssl-ca-cert | Empty | CA certificate path; configuring this parameter enables the SSL connection |
| --gssapi-auth-type | Empty | GSSAPI auth method: KEYTAB means keytab-file authentication, USER means username and password authentication |
| --gssapi-kerberos-config-path --gssapi-keytab-path --gssapi-user --gssapi-password --gssapi-realm --gssapi-service-name |
Empty | Configuration paths and identity information required for GSSAPI authentication. For examples, see Kafka Encrypted Connection |
| --gssapi-disable-pafxfast | false | Whether to disable GSSAPI fast authentication (PAFXFAST); set it to true and retry when the connection fails during fast authentication |
| [source] Category | [[source.kafka.topic]] |
Note: One entry per topic; multiple consumer members can be configured for the same topic |
| --topic | None | Kafka topic to consume |
| --partition | None | Partition to consume; * means all partitions |
| --job | None | Name of the write job corresponding to the topic |
| --consumer-group | Empty | Consumer group used for the topic; when not configured, no consumer group is used |
| --consumer-group-members | — | Number of consumer members for the topic. When multiple members are configured, they consume the topic together within the same consumer group |
| [transform] Category | ||
| --transform | plain | Converts the format or type of the data to be written. Supports plain / json / nil / tsbs / hanagdbc |
| [writer] Category | ||
| --interval | 100 milliseconds | Time period of MatrixGate batch data loading |
| --writer | stream | MatrixGate writes data to YMatrix through a Writer. Supports stream / nil |
--stream-prepared |
10 | Number of slot processes called simultaneously within one job |
| --stream-host | mdw | Host name through which the Master in YMatrix connects to MatrixGate. This is for systems with multiple network interfaces |
| --use-gzip | auto | Whether to enable compression when MatrixGate sends data to Segments. Configurable values are auto/yes/no:The auto setting prefers the zstd compression algorithm;--use-gzip=yes means using the gzip compression algorithm--use-gzip=no disables compressed transfer. Although this saves a small amount of CPU, it greatly increases the volume of network traffic. Unless the database is deployed on a single host and mxgate and the database are all on the same host, avoid the no setting and use the default auto |
| --max-seg-conn | 128 | Number of Segments started when external tables pull data from MatrixGate. Increasing this parameter consumes more network connection resources |
| --timing | false | When this parameter is set to true, MatrixGate adds timing information to each INSERT in the logs |
| --insert-timeout | 600000 | Timeout for MatrixGate to execute an INSERT statement.When set to a value greater than 0, it times out after the configured number of milliseconds |
| -I / --instrumentation | disable | Enables detection of slot(s); supports the disable / single / all options:disable disables this feature;single enables detection for slot[0] only;all enables detection for all slots |
--bytes-limit |
Size limit for batch data loading. It keeps ingested data uniform when the data stream input to MatrixGate is uneven. This feature is disabled by default; to enable it, configure the size manually. The value range is 0~INT_MAX |
|
--auto-tune |
false | When this parameter is set to true, MatrixGate can adjust the number of slots for write jobs |
--abort-by-pause-timeout |
10000 | After a job is paused with the Pause command, once the timeout specified by this parameter is reached, the data accumulated in memory and pending to be written to the database is automatically discarded. The value range is 0~INT_MAX; the recommended range is 1000~10000, in milliseconds (ms). If set to 0, this feature is disabled. It is recommended to set this value clearly lower than --request-timeout, so that --request-timeout remains the timeout when the job is not paused. Once the job is paused, mxgate automatically compares the two timeout parameters and triggers a timeout error according to the lower one. This parameter is configurable only when writer="stream" |
Note!
The parameters in this section are those that can be run on the command line after mxgate is started.
| Parameter Name | Sub-parameter Name | Description |
| run | Run MatrixGate from the command line | |
| start | Start the MatrixGate background process | |
| stop | Terminate the MatrixGate background process | |
| -f / --force | Forcefully terminate the MatrixGate process | |
| --grpc-port | Terminate the MatrixGate process listening on the specified port by sending a gRPC request | |
| status | Print the status of the MatrixGate background process | |
| config | Print the complete configuration file | |
| log | Show the latest 10 log entries | |
| -n / --lines int | Specifies how many of the latest log entries to show; the int parameter cannot be omitted | |
| version | Show the version | |
| help | Show usage and the parameter list | |
| -C / --config | Load a configuration file; currently supported as a sub-parameter of start / run | |
| -p / --pid | ID of the MatrixGate process; currently supported as a sub-parameter of stop / log / watch / pause / resume | |
| --job | The job to set parameters for. Since each job corresponds to a table, you can specify the corresponding table name in practice; currently supported as a sub-parameter of stop / log / watch / pause / resume | |
| set | --stream-prepared-cli | Set the number of active write slots in a write job ࠾ The default of --stream-prepared-cli is 10 ࠾ Example: mxgate set --stream-prepared-cli 3 |
| --job-interval | Set the ETL (Extract-Transform-Load) working time of a write job (in milliseconds, ms) ࠾ The default of --job-interval is 100 ࠾ Example: mxgate set --job-interval 150 |
|
| --high-water-mark | Set the high-water mark threshold (write load) of the target job ࠾ The default of --high-water-mark is 0 ࠾ Example: mxgate set --high-water-mark 20 |
|
| --disable-high-water-mark | Set whether to disable the high-water mark threshold ࠾ The default of --disable-high-water-mark is false ࠾ Example: mxgate set --disable-high-water-mark=true |
|
--auto-tune | Set whether to enable automatic slot count adjustment ࠾ Enable for all jobs: mxgate set --auto-tune=true ࠾ Disable for all jobs: mxgate set --auto-tune=false ࠾ Enable for a specific job: mxgate set --auto-tune=true --job public.t1 |
|
| get | --stream-prepared-get | Get the number of write slots in a write job ࠾ Example: mxgate get --stream-prepared-get |
| --stream-status-get | Get the status of the write slots in a write job ࠾ Example: mxgate get --stream-status-get |
|
| --job-interval-get | Get the working time of a write job ࠾ Example: mxgate get --job-interval-get |
|
| --job-list | Get information about all write jobs ࠾ Example: mxgate get --job-list |
|
| --job-state | Get the status of all write jobs ࠾ Example: mxgate get --job-state |
|
| --current-config | Get the configuration of the current MatrixGate process ࠾ Example: mxgate get --current-config |
|
| --kafka-topics | Get all test topics (Kafka Topics) ࠾ Example: mxgate get --kafka-topics |
|
| --kafka-messages | Get all messages of a Kafka topic ࠾ Example: mxgate get --kafka-messages |
|
| --source-pressure | Get the source-side data write load of the target job ࠾ Example: mxgate get --source-pressure |
|
| --high-water-mark-get | Get the high-water mark threshold (write load) set for the target job ࠾ Example: mxgate get --high-water-mark-get |
|
--auto-tune-status | Get the status of automatic slot count adjustment for write jobs ࠾ Get the status for all write jobs: mxgate get --auto-tune-status ࠾ Get the status for a specific job: mxgate get --auto-tune-status --job public.t1 |
|
| pause | -X / --disconnect | When pausing mxgate, whether to disconnect the slots between Segments and mxgate (this interrupts all slots of all jobs) ࠾ Example: mxgate pause -X |
| -S / --sync | Without this sub-parameter, mxgate pauses asynchronously. When this sub-parameter is explicitly configured, mxgate waits for all write jobs to finish pausing before returning the result ࠾ Example: mxgate pause -S |
|
| resume | -R / --reload | When resuming writes, whether to reload the metadata of the job ࠾ Example: mxgate resume -R |
| offset-reset | Reset the consumer offsets of a stopped Kafka data stream; when there are multiple data streams, first use the selection parameters to locate one (a single stream is selected automatically) ࠾ Example: mxgate offset-reset --topic csv_test --to-earliest --dry-run ࠾ See "Resetting Kafka Consumer Offsets" in Kafka Integration |
|
| --topic / --consumer-group / --job | Locates the data stream to reset; when these are configured, they must exactly match one flow. Omit `--consumer-group` when there is no consumer group | |
| --to-earliest | Reset to the earliest offset still retained by Kafka | |
| --to-latest | Reset to the latest offset (skip backlog) | |
| --to-datetime | Reset to the first message at or after the specified date and time | |
| --to-offsets | Specify offsets by partition, for example `'0:100,1:200'` | |
| --dry-run / --output / -y | Print the reset plan only and change nothing; `--output json` outputs a single JSON object and must be used with `--dry-run` or `-y`; `-y` skips interactive confirmation and is suitable for automation | |
| watch | ||
| -D / --daemon-addr | gRPC service address (host:port) of the MatrixGate process | -i / --info | Continuously print the meaning of each column of the metrics shown by mxgate watch ࠾ Example: mxgate watch --info |
| -T / --time | Specifies how long the monitoring metrics are displayed (in seconds, s) ࠾ Default is -1, meaning continuous printing ࠾ Example: mxgate watch --time 200 |
|
| -H / --history | Show historical metrics ࠾ Example: mxgate watch --history |
|
| --watch-latency | Continuously print Latency-related monitoring metrics ࠾ Example: mxgate watch --watch-latency |
|
| --watch-start | View historical start records; by default within 24 hours before the current time, for example 2022-04-11 11:09:20 ࠾ Example: mxgate watch --watch-start |
|
| --watch-end | View historical end records; by default the current time, for example 2022-04-11 11:09:20 ࠾ Example: mxgate watch --watch-end |
|
| --watch-duration | View historical durations ࠾ Example: mxgate watch --watch-duration |
Note!
For an introduction to the main features of MatrixGate, see Main Features of MatrixGate.
