YMatrix 5 Release

5.3.11

Date: 2025-09-23

Bug Fixes

• Fixed several known issues

5.3.10

Date: 2025-09-10

Bug Fixes

• Fixed several known issues

5.3.9

Date: 2025-05-22

Bug Fixes

• Fixed several known issues

5.3.8

date: 2024-12-30

Enhancements and Improvements

1. Storage Engine
   • Added RangeFile verification logic for ColumnStore (#IB7V4S)
2. MatrixShift
   • When there are duplicate tables or parent tables and some child tables appear in table-list in the scope paragraph of the configuration file, the ability to automatically remove duplicate tables and report warnings (#72ffcea)
   • A stricter check was introduced for the configuration file, and the wrong configuration item will be reported and prompted when mxshift is started. For example, disable-connector=true should be written in the [scope] paragraph. If it is written incorrectly in other paragraphs, an error will be reported (#IB5002)
3. MatrixGate
   • Data transfer between mxgate and segment nodes does not enable compression by default to save more memory resources. The original default value is Enable compression (#IBD0PW)
   • In transfer mode, allow the mirror node to move data when it is in Down state (#IBCQPB)
4. MatrixUI
   • When installing or expanding a new machine, check whether there are AllowUsers/AllowGroups/DenyUsers/DenyGroups and other configuration items in /etc/ssh/sshd_config that may cause the ssh mutual trust operation of the host in the cluster to fail during the installation process, and give specific error information and modification prompts (#IB80HF)
5. mxbox
   • When using mxrecover to perform a rebalance operation on the cluster, the database downtime caused by this operation is shortened by adding CHECKPOINT. At the same time, the --skip-checkpoint parameter is added to skip this behavior (#IBCVOL)
   • Improved output of this command in console when using mxrecover to perform rebalance operations on the cluster (#IBCVOS)
   • mxrecover added --prepare-timeout parameter. When mxrecover starts, the timeout for the cluster status information can be collected through etcd, default is 1 minute (#IBCVOS)
   • The default timeout of mxstart is increased from 1 minute to 5 minutes, and it still supports users to customize the timeout with the -t parameter (#9ab2957c)
   • Support user-defined environment variables when the postmaster process starts (#39575ad3)
   • When a double failure appears in the cluster, it allows the nearest primary node (#IBCF5U) ​​to be started by restarting the database (#IBCF5U)
6. High availability
   • The newly deployed cluster, its supervisor starts etcd parameter removes the --enable-pprof option, which can improve the stability of multi-node large-scale clusters (#44edf1e)
   • When the localhost entry is deliberately deleted from the user's /etc/hosts file, the supervisor can also start normally after YMatrix is ​​installed (#10435b7)
   • When the host restarts or runs YMatrix for the first time, it can automatically create the relevant paths that the resource group cgroup depends on (#IB5G9O)
   • Improved the number of max open files for supervisor service and etcd process, solving the problem of frequent too many open files errors in super-large clusters (#6ca5293dcd0)

Bug Fix

• Fixed several known issues

5.3.7

Date: 2024-11-27

Enhancements and Improvements

1. Storage Engine
   • Fixed an issue where mars3 freeze failed to handle index transactions correctly (#IB51MP)
   • Fixed the issue where mars3 freeze cannot freeze too large RUN
   • Fixed the issue of mars3 run recycle disorder (#IB5CR4)
   • Fixed an issue where AlteredTableInfo cannot be deserialized correctly (#IB6980)
   • When adding GUC mars3_orderkey_contain_partkey, allow partition keys under unique mode to be used without being included in the sort key (#IALONR)
   • Fixed the issue where the not null attribute of the first sort key can be changed under unique mode (#IAMQGP)
   • Allows operations to add primary key on a single node (#IALONR)
2. MatrixGate
   • mxgate optimization: Supports segments connection timeout as configurable GUC (#IB25VC)
3. MatrixShift
   • mxshift optimization: /tmp use data dir when there is no permission (#IB0UY1)
   • mxshift optimization: migration from gp6 data can be skipped external table (#IB0IUE)
   • mxshift optimization: compresslevel is forced to 1 when compresstype is automatically mapped from zlib to zstd (#IB0GA9)
4. mxbox
   • Fixed a bug where the cache caused the shard status to not be updated
   • Fixed processing when etcd compaction causes the current version of meta information to be eliminated
   • Fixed the bug where the primary node in an abnormal state was not stopped in time when the network partition was abnormally restored.
   • moveseg/initstandby tool detects whether the host added by the --add subcommand is already in the physical machine cluster and gives an error message
5. MatrixUI
   • In the query load page, the calls_end value sum after 0 causes a query error (#IAMZI0)

5.3.6

Date: 2024-11-01

Enhancements and Improvements

1. Execution Engine

   • Optimize the run pick logic of TableCleanJob

Bug Fix

• Fixed several known issues

5.3.5

Date: 2024-09-05

Enhancements and Improvements

1. Storage engine:
   • Fixed the problem of dead loop when rowstore insert in uniquemode (#IAFOX2)
   • Fixed the heap tuple layout issue in rowstore in uniquemode
   • Fixed old version compatibility issues for ColumnStore
   • Fixed the issue where mars3 worker does not exit when dropdb
   • Fixed the problem that mars3 btree results due to key truncate in uniquemode
   • mars3: Fix crash problem caused by fullmatch not taking into account qual
   • mars3: Added autoprobe mechanism to automatically compress historical level
   • mars3: Fixed the issue of compaction outputting empty run

5.3.4

Date: 2024-07-19

Enhancements and Improvements

1. Storage engine:
   • Fixed the issue with mars3 Rescan crash (#IA4SAA)
   • Fixed crash issue with mars3 VindexScan in conditional index case (#IA4YOQ)
   • Fixed crash issue with mars3 BitmapScan in case of BitmapAnd (#IA7A16)
2. Execution Engine:
   • Fixed the fast distributor supports named cursors under extended mode (#7536)
   • Fixed the hnsw index error of the pgvector plugin, which caused it to be unusable (#7547)
   • Optimized conversion of row data to column data in mars3 storage engine (#7515)
   • Fixed memory leaks when vectorized aggregation operators dealing with partial types (#7503)
   • Fixed an issue where vectorized scan operators might crash when loading text columns on aocs (#IA5N8H)
   • Fixed processing of NULL values ​​in vectorized IS DISTINCT FROM expressions (#IA5APT)
   • Fixed projection processing on grouping keys in vectorized hash aggregation operator v1 version (#IA5OL6)
   • Fixed an issue where the alignment is used when the alter table is added to the column in the mars3 storage engine (#IA4RPX)
   • Fixed the issue of type use when deserializing the vectorized hash aggregation operator v2 version (#IAA9QI)
   • Supports scenarios where vectorized versions in filters are empty set sums (IA4R4Z)
   • Optimized process of vectorized seqscan path (IA4R4Z)
   • Enhanced vectorized motion check (#IA4RPK)
   • Fixed kv issue in mxkv2 deserialization expression (#IA95IP)
   • Fixed an issue when a row of inputs in mxkv2 has duplicate keys (#IA9OW3)
   • Fixed the issue where the Subquery node cannot be vectorized (IAAXTQ)
   • Fixed vectorized executor fallback expr evaluation issue (#IA9ZXC, IAAISQ)
   • Optimized vectorized executor check whether expressions are supported (#IAA5FL)
   • Fixed the hash condistion selection issue for vectorized executor hash aggregation (#IAA5FL)
   • Fixed the problem of unrecognized join type: 10 (#IA9Z4J)
   • Fixed the issue that partition table distinct agg may report an error variable not found in subplan target list (#IAAWE4)
   • Fixed the issue of Motion to the wrong node or triggering crash when the table update with segment_set is set (#IACDGV, IAD7W9)
   • Fixed CVE-2024-0985: PostgreSQL non-owner REFRESH MATERIALIZED VIEW CONCURRENTLY executes arbitrary SQL (#IA6KD7)
   • Fixed CVE-2023-39417: Extension script @substitutions@ within quoting allow SQL injection (#IA6KCP)
   • Fixed CVE-2023-5870: Role "pg_signal_backend" can signal certain superuser processes (#IA6KC9)
   • Fixed CVE-2023-5869: Buffer overrun from integer overflow in array modification (#IA6KBU)
   • Fixed CVE-2023-5868: Memory disclosure in aggregate function calls (#IA6KBE)
   • Fixed CVE-2023-2455: Row security policies disregard user ID changes after inlining (#IA6KA6)
   • Fixed CVE-2023-2454: CREATE SCHEMA ... schema_element defeats protected search_path changes (#IA6K9X)
   • Fixed CVE-2022-2625: Extension scripts replace objects not belonging to the extension (#IA6K8P)
   • Fixed CVE-2021-32029: Memory disclosure in partitioned-table UPDATE ... RETURNING (#IA6K7G)
   • Fixed CVE-2021-32028: Memory disclosure in INSERT ... ON CONFLICT ... DO UPDATE (#IA6K76)
   • Fixed CVE-2021-32027: Buffer overrun from integer overflow in array subscription calculations (#IA6K6K)
   • Fixed CVE-2021-23222: libpq processes unencrypted bytes from man-in-the-middle (#IA6K5Z)
   • Fixed CVE-2021-23214: Server processes unencrypted bytes from man-in-the-middle (#IA6K5D)
   • Fixed CVE-2021-3677: Memory disclosure in certain queries (#IA6K4Z)
   • Fixed CVE-2021-3393: Partition constraint violation errors leak values ​​of denied columns (#IA6JTZ)
   • Fixed CVE-2020-14350: Uncontrolled search path element in CREATE EXTENSION (#IA6JQD)
   • Fixed CVE-2020-14349: Uncontrolled search path element in logical replication (#IA6JPP)
   • Fixed CVE-2020-1720: ALTER ... DEPENDS ON EXTENSION is missing authorization checks (#IA6JOV)
3. MatrixUI
   • Fixed the issue of incorrect cluster metric time display on Dashboard page (#IABL52)
   • Fixed the issue of failing to save data acquisition configuration on load analysis page in Ubuntu20 (#IAAFSS)
   • Fixed the problem of self-service inspection of some shell script reports in Ubuntu20 (#IA9SZW)
   • Fixed security vulnerability issue for UI runtime dependencies (#IAD077)
4. MatrixGate
   • Fixed process memory leaks in versions after 5.2.2 (#IAD077)

5.3.3

Date: 2024-06-14

Enhancements and Improvements

1. Tools
   • In transfer mode, when mxgate connects to the source database, set idle_in_transaction_session_timeout = 0 to prevent the failure of the entire data transmission due to the idle connection timeout that has completed the data transmission in advance. (#7506)
   • Add automatic partitioning to insert catalog tables and insert error log tables for different data sources. (#7499)
2. Storage engine
   • MARS3 adds a new insertion mode "single" to obtain lower insertion delay.
   • Optimize the creation of CVs with long-term blocking problems with Insert. (#7266)
   • Optimize the in-lock time of mutexes in ParallelScan in MARS3.
3. Execution Engine
   • Enhanced vectorized window operator, adding support for most window functions and window characteristics. (#6751)
   • Optimized performance of vectorized window operator. (#6751)
   • Enhanced vectorized aggregation operator to support INTERNAL types. (#6881)
   • Enhanced support for runtime filter. (#7049)
   • Enhanced single node distribution table to support any primary or unique keys. (#7125)
   • Enhanced single node distribution table to support arbitrary triggers. (#7127)
   • Enhanced support for ON CONFLICT clauses by single-node distribution table. (#7168)
   • Enhanced fast distributor to support explicit transactions. (#7124)
   • Enhanced fast distributor to support UPDATE & INSERT & DELETE operations. (#7124)
   • Enhanced fast distributor to support processing multiple statements at once. (#7388)
   • Enhanced support for single-node distribution tables by fast distributors. (#7130)
   • Enhanced support for replication tables by fast distributors. (#7131)
   • Enhanced fast distributor to support auto-increment types. (#7202)
   • Optimized performance of truncate ON COMMIT DELETE ROWS temporary tables. (#7279)
   • Supports the update table_name set table_name.column_name syntax.
   • When creating tables in masteronly mode, there is no need to check the node's separate information. (#7519)
   • matrixmgr: Supports statistics query whether it is fastdd mode, and transactions are first-stage and second-stage transactions. (#7470)
   • Enhanced fast distributors support prepare statment and extended mode. (#7139)
   • Added yonyou extension: implementing implicit conversions of ROWNUM, to_char, text and other types (#7374 #7375 #7376)

Bug Fix

• Fixed several known issues

5.3.2

Date: 2024-04-19

Enhancements and Improvements1. Graphical interface (MatrixUI)

• Added a MatrixGate process monitoring page. Provides intuitive graphical performance indicator display to simplify performance analysis.

2. MatrixGate
   • mxgate_process_view Added the mxgate_config field. ([Document](/en/doc/5.3/tools/mxgate/feature#11%20mxgate%20 data write process information, insert system tables and error log tables))
   • The default value of the --metrics-sample-interval parameter under the [metrics] category has been adjusted from 3 seconds to 15 seconds. ([Document](/en/doc/5.3/tools/mxgate/parameter#1%20 configuration file parameters))
   • pause action If the timeout (50 seconds), the insertion operation corresponding to Slot is cancelled to ensure that the pause operation can complete normally. ([Document](/en/doc/5.3/tools/mxgate/feature#4%20 updates the table structure without shutting down))
   • Adjust pause and resume in the mxgate command help manual (mxgate --help) from arguments to commands.
   • A large number of the same run logs will be combined to display (printed once in 10 seconds).
3. Vectorized execution engine
   • Upgraded vectorized query planner to v3. (#3427)
   • Upgraded vectorized aggregation (Agg) operator to v2. (#3328)
   • Optimized support for distinct operations by vectorized aggregation operator. (#3355,#3386,#3410)

Bug Fix

• Fixed several known issues

5.3.1

Date: 2024-03-29

Enhancements and Improvements

1. Continuous Gathering (CV)
   • The default index of the underlying storage for continuous clustering is changed from mars3_brin (v5.2.0-v5.3.0) to mars3_btree (starting with v5.3.1).

Bug Fix

• Fixed several known issues

5.3.0

Date: 2024-03-22

Feature Release

1. Graphical interface (MatrixUI)
   • Added a new homepage, providing resource indicator analysis and query statistical analysis functions.
   • Added an English version interface.
2. Data node instance (Segment) migration tool mxmoveseg
   • Supports migration of standby master nodes (Standby). (Document)
3. MatrixGate
   • Support querying mxgate data to write process information, insert system tables and error logs. ([Document](/en/doc/5.3/tools/mxgate/feature#11%20mxgate%20 data write process information, insert system tables and error log tables))
   • Fine the --log-dir parameter specifies the timing information in the mxgate log under the directory (usually /home/mxadmin/gpAdminLogs). Adjust the original 4 time periods to 5: t1: Slot insert The time when the connection was established; t2: Slot waits for scheduled t3: Segment Time to receive data from mxgate t4: Data redistribution between segments and dropping t5: insert transaction commit time.

Enhancements and Improvements

1. Data node instance (Segment) migration tool mxmoveseg (Document)
   • When the source migration server is offline, the offline old server can also be restarted after the migration is completed, and the cluster exception will not be caused.
2. High availability
   • Fixed a double failure issue in a cluster with Standby/Mirror configured (all data instances under a data shard (including a Primary and its corresponding Mirror) are offline).
3. mxshift feature enhancement
   • Supports data migration and verification decoupling. ([Document](/en/doc/5.3/tools/mxshift#3.4.1%20 Old version of data verification and data migration decoupling))
   • Upgrade the data consistency verification algorithm before and after migration. In addition to verifying by configuring the --verify parameter under the [transfer] category, the [verify] category has been added, and the upgraded verification can be performed by configuring the parameters in it. The two verification methods are compatible with each other. ([Document](/en/doc/5.3/tools/mxshift#3.4.2%20v5.2.1%20 version data verification))
4. Vectorized execution engine function enhancement
   • Implement vectorized shared scanning operator.
   • Implement vectorized deduplication operator.
   • Optimized stability of vectorized actuators when using ORCA optimizers.
   • Optimized performance of vectorized join operators when joining right.
5. MARS3 storage engine functionality enhancement
   • Added support for btree index by MARS3 storage engine.
   • Optimized stability of MARS3 storage when combined with ORCA.
6. MatrixGate feature enhancement
   • Supports termination of mxgate process listening to a specified port by sending a gRPC request. The corresponding command is: mxgate stop --grpc-port [port]. ([Document](/en/doc/5.3/tools/mxgate/parameter#2%20 command line parameters))

Bug Fix

• Fixed several known issues

5.2.2

Date: 2024-01-16

Bug Fixes

• Fixed several known issues

5.2.1

Date: 2024-01-02

Enhancements and Improvements

1. MXKV2
   • Optimized MXKV2 memory usage (#3178).
   • Optimized MXKV2 compression performance (#3180).
   • Optimized MXKV2 data import performance (#3162).
2. Vectorized Execution Engine
   • Optimized memory usage of vectorized motion operators in multi-column queries (#3175).
   • Optimized performance of external hash algorithms for vectorized hash aggregates (#3213).
   • Optimized the efficiency of sequential comparisons for text types (#3163).

Bug fixes

• Fixed several known issues

5.2.0

Date: 2023-12-08

Feature Release

1. Graphical interface
   • Supports graphical installation and deployment of YMatrix on Tongxin UOS20, Dragon Lizard 8.X, Kirin Xinan 3.X, and Zhongbiao Kirin 7.X operating systems.
   • Added SQL executor function.
   • Added load analysis function (document).
2. Vectorization execution engine (document)
   • Support docking ORCA optimizer. HEAP, AO storage engine supports direct docking, while MARS2/3 does not support docking. In particular, AOCO can connect to vectorized execution engines and ORCAs at the same time.
3. Added a downgrading storage function, supporting automatic separation of hot and cold data, and downgrading cold data to more economical object storage (Document).
4. Added the data type MXKV2, which simplified the use steps and improved compression performance based on MXKV (document).
5. Added the pgvector plug-in tool to support vector scenarios (document).

Enhancements and Improvements

1. Continuous aggregation parameter populate default value is changed to false ([Document](/en/doc/5.2/datamodel/cv#2.5%20 query Continuous aggregation view)).
2. If the original two RPMs are combined into one, the volume may become larger. However, users do not need to install the DebugInfo package, which is more beneficial for debugging and analysis of the business environment.
3. MatrixGate
   • Supports the ability to enable or turn off the automatic adjustment of slot number for specified tasks, and supports querying the adjustment status of all tasks or specified tasks ([Document](/en/doc/5.2/tools/mxgate/feature#9%20mxgate%20automatic adjustment of %20slot%20number)).
   • Introduce the --abort-by-pause-timeout parameter under the [writer.stream] category. Supports the data accumulated in memory when the Job enters the pause state according to the timeout specified by this parameter (Document).
   • Supports batch writing of JSON type data over HTTP (document).
4. Graphical interface
   • Optimized interaction of etcd storage path configuration steps in multi-node deployment function ([document](/en/doc/5.2/install/mx5_cluster/mx5_cluster_centos7#4%20 database deployment)).
   • Optimized steps and interactions of Kafka data stream writing function ([Document](/en/doc/5.2/datainput/kafka#2%20 Create %20Kafka%20 data stream)).
5. Data migration tool mxshift
   • DDL (Data Definition Language) migration process allows the execution to be continued by ignoring the specified error. The type of error that is ignored can be configured with the --ignore-error parameter ([document](/en/doc/5.2/tools/mxshift# configuration file)).
   • Simplifies permissions for the migration process, and only configures the source cluster access to the Master node of the target cluster. To turn off this feature, configure --disable-connector=true ([Document](/en/doc/5.2/maintain/migrate/best_practice_gp6_to_ymatrix5#1.10%20 to add a whitelist to the %20Master%20 of the source cluster)).
   • Support log separation, the format difference between the logs output by the terminal and the logs saved by the file are optimized, and the user experience is optimized.
   • Supports the migration table meta information is written to the specified file by configuring the --write-table-meta-file parameter, and the information in the file is read directly through the --read-table-meta-file parameter during the subsequent migration process to shorten the business downtime caused by migration ([Document](/en/doc/5.2/tools/mxshift#3.1%20 import/export table meta information)).
   • Added the --small-table-mb parameter, and configuring this parameter reasonably can optimize the transmission speed of small tables ([document](/en/doc/5.2/tools/mxshift# configuration file)).
   • Support dynamic adjustment of concurrency count ([Document](/en/doc/5.2/tools/mxshift#3.3%20 dynamic adjustment of concurrency count)).

Bug Fix

• Fixed several known issues

5.1.3

Date: 2023-09-25

Bug Fixes

• Fixed several known issues

5.1.2

Date: 2023-09-11

Bug Fixes

• Fixed several known issues

5.1.1

Date: 2023-08-25

Bug Fixes

• Fixed several known issues

5.1.0

Date: 2023-08-22

Feature Releases

1. Added the MARS3 storage engine (Documentation), which, compared to MARS2,
   • Implements row-column co-storage, writing data rows to disk, significantly improving write performance
   • Enables table creation without the need to create additional indexes
   • Supports data updates and deletions, column additions and removals, and the MVCC mechanism
   • Not only does it inherit the performance advantages of MARS2 in time series scenarios, but it also adds support for AP and TP scenarios.
2. New mirror instance migration tool mxmoveseg (Documentation)
   • Supports migration of any database instance that is currently a mirror to other servers.
   • Does not support migration of standbys.
3. Enhanced functionality of the data migration tool mxshift (Documentation)
   • Supports incremental migration: migrates the data definition language (DDL) and data of selected tables from a Greenplum 4.3.33/5/6 or YMatrix cluster to a target database in a new YMatrix cluster in parallel
   • Supports specifying the IP address corresponding to the hostname via parameters under the [database.source.hostname-to-ip] category to ensure routing accuracy
   • Added the install-dir parameter to replace the gphome parameter
4. MatrixGate feature enhancements
   • Supports executing DDL modification commands immediately during the write process (Documentation)
   • Supports automatically adjusting the number of connections based on the load of different write tasks (Documentation)
   • Supports quantifying the write load of source-end data as a percentage value (Documentation)
   • Supports limiting the size of batch data loading (Documentation)
5. Enhanced encoding chain compression algorithm functionality (Documentation: [/en/doc/5.1/reference/storage/compression])
   • Added Gorilla2 and fds algorithms
6. Graphical interface upgrade (Documentation: /en/doc/5.1/maintain/self_service_inspection)
   • Added self-service inspection page, supporting the creation of inspection plans by selecting inspection options and generating detailed inspection reports
7. Support for domestic platforms OpenEuler 20.03-22.03, Fusion 22-23, Kylin Security V3.3-3.5, Anolis OS 8, and Tongxin V20 (1060a/1060e) support (Enterprise Edition features) (Documentation: /en/doc/5.1/install/mx5_cluster/mx5_cluster)

错误修复1. Storage Engine

• Fixed a crash caused by the AOCS storage engine during index scan
• Fixed a crash caused by MARS2 storage engine defining more than 32 minmax indexes

2. MatrixGate
   • Fixed memory overuse issues caused by MatrixGate when starting compression
   • Fixed the issue of MatrixGate's internal logic errors caused by connection timeout
3. MatrixManager
   • Fixed memory overuse problem of MatrixMgr due to long-term failure to release connections
4. Execution Engine
   • Fixed a crash caused by mxvector due to no alignment when processing NULL values
   • Fixed the crash problem caused by handling InitPlan in parallel mode
   • Fixed the problem of calculating statistics in the partition table
5. Tools
   • Fixed the data verification error caused by the mxaddmirrors command when handling checksum on AO storage
   • Fixed an error when executing DDL operations in mxshift

5.0.1

Date: 2023-06-06

Bug Fix

• Fixed several known issues

5.0.0

Date: 2023-03-31

Feature Release

1. A brand new database architecture

   • Based on the Raft algorithm, realize automatic fault transfer between the cluster master node (Master) and the master node standby node (Standby) (Document)
   • Introducing a service-oriented architecture to make the deployment, operation and maintenance of clusters simpler and more convenient (document)
   • Added cluster management tools: mxaddmirrors(document), mxdeletes(document), mxstop(document), mxrecover(document), mxstart(document), mxstate(document), mxstop(document))
   • Support for deployment of monitoring for etcd clusters (document)

2. Added a vectorized execution engine (document)

   • Support batch processing: using batch processing can effectively reduce the execution overhead of function calls and other
   • Make full use of hardware and compiler capabilities: Use CPU cache friendly algorithms to make full use of hardware and compiler optimization capabilities, such as generating SIMD (Single Instruction, Multiple Data) instructions
   • Supports multiple operators: Scan, Sort, Aggregate, Join, Window Function, Sub Query, Limit, Motion and other operators are implemented. There are also different algorithm implementations for the same operator, effectively covering most query scenarios and improving query performance

   Based on the above three core principles, vectorized execution engines can achieve performance improvements of 1 to 2 orders of magnitude compared to traditional execution engines oriented towards row processing.

3. Added Runtime Filter query performance optimization technology (Document)

   • RuntimeFilter is one of the key technologies to improve the performance of YMatrix execution engine. It refers to a filter (Filter) that is dynamically built when valuating in the execution engine after the optimizer generates a physical execution plan, which is different from filters pre-planned by the optimizer.

4. The performance of MARS2 storage engine is greatly optimized (document)

   • Optimize compression strategies to reduce memory usage
   • Optimize partition table writing and reduce memory usage
   • Optimize query filtering effect to reduce unnecessary I/O consumption
   • Optimize data volume estimation and make more accurate selection of execution plans
   • Enhanced aggregation capability, support uniquemode and IN expressions

5. Added a new coded chain compression algorithm (document)

   • Supports multiple encoding and compression algorithms: supports lz4, zstd, simple8b, doubledelta, gorilla, deltazigzag, floatint and other encoding and compression algorithms. The above algorithms can be flexibly combined to fully improve the compression rate
   • Supports column-level or table-level custom compression algorithms
   • Support adaptive encoding
   • Used with vectorized execution engine to improve data compression/decompression speed: In SSB scenarios, the decompression speed is 3 times that of LZ4

   The coding chain compression scheme can greatly reduce space usage and reduce costs. In the new energy vehicle scenario, the compression rate is 3.9 times that of LZ4 compression algorithm scheme, and 2.7 times that of LZ4 in the TSBS (Time Series Benchmark Suite) scenario.

6. Storage diagnostic tool Datainspect upgrade (document)

   • Added a new function desc_compress, which can compare and select the best two specified compression algorithms

7. MatrixGate feature enhancement

   • Automatically switch the service to the corresponding healthy backup node when the automatic failure transfer mechanism is started (document)
   • Add detailed monitoring when writing data, which is convenient for debugging when problems are found. You can enable the -I startup parameter (Document)
   • A series of optimizations were made for the stdin loading scenario, and the data loading performance was twice as high as 4.8.0

8. Added smooth capacity expansion function (document)

   • Supports expansion tasks without shutting down
   • Support custom data node sets (Segment Set) to realize redistribution of partial tables (Document)
   • Supports parallel execution of table data redistribution
   • Supports connection of unredistributed tables and redistributed tables
   • Supports one-click smooth expansion of graphical interface (Document)

9. Data migration tool mxshift feature enhancement (document)

   • Migrate data table DDL in database units, and you can choose whether to migrate only DDL

10. Graphical interface upgrade

    • Add query monitoring pages, support slow query information displayed on custom management pages, and can terminate slow query with one click (Document)
    • Add write test function to experience YMatrix's powerful data writing performance (document)
    • Add query testing function, support custom query testing schemes according to business scenarios, and experience strong query performance of YMatrix in high concurrency scenarios with one click (Document)
    • The deployment page is completely upgraded, and the steps are simpler (Document)
    • The expansion page has been completely upgraded, with smooth expansion function added, and the steps are more lightweight (Document)
    • Added high availability status detection function (document)

11. Add new policy to automatic partition management auto_partitioning_ex (document)

    • Support batch creation of partitions
    • Support forcing the retention of specific historical partitions
    • Support custom automatic partition operation period