PXB-3359 - PXC concurrent backups - metadata lock hang with xtrabackup_history

[fmbiete]

Hi Percona,

We have found a hang while executing XtraBackup concurrently in multiple PXC nodes.
We cannot reproduce always the problem, but when it happens we see "metadata lock" in the wsrep_applier thread and the only fix is to kill one of the xtrabackup processes.

The metadata lock corresponds to the "CREATE DATABASE IF NOT EXISTS PERCONA_SCHEMA" statement in XtraBackup code.
It makes us suspect that TOI is causing this issue, because it creates the lock that conflicts with the backup in other nodes, even when the object exists.

Reproduce Scenario
PXC - 3 nodes
Execute XtraBackup with xtrabackup_history enabled, full backup, in all 3 at the same time

Fix
Execute DDL only if the object doesn't exist.

1. Check if database PERCONA_SCHEMA exists - Create if not
2. Check if table XTRABACKUP_HISTORY exists - Create if not
3. Check if column BINLOG_POS has the required type - Alter if not (I believe we don't need this migration anymore)

With those changes the hang is not happening anymore. I will raise a pull request so you can review them.

[satya-bodapati]

Hi @fmbiete I think this is only a band-aid fix. The real problem is DDLs during backup, Backup Locks and TOI locks. Looks like a duplicate of https://perconadev.atlassian.net/browse/PXC-2593

[fmbiete]

Hi Satya,
I'm aware of the DDL limitation with TOI+Backup locks. That's unfixable...
The problem is that xtrabackup itself is executing DDL when saving the history even if it doesn't require it. The backup tool is blocking the backup tool...