ClickHouse Backup - RSYNC

[karthik-thiyagarajan]

I've a primary cluster and passive clickhouse cluster and i wanted to do daily / hourly synch from primary to passive cluster.

In our use case, clickhouse-backup takes approximately 2.5 hours to take backup and 2 hours to do the restoration for a volume size of 900 Gi total. However, when i do the rsync it is much faster that it takes hardly 10 to 15 seconds for 5 GB (though i understand that there might be some inconsistencies for the ongoing operations).

However, simple rsynch is unable to have the clickhouse in the passive cluster detect the new table data, and the select table does not detect new tables or table data . Can you let me know how do i make this happen ?

[Slach]

try to use increment backup instead of full snapshot (look readme.md about upload --diff-from-remote or watch command) together with CLICKHOUSE_USE_EMBEDDED_BACKUP_RESTORE=true environment variable (or config option use_embdded_backup_restore in clickhouse section

[karthik-thiyagarajan]

do you use kubernetes?

Yes - we have deployed CH in Kubernetes Cluster.

[Slach]

Two approaches are valid for k8s in cross-dc.

The first approach is more Kubernetes native: deploy a cross-dc pod network something like that https://github.com/submariner-io/submariner and allow the DC2 ClickHouse cluster use zookeeper in DC1

The second approach is more complex on clickhouse side, but without changing ifrastructure

• create Load Balancer k8s service in DC2

• create manually via spec.files cluster DC1 in DC1 kind: ClickHouseInstallation manifests where define only one node which point to this load balancer dns name (look kubectl get svc)

• add MV to each table with destination table engine=Distributed which use DC2 cluster and use engine=Distributed tables as underlying (MV->Distributed DC1 pointed to LB DC2-> undelying Distributed in DC2)

p.s.
Look to costs in your cloud provider about cross DC traffic

[karthik-thiyagarajan]

Thanks. This helps definitely - is it not possible to do solution #1 without sharing the zookeeper ? We are having the DC1 / DC2 approach only for handling the use cases when DC1 is not accessible and in that we dont want to have dependency on Zookeeper from DC1 for the DC2 to work good.

I prefer embedded backup as a bummer with watch - but we also wanted to have continouous restoration done on DC2 for every few hours - and i see "watch" is not available for restoration - which is leaving the room to start thinking about rsync now.

What is your thought process on having rsync done between DC1 and DC2 for both ZK and CH ?

[Slach]

is it not possible to do solution #1 without sharing the zookeeper ?

no way, keeper is coordinator for data replication

watch + use_embbeded_backup_restore + and remote storage on DC2 (minio or something similar) could help, but this is a bad decision when you will try to use backup solution as data replication, this is wrong use case for clickhouse-backup

What is your thought process on having rsync done between DC1 and DC2 for both ZK and CH ?

you can do it, but you can't apply it without restart, zookeeper and clickhouse
and you need more aggressive fsync options for CH in this case

[karthik-thiyagarajan]

Got it - thank you very much for your help. This helps.