Skip to content

K8SPSMDB-1219 Documented support for multiple storages #231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions docs/backups.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,40 @@
# About backups

You can backup your data in two ways:
You can back up your data in two ways:

* *On-demand*. You can do them manually at any moment.
* *Scheduled backups*. Configure backups and their schedule in the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml). The Operator makes them automatically according to the specified schedule.

To make backups and restores, the Operator uses the [Percona Backup for MongoDB :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb) tool.
To make backups and restores, the Operator uses the [Percona Backup for MongoDB (PBM) :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb) tool. The Operator runs PBM as [a sidecar container](sidecar.md) to the database Pods. It configures PBM in the following cases:

* when it creates a new cluster if you defined the [backup storage configuration](backups-storage.md) for it.
* when you configure the backup storage for a backup
* when you [start a restore on a new cluster](backups-restore-to-new-cluster.md) and defined the backup storage configuration within the `backupSource` subsection of the Restore resource.

## Backup storage

You can store Percona Server for MongoDB backups outside the Kubernetes
cluster using the following remote backup storages:

* [Amazon S3 or S3-compatible storage :octicons-link-external-16:](https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services),
* [MinIO :octicons-link-external-16:](https://min.io/) S3-compatible storage
* [Azure Blob Storage :octicons-link-external-16:](https://azure.microsoft.com/en-us/services/storage/blobs/)

![image](assets/images/backup-cloud.svg)

### Multiple backup storages

Starting with version 1.20.0, the Operator natively supports [multiple backup storages :octicons-link-external-16:](https://docs.percona.com/percona-backup-mongodb/features/multi-storage.html), inheriting this feature from Percona Backup for MongoDB (PBM). This means you don't have to wait till the Operator reconfigures a cluster after you select a different storage for a backup or a restore. And you can make a point-in-time recovery from any backup stored on any storage - PBM and the Operator maintain the data consistency for you.

Find more information in the [Multiple storages for backups](multi-storage.md) chapter.

## Backup types

<a name="physical"></a> The Operator can do either *logical* or *physical* backups.
The Operator can do either *logical* or *physical* backups.

* *Logical backup* means querying the Percona Server for MongoDB for the database data and writing the retrieved data to the remote backup storage.
* *Logical backup* means querying Percona Server for MongoDB for the database data and writing the retrieved data to the remote backup storage.

* *Physical backup* means copying physical files from the Percona Server for MongoDB `dbPath` data directory to the remote backup storage.
* *Physical backup* means copying physical files from Percona Server for MongoDB `dbPath` data directory to the remote backup storage.

Logical backups use less storage, but are much slower than physical backup/restore.

Expand Down
66 changes: 66 additions & 0 deletions docs/multi-storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Multiple storages for backups

You can define several storage locations for backups in the Operator. However, previously you were limited to only a single storage for point-in-time recovery, because Percona Backup for MongoDB (PBM) couldn't maintain oplog consistency across multiple storages. Also, you had to wait for the Operator to reconfigure the cluster and sync metadata after you make the next backup or a restore to a different storage.

This behavior is improved. The Operator differentiates the storages as the main storage and profiles. The difference between them is that the Operator uses the main storage to save both backups and oplog chunks for point-in-time recovery. Profiles are used only for backups. This is done for data consistency and to enable point-in-time recovery from a backup on any storage.

## Define the main storage

When you configure only one storage, the Operator automatically uses it as the main one until you add more. When you add another storage, you must mark which one is the main using the `main: true` flag in the `deploy/cr.yaml` Custom Resource manifest.

```yaml
storages:
s3-us-west:
main: true
type: s3
```

Note that you can have only one main storage. All other storages are added as profiles.

To check the list of profiles, connect to the database Pod and run the `pbm profile list` command. For example, for the cluster `cluster1`, the command looks as follows:

```{.bash .data-prompt="$"}
$ kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile list
```

You can run other [profile management commands :octicons-link-external-16:](https://docs.percona.com/percona-backup-mongodb/reference/pbm-commands.html#pbm-profile-add) in the same way.

## Change the main storage

You can change the main storage by reassigning the `main:true` flag for another one. The Operator then:

* Resyncs the metadata for the new main storage
* Deletes the profile for it
* Adds the previous main storage as a profile

## Pass storage configuration via restore objects

Usually you define the storage configuration within the `deploy/cr.yaml` Custom Resource manifest. You can also pass it to the Operator within the `backupSource` option of a Restore object. For example, when you restore the failed site after a disaster. The Operator then checks the current configuration and:

* If there is no storage configured there, it uses the one from the Restore object as the main storage until you define it in the `deploy/cr.yaml` file.
* If the `deploy/cr.yaml` Custom Resource manifest has the storage configured and it differs from the one from the Restore object, the Operator adds this storage as a profile.

## Backup metadata resync

The Operator resyncs the metadata in the following cases:

For the main storage:

* When the main storage changes
* When you manually start a resync using the `pbm config --force resync`

For the profile storage:

* When you start a restore from a backup on a profile
* When you manually resync the metadata on a profile using the `pbm profile sync <storage-name>` command

For the main storage and profiles:

* When you added an annotation `percona.com/resync-pbm=true` to the `deploy/cr.yaml` Custom Resource manifest.

Note that resync is a resource consuming task and we don't recommend to run it manually. Read more when you need to run it in [PBM documentation :octicons-link-external-16:](https://docs.percona.com/percona-backup-mongodb/reference/config.html#syncronize-configuration)

The improved support for multiple backup storages brings the following benefits:

* Enables you to make a point-in-time recovery from any storage with guaranteed data consistency
* Reduces the load on the cluster for reconfiguration when the storage for a next backup changes
10 changes: 10 additions & 0 deletions docs/operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -2460,6 +2460,16 @@ A custom [Kubernetes Security Context for a Container :octicons-link-external-16
| ----------- | ---------- |
| :material-text-long: subdoc | `privileged: false` |

### `backup.storages.STORAGE-NAME.main`

Marks the storage as main. All other storages you define are added as profiles. The Operator saves backups to all storages but it saves oplog chunks for point-in-time recovery only to the main storage. You can define only one storage as main. Read more about [multiple storages for backups](multi-storage.md).


| Value type | Example |
| ----------- | ---------- |
| :material-toggle-switch-outline: boolean | `true` |


### `backup.storages.STORAGE-NAME.type`

The cloud storage type used for backups. Only `s3`, `azure`, and `filesystem` types are supported.
Expand Down
1 change: 1 addition & 0 deletions mkdocs-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@ nav:
- Backup and restore:
- "About backups": backups.md
- "Configure storage for backups": backups-storage.md
- multi-storage.md
- "Making scheduled backups": backups-scheduled.md
- "Making on-demand backup": backups-ondemand.md
- "Storing operations logs for point-in-time recovery": backups-pitr.md
Expand Down