Skip to content

Commit f9d9ca3

Browse files
jwierzbojwierzbo
authored
[Documentation] Manual recovery documentation (#1281)
1 parent 6a3dde9 commit f9d9ca3

File tree

3 files changed

+272
-1
lines changed

3 files changed

+272
-1
lines changed

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
- (Improvement) Add new field to CR for more precise calculation of DC2DC replication progress
2424
- (Maintenance) Bump GO Modules
2525
- (Feature) Optional Graceful Restart
26+
- (Maintenance) Manual Recovery documentation
2627

2728
## [1.2.24](https://github.com/arangodb/kube-arangodb/tree/1.2.24) (2023-01-25)
2829
- (Bugfix) Fix deployment creation on ARM64

‎docs/design/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@
1616
- [Configuring timezone](./configuring_tz.md)
1717
- [Operator API](./api.md)
1818
- [Logging](./logging.md)
19-
19+
- [Manual Recovery](./recovery.md)

‎docs/design/recovery.md

+270
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
# Manual Recovery
2+
3+
## Overview
4+
Let's consider a situation where we had a ArangoDeployment in Cluster mode (3 DbServers, 3 Coordinators, 3 Agents)
5+
with Local storage attached (only one K8s Node in the K8s cluster).
6+
7+
Due to some reason the ArangoDeployment was deleted (e.g. ETCD storage has been wiped out) and we want to recover it.
8+
Fortunately, we have a backup of the data on the disk.
9+
10+
To recover the ArangoDeployment we need to:
11+
1. Create PV and PVC for each member with persistent storage (agent, dbservers, single)
12+
2. Create a new ArangoDeployment with the same members IDs
13+
14+
## Local storage data
15+
16+
We have a members (Agents & DbServers) data in the following directories:
17+
```bash
18+
> ls -1 /var/data/
19+
f9rs2htwc9e0bzme
20+
fepwdnnbf0keylgx
21+
gqnkahucthoaityt
22+
vka6ic19qcl1y3ec
23+
rhlf8vixbsbewefo
24+
rlzl467vfgsdpofu
25+
```
26+
27+
To find out the name of the members to which data should be attached,
28+
we need to check the `UUID` file content in each directory:
29+
```bash
30+
> cat /var/data/*/UUID
31+
AGNT-pntg5yc8
32+
AGNT-kfyuj8ow
33+
AGNT-bv5rofcz
34+
PRMR-9xztmg4t
35+
PRMR-l1pp19yl
36+
PRMR-31akmzrp
37+
```
38+
39+
## Initial ArangoDeployment
40+
41+
Here is an example of the initial ArangoDeployment before deletion:
42+
```yaml
43+
cat <<EOF | kubectl apply -f -
44+
apiVersion: "database.arangodb.com/v1"
45+
kind: "ArangoDeployment"
46+
metadata:
47+
name: "cluster"
48+
spec:
49+
externalAccess:
50+
type: NodePort
51+
mode: Cluster
52+
agents:
53+
volumeClaimTemplate:
54+
spec:
55+
storageClassName: my-local-ssd
56+
accessModes:
57+
- ReadWriteOnce
58+
resources:
59+
requests:
60+
storage: 1Gi
61+
volumeMode: Filesystem
62+
dbservers:
63+
volumeClaimTemplate:
64+
spec:
65+
storageClassName: my-local-ssd
66+
accessModes:
67+
- ReadWriteOnce
68+
resources:
69+
requests:
70+
storage: 1Gi
71+
volumeMode: Filesystem
72+
EOF
73+
```
74+
75+
## Create PV and PVC
76+
77+
1. We need to create ArangoLocalStorage first:
78+
```yaml
79+
cat <<EOF | kubectl apply -f -
80+
apiVersion: "storage.arangodb.com/v1alpha"
81+
kind: "ArangoLocalStorage"
82+
metadata:
83+
name: "local-storage"
84+
spec:
85+
storageClass:
86+
name: my-local-ssd
87+
isDefault: true
88+
localPath:
89+
- /mnt/data
90+
EOF
91+
```
92+
2. Now create PV and PVC for every directory listed above
93+
- Agents - here is an example for `AGNT-pntg5yc8`(`f9rs2htwc9e0bzme` directory)
94+
- PV
95+
```yaml
96+
cat <<EOF | kubectl apply -f -
97+
apiVersion: "v1"
98+
kind: PersistentVolume
99+
metadata:
100+
labels:
101+
arango_deployment: cluster
102+
role: agent
103+
name: agent-pntg5yc8-f9rs2htwc9e0bzme
104+
spec:
105+
accessModes:
106+
- ReadWriteOnce
107+
capacity:
108+
storage: 1Gi
109+
local:
110+
path: /mnt/data/f9rs2htwc9e0bzme
111+
persistentVolumeReclaimPolicy: Retain
112+
storageClassName: my-local-ssd
113+
volumeMode: Filesystem
114+
nodeAffinity:
115+
required:
116+
nodeSelectorTerms:
117+
- matchExpressions:
118+
- key: kubernetes.io/hostname
119+
operator: In
120+
values:
121+
- minikube
122+
EOF
123+
```
124+
- PVC
125+
```yaml
126+
cat <<EOF | kubectl apply -f -
127+
apiVersion: v1
128+
kind: PersistentVolumeClaim
129+
metadata:
130+
labels:
131+
app: arangodb
132+
arango_deployment: cluster
133+
role: agent
134+
name: agent-pntg5yc8
135+
spec:
136+
accessModes:
137+
- ReadWriteOnce
138+
resources:
139+
requests:
140+
storage: 1Gi
141+
storageClassName: my-local-ssd
142+
volumeMode: Filesystem
143+
volumeName: agent-pntg5yc8-f9rs2htwc9e0bzme
144+
EOF
145+
```
146+
- DbServers - here is an example for `PRMR-9xztmg4t` (`vka6ic19qcl1y3ec` directory)
147+
- PV
148+
```yaml
149+
cat <<EOF | kubectl apply -f -
150+
apiVersion: "v1"
151+
kind: PersistentVolume
152+
metadata:
153+
labels:
154+
arango_deployment: cluster
155+
role: dbserver
156+
name: dbserver-9xztmg4t-vka6ic19qcl1y3ec
157+
spec:
158+
accessModes:
159+
- ReadWriteOnce
160+
capacity:
161+
storage: 1Gi
162+
local:
163+
path: /mnt/data/vka6ic19qcl1y3ec
164+
persistentVolumeReclaimPolicy: Retain
165+
storageClassName: my-local-ssd
166+
volumeMode: Filesystem
167+
nodeAffinity:
168+
required:
169+
nodeSelectorTerms:
170+
- matchExpressions:
171+
- key: kubernetes.io/hostname
172+
operator: In
173+
values:
174+
- minikube
175+
EOF
176+
```
177+
- PVC
178+
```yaml
179+
cat <<EOF | kubectl apply -f -
180+
apiVersion: v1
181+
kind: PersistentVolumeClaim
182+
metadata:
183+
labels:
184+
app: arangodb
185+
arango_deployment: cluster
186+
role: dbserver
187+
name: dbserver-9xztmg4t
188+
spec:
189+
accessModes:
190+
- ReadWriteOnce
191+
resources:
192+
requests:
193+
storage: 1Gi
194+
storageClassName: my-local-ssd
195+
volumeMode: Filesystem
196+
volumeName: dbserver-9xztmg4t-vka6ic19qcl1y3ec
197+
EOF
198+
```
199+
200+
### Create ArangoDeployment with previously created PVC
201+
202+
Now we can create ArangoDeployment with previously created PVCs:
203+
```yaml
204+
cat <<EOF | kubectl apply -f -
205+
apiVersion: "database.arangodb.com/v1"
206+
kind: "ArangoDeployment"
207+
metadata:
208+
name: "cluster"
209+
spec:
210+
externalAccess:
211+
type: NodePort
212+
mode: Cluster
213+
agents:
214+
volumeClaimTemplate:
215+
spec:
216+
storageClassName: my-local-ssd
217+
accessModes:
218+
- ReadWriteOnce
219+
resources:
220+
requests:
221+
storage: 1Gi
222+
volumeMode: Filesystem
223+
dbservers:
224+
volumeClaimTemplate:
225+
spec:
226+
storageClassName: my-local-ssd
227+
accessModes:
228+
- ReadWriteOnce
229+
resources:
230+
requests:
231+
storage: 1Gi
232+
volumeMode: Filesystem
233+
status:
234+
agency:
235+
ids:
236+
- AGNT-pntg5yc8
237+
- AGNT-kfyuj8ow
238+
- AGNT-bv5rofcz
239+
size: 3
240+
members:
241+
agents:
242+
- id: AGNT-pntg5yc8
243+
persistentVolumeClaim:
244+
name: agent-pntg5yc8
245+
persistentVolumeClaimName: agent-pntg5yc8
246+
- id: AGNT-kfyuj8ow
247+
persistentVolumeClaim:
248+
name: agent-kfyuj8ow
249+
persistentVolumeClaimName: agent-kfyuj8ow
250+
- id: AGNT-bv5rofcz
251+
persistentVolumeClaim:
252+
name: agent-bv5rofcz
253+
persistentVolumeClaimName: agent-bv5rofcz
254+
dbservers:
255+
- id: PRMR-9xztmg4t
256+
persistentVolumeClaim:
257+
name: cluster-dbserver-9xztmg4t
258+
persistentVolumeClaimName: cluster-dbserver-9xztmg4t
259+
- id: PRMR-l1pp19yl
260+
persistentVolumeClaim:
261+
name: cluster-dbserver-l1pp19yl
262+
persistentVolumeClaimName: cluster-dbserver-l1pp19yl
263+
- id: PRMR-31akmzrp
264+
persistentVolumeClaim:
265+
name: cluster-dbserver-31akmzrp
266+
persistentVolumeClaimName: cluster-dbserver-31akmzrp
267+
EOF
268+
```
269+
270+
That's it! Now you can use ArangoDB with restored data.

0 commit comments

Comments
 (0)