Old Method ¶
Sometimes you need to migrate data from one Redis server to another, or between other databases. Previously, I could transfer data from a server only through a full backup, but this does not allow for migrating data between bases. Here is the algorithm.
- Enter into the original Redis and make Save
1
2
3
| REDISCLI_AUTH="password"
redis-cli CONFIG GET DIR
redis-cli SAVE
|
The first command will print the path to the directory where the dump will be stored.
- Add arguments to the target Redis to disable AppendOnly and restart it
1
2
| --dbfilename dump.rdb
--appendonly no
|
- Copy the dump to the target Redis and execute its loading
1
2
| REDISCLI_AUTH="password"
redis-cli BGREWRITEAOF
|
- Move the target Redis to the AppendOnly mode by removing previously added keys
Why migrate data between bases?
In my old cluster, I had two services, each with their own Redis server. In any Redis there are 16 default databases, why not use #0 for service A and #2 for service B? Here you hit into the migration problem from base to base.
New Method ¶
Based on MIGRATE. The script is presented below. Its advantages are also.
- can migrate between bases
- no need to restart Redis
- no need to copy backups
- There is a check after transferring by calculating the hash sum considering both key contents
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| #!/usr/bin/env sh
set -e
SRC_HOSTNAME='source-redis-hostname'
SRC_PORT='6379'
SRC_PASSWORD='password'
SRC_DATABASE='0'
DEST_HOSTNAME='destination-redis-hostname'
DEST_PORT='6379'
DEST_PASSWORD='password'
DEST_DATABASE='1'
main() {
migrate | xargs -0 printf 'migration status: %s'
CHECKSUM="$(mktemp)"
hash --no-auth-warning -h "${SRC_HOSTNAME}" -p "${SRC_PORT}" \
-a "${SRC_PASSWORD}" -n "${SRC_DATABASE}" | tee "${CHECKSUM}" | xargs -0 printf 'source checksum: %s'
CHECKSUM_VERIFY="${CHECKSUM}" hash --no-auth-warning -h "${DEST_HOSTNAME}" -p "${DEST_PORT}" \
-a "${DEST_PASSWORD}" -n "${DEST_DATABASE}" | xargs -0 printf 'target checksum validation: %s'
rm -f "${CHECKSUM}"
}
hash() {
EXTRA_ARGS="${@}"
redis-cli ${EXTRA_ARGS} --raw --scan | \
sed -r 's/"/\\"/g; s/^(.+)$/GET "\1"/g;' | \
redis-cli ${EXTRA_ARGS} | sort | \
if [ -f "${CHECKSUM_VERIFY}" ]; then
sha256sum -c "${CHECKSUM_VERIFY}"
else
sha256sum -
fi
}
migrate() {
AUTH="--no-auth-warning -h ${SRC_HOSTNAME} -p ${SRC_PORT} -a ${SRC_PASSWORD} -n ${SRC_DATABASE}"
redis-cli ${AUTH} --raw --scan | \
tr '\n' '\0' | \
xargs -0 redis-cli ${AUTH} \
MIGRATE "${DEST_HOSTNAME}" "${DEST_PORT}" "" "${DEST_DATABASE}" 5000 \
COPY REPLACE AUTH "${DEST_PASSWORD}" KEYS
}
main
|
