Redis 集群

Redis 是 Remote Dictionary Service 远程字典服务，它在近年来风靡一时，不仅限于传统的 memcached 之类的缓存，还通过其丰富的数据结构支持许多其他的应用，比如少量数据存储，分布式锁，简单的 pub/sub 服务等等。

当然单台 Redis 肯定不满足高可用性的要求，Redis 的 HA 方案主要有两种

Redis Sentinel 的扩展性不好，所以从略，主要以 Redis Cluster 为例，它将所有的数据划分为 16384 个槽位 slot, 每个节点负责一部分, 如上图，每对redis 节点分配三分之一的槽位。

当然客户端连接时需要获取槽位的配置信息才知道到哪个节点上去存取数据，这个配置信息由于节点可能增减或者崩溃，所以也不一定准，在访问到不匹配的节点时 ，Redis 会返回 MOVED 响应让客户端去其他节点

构建 Redis 集群

Redis 5.0 自身就提供了构建 redis 的命令，我用 Fabric 简单写一个脚本，封装了繁琐的命令，用来快速构建 Redis Cluster

为简单起见，我在一台 Ubuntu Server 上生成 6个 Redis 实例来模拟6 台 Redis 服务器节点

fab generate_config
启动 6个redis 实例
fab start_redis
创建包含这6个redis 实例的 Redis 集群
fab create_redis_cluster
准备工作:
找一台 ubuntu server
到 Redis 官网上下载 redis 包
安装 python3 和 fabric3 (pip install fabric3)

（有关 Fabric 的介绍，请参见旧文 程序员瑞士军刀之 Fabric）
这个 fabfile.py 脚本的源代码如下
from fabric.api import *
from fabric.api import settings
from fabric.context_managers import *
from fabric.contrib.console import confirm
import os, subprocess
redis_path = '/home/walter/package/redis-5.0.8/src'
redis_config = '''daemonize yes
bind 0.0.0.0
port 9001
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
@task
def clean_config():
    for port in range(9001,9007):
        local("rm -rf {}".format(port))
@task
def write_config(file_path, port):
    config_content = redis_config.replace('9001', str(port))
    with open(file_path, "w") as fp:
        fp.write(config_content)
@task
def generate_config():
    for port in range(9001,9007):
        local("mkdir -p {}".format(port))
        config_file = '{}/redis.conf'.format(port)
        print("write {}".format(config_file))
        write_config(config_file, port)
@task
def start_redis():
    for folder in range(9001,9007):
        with lcd(str(folder)):
            local("{}/redis-server ./redis.conf".format(redis_path))
@task
def stop_redis():
    cmd = redis_path + "/redis-cli -p {} shutdown nosave"
    for port in range(9001,9007):
        local(cmd.format(port))
@task
def kill_redis():
    cmd = "ps -efw --width 1024|grep redis-server |grep -v grep|awk '{print $2}"
    pids = subprocess.check_output(cmd, shell=True)
    print(pids)
    with settings(warn_only=True):
        for pid in pids.decode("utf-8").split('\n'):
            local("kill -9 {}".format(pid))
@task
def check_ports_mac():
    with settings(warn_only=True):
        for port in range(9001,9007):
            local("lsof -nP -iTCP:{} | grep LISTEN".format(port))
@task
def check_redis():
    cmd = "ps -ef|grep redis-server |grep -v grep"
    with settings(warn_only=True):
        local(cmd)
@task
def create_redis_cluster():
    cmd = redis_path + "/redis-cli --cluster create {} {}"
    host_and_ports = ""
    for port in range(9001,9007):
            host_and_ports = host_and_ports + "0.0.0.0:{} ".format(port)
    option = "--cluster-replicas 1"
    local(cmd.format(host_and_ports, option))
@task
def redis_cli(command=''):
    if command:
        local(redis_path + "/redis-cli -c -p 9001 %s" % command)
    else:
        local(redis_path + "/redis-cli -p 9001")
注意将以上脚本中的 redis_path 改为自己的 redis 源代码路径

redis 代码下载后直接执行 make 就可以编译生成可执行文件

将以上的 fabfile 放在一个任一个目录下，执行
fab generate_config
fab start_redis
fab create_redis_cluster
即可生成  Redis 集群
$ ps -ef|grep redis|grep -v grep
root     19702     1  0 Jun06 ?        00:12:42 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9001 [cluster]
root     19708     1  0 Jun06 ?        00:12:38 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9002 [cluster]
root     19714     1  0 Jun06 ?        00:12:41 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9003 [cluster]
root     19720     1  0 Jun06 ?        00:14:27 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9004 [cluster]
root     19726     1  0 Jun06 ?        00:14:20 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9005 [cluster]
root     19732     1  0 Jun06 ?        00:14:25 /home/walter/package/redis-5.0.8/src/redis-server 0.0.0.0:9006 [cluster]
root@node5:~#                                                                                                            
通过Redis 的命令行 redis-cli 可以察看集群内部的信息：




    

察看集群中的节点信息
fab redis_cli:"cluster nodes"
[localhost] local: /home/walter/package/redis-5.0.8/src/redis-cli -c -p 9001 cluster nodes
be6213517632bdc1dc21ecdc6db99718ad996227 127.0.0.1:9006@19006 slave 5999be7b420c0b5efadb3adaca8d1fc96b6a2494 0 1592020538000 6 connected
907f0344f54a5df28ef1cf548da32e64cfab8d16 127.0.0.1:9001@19001 myself,master - 0 1592020538000 1 connected 0-5460
50f321d0f3007da039f9350930b176f2b22ec1e0 127.0.0.1:9004@19004 slave 5f4f1fe7432dff359ea5b05f8997432ae726f7bd 0 1592020537000 4 connected
ba918ea80644efca8405b8480c2ccb941ed831b0 127.0.0.1:9005@19005 slave 907f0344f54a5df28ef1cf548da32e64cfab8d16 0 1592020537378 5 connected
5999be7b420c0b5efadb3adaca8d1fc96b6a2494 127.0.0.1:9002@19002 master - 0 1592020537578 2 connected 5461-10922
5f4f1fe7432dff359ea5b05f8997432ae726f7bd 127.0.0.1:9003@19003 master - 0 1592020538380 3 connected 10923-16383
察看整个集群的状态
fab redis_cli:"cluster info"
[localhost] local: /home/walter/package/redis-5.0.8/src/redis-cli -c -p 9001 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1053876
cluster_stats_messages_pong_sent:1054292
cluster_stats_messages_sent:2108168
cluster_stats_messages_ping_received:1054292
cluster_stats_messages_pong_received:1053871
cluster_stats_messages_received:2108163
察看集群中的 slots
 fab redis_cli:"cluster slots"
[localhost] local: /home/walter/package/redis-5.0.8/src/redis-cli -c -p 9001 cluster slots
1) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 9001
      3) "907f0344f54a5df28ef1cf548da32e64cfab8d16"
   4) 1) "127.0.0.1"
      2) (integer) 9005
      3) "ba918ea80644efca8405b8480c2ccb941ed831b0"
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 9002
      3) "5999be7b420c0b5efadb3adaca8d1fc96b6a2494"
   4) 1) "127.0.0.1"
      2) (integer) 9006
      3) "be6213517632bdc1dc21ecdc6db99718ad996227"
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 9003
      3) "5f4f1fe7432dff359ea5b05f8997432ae726f7bd"
   4) 1) "127.0.0.1"
      2) (integer) 9004
      3) "50f321d0f3007da039f9350930b176f2b22ec1e0"
如果我们想知道某个 key 属于哪一个 slot ，也可以用命令 cluster KEYSLOT $key 来查询，例如 “hello” 通过如下命令可以查出它属于 slot 866, 应该落在 “ 127.0.0.1:9001” 这个实例上。
fab redis_cli:"cluster KEYSLOT hello"
[localhost] local: /home/walter/package/redis-5.0.8/src/redis-cli -c -p 9001 cluster KEYSLOT hello
(integer) 866
访问 Redis 集群
Redis 的 RESP 协议非常简单明了，几乎各种语言都有相应的库可以访问  Redis

以 Python 为例, 创建如下两个文件
requirements.txt
loguru
redis
hiredis
redis-py-cluster
redis-test.py
import sys
from rediscluster import RedisCluster
import redis
from redis.client import Redis
from loguru import logger
logger.add(sys.stderr,
           format="{time} {message}",
           filter="client",
           level="INFO")
logger.add('logs/redis_client_{time:YYYY-MM-DD}.log',
           format="{time} {level} {message}",
           filter="client",
           level="ERROR")
class RedisClient:
    def __init__(self, connection_string, password=None):
        self.startup_nodes = []
        nodes = connection_string.split(',')
        for node in nodes:
            host_port = node.split(':')
            self.startup_nodes.append({'host': host_port[0], 'port': host_port[1]})
        self.password = password
        logger.info(self.startup_nodes)
        self.redis_pool = None
        self.redis_instance = None
        self.redis_cluster = None
    def connect(self):
        if(len(self.startup_nodes) < 2):
            host = self.startup_nodes[0].get('host')
            port = self.startup_nodes[0].get('port')
            if self.password:
                self.redis_pool = redis.ConnectionPool(host=host, port=port, db=0)
            else:
                self.redis_pool = redis.ConnectionPool(host=host, port=port, password = self.password, db=0)
            self.redis_instance = Redis(connection_pool=self.redis_pool, decode_responses=True)
            return self.redis_instance
        #, skip_full_coverage_check=True
        self.redis_cluster = RedisCluster(startup_nodes=self.startup_nodes, password=self.password)
        return self.redis_cluster
def quick_test():
    client = RedisClient("10.224.112.73:9001")
    conn = client.connect()
    key = "hello"
    value = "world"
    conn.set(key, value)
    conn.expire(key, 300)
    ret = conn.get(key)
    logger.info("value={}", ret)
    conn.hsetnx("walter", "age", 30)
    conn.hsetnx("walter", "gender", 'male')
    conn.expire(key, 300)
    values = conn.hgetall("walter")
    for key, value in values.items():
        logger.info("{}={}".format(key, value))
if __name__ == "__main__":
    quick_test()
pip install -r requirements.txt
python redis-test.py
输出结果如下
2020-06-13 13:05:52.071 | INFO     | __main__:__init__:27 - [{'host': '10.224.112.73', 'port': '9001'}]
2020-06-13 13:05:52.229 | INFO     | __main__:test_cluster:110 - value=b'world'
2020-06-13 13:05:52.354 | INFO     | __main__:test_cluster:118 - b'age'=b'30'
2020-06-13 13:05:52.354 | INFO     | __main__:test_cluster:118 - b'gender'=b'male'
redis 的命令行工具可以获取 Redis 的各种度量信息，除了上面提到的 cluster 相关命令，还有很多，比如最常用的 info 命令可以获取如下9大块的信息
Server
Clients
Memory
Persistennce
Stats
Replication
Cluster
Keyspace
fab redis_cli:info
[localhost] local: /home/walter/package/redis-5.0.8/src/redis-cli -c -p 9001 info 
#-------------------------------
# Server 服务器信息
#-------------------------------
redis_version:5.0.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:391103cbae5277b6
redis_mode:cluster
os:Linux 4.4.0-135-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:5.4.0
process_id:19702
run_id:0c3d2c132eefbf68a23dab136cdc79999b6c8ca3
tcp_port:9001
uptime_in_seconds:1444
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:14386466
executable:/home/walter/package/redis-5.0.8/src/redis-server
config_file:/home/walter/mdd/oss/redis/9001/./redis.conf
#-------------------------------
# Clients 客户端信息
#-------------------------------
connected_clients:1
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0
#-------------------------------
# Memory 内存信息
#-------------------------------
used_memory:2652664
used_memory_human:2.53M
used_memory_rss:5267456
used_memory_rss_human:5.02M
used_memory_peak:2693616
used_memory_peak_human:2.57M
used_memory_peak_perc:98.48%
used_memory_overhead:2578384
used_memory_startup:1463192
used_memory_dataset:74280
used_memory_dataset_perc:6.24%
allocator_allocated:2638792
allocator_active:2822144
allocator_resident:5382144
total_system_memory:8370958336
total_system_memory_human:7.80G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.07
allocator_frag_bytes:183352
allocator_rss_ratio:1.91
allocator_rss_bytes:2560000
rss_overhead_ratio:0.98
rss_overhead_bytes:-114688
mem_fragmentation_ratio:2.03
mem_fragmentation_bytes:2677952
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:16922
mem_clients_normal:49694
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
#----------------------------------
# Persistence 持久化信息
#----------------------------------
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1591443327
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:274432
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:0
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
#----------------------------------
# Stats 统计信息
#----------------------------------
total_connections_received:4
total_commands_processed:1448
instantaneous_ops_per_sec:0
total_net_input_bytes:52741
total_net_output_bytes:14030
instantaneous_input_kbps:0.02
instantaneous_output_kbps:0.01
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:178
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
#----------------------------------
# Replication 复制信息
#----------------------------------
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=9005,state=online,offset=2016,lag=1
master_replid:d633853458a2973c12ab79442bc807d35e387f5d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:2016
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:2016
#----------------------------------
# CPU 中央处理器信息
#----------------------------------
used_cpu_sys:0.912000
used_cpu_user:0.844000
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
#----------------------------------
# Cluster 集群信息
#----------------------------------
cluster_enabled:1
但普通用户不可能经常登录服务器执行，通过服务器端的 agent 来定时采样和收集 redis 所在服务器及其 redis 自身的度量数据是实际中的常用做法。
我曾经用过两种方案：
MetricBeat --> ElasticSearch -> Kibana
Telegraf --> Influxdb --> Grafana
还有其他基于 Collectd, Prometheus 的方案，按下不表.
在实践中，为了防止发送端和接收端的吞吐量不匹配的问题，常用 Kafka 来中转，这里先介绍 Telegraf --> Influxdb --> Grafana 这种方案。
Telegraf

安装参照 https://docs.influxdata.com/telegraf/v1.14/introduction/downloading/

以 Ubuntu16 系统为例：
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update
sudo apt-get install telegraf
sudo systemctl start telegraf
启动之后会有以下默认的配置
2020-06-13T10:28:56Z I! Starting Telegraf 1.14.4
2020-06-13T10:28:56Z I! Using config file: /etc/telegraf/telegraf.conf
2020-06-13T10:28:56Z I! Loaded inputs: diskio kernel mem processes swap system cpu disk
2020-06-13T10:28:56Z I! Loaded aggregators:
2020-06-13T10:28:56Z I! Loaded processors:
2020-06-13T10:28:56Z I! Loaded outputs: influxdb
配置文件位于 /etc/telegraf/telegraf.conf, 配置项非常多，不过主要的也就几条
- input  度量数据从哪里来
- output  度量数据到哪里去
- aggregators 聚合器
- processors 处理器
InfluxDB

有关 InfluxDb 的介绍请参见旧文度量驱动开发之 InfluxDB
先快速安装它
sudo apt-get install influxdb
sudo systemctl unmask influxdb.service
sudo systemctl start influxdb
用 influx 命令察看一下 telegraf 默认创建的度量表
 influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> show databases
name: databases
telegraf
_internal
> use telegraf
Using database telegraf
> show measurements
name: measurements
diskio
kernel
processes
system
因为我们要监控是的 Redis cluster, 所以我们需要更改  telegraf 的配置, 通过它的 Redis 监控插件来获取  Redis 的度量数据
参见 https://github.com/influxdata/telegraf/blob/release-1.14/plugins/inputs/redis/README.md
The Redis input plugin gathers the results of the INFO Redis command. There are two separate measurements: redis and redis_keyspace, the latter is used for gathering database-related statistics.

Additionally the plugin also calculates the hit/miss ratio (keyspace_hitrate) and the elapsed time since the last RDB save (rdb_last_save_time_elapsed).
可以直接修改 vi /etc/telegraf/telegraf.conf 或者直接生成一个新的配置文件
telegraf --input-filter redis:cpu:mem:net:swap --output-filter influxdb config > telegraf.conf
主要是增加了这两行
[[inputs.redis]]
servers = ["tcp://127.0.0.1:9001"] 
重新启动 telegraf
sudo systemctl stop influxdb
sudo systemctl start influxdb
这样我们再运行 influx 命令行可以看到出现了4 张新表
redis
redis_cmdstat
redis_keyspace
redis_replication
influx -database='telegraf' -execute='show measurements'
name: measurements
diskio
kernel
processes
redis
redis_cmdstat
redis_keyspace
redis_replication
system
看看 redis measurement 的结构，我们之前在 redis-cli info 的输出看到的在 influxdb 的 redis measurement 都能看到
influx -database='telegraf' -execute='SHOW TAG KEYS FROM "redis"'
name: redis
tagKey
------
replication_role
server
root@node5:~# influx -database='telegraf' -execute='SHOW FIELD KEYS FROM "redis"'
name: redis
fieldKey                        fieldType
--------                        ---------
active_defrag_hits              integer
active_defrag_key_hits          integer
active_defrag_key_misses        integer
active_defrag_misses            integer
active_defrag_running           integer
allocator_active                integer
allocator_allocated             integer
allocator_frag_bytes            integer
allocator_frag_ratio            float
allocator_resident              integer
allocator_rss_bytes             integer
allocator_rss_ratio             float
aof_base_size                   integer
aof_buffer_length               integer
aof_current_rewrite_time_sec    integer
aof_current_size                integer
aof_delayed_fsync               integer
aof_enabled                     integer
aof_last_bgrewrite_status       string
aof_last_cow_size               integer
aof_last_rewrite_time_sec       integer
aof_last_write_status           string
aof_pending_bio_fsync           integer
aof_pending_rewrite             integer
aof_rewrite_buffer_length       integer
aof_rewrite_in_progress         integer
aof_rewrite_scheduled           integer
blocked_clients                 integer
client_recent_max_input_buffer  integer
client_recent_max_output_buffer integer
clients                         integer
cluster_enabled                 integer
connected_slaves                integer
evicted_keys                    integer
expired_keys                    integer
expired_stale_perc              float
expired_time_cap_reached_count  integer
instantaneous_input_kbps        float
instantaneous_ops_per_sec       integer
instantaneous_output_kbps       float
keyspace_hitrate                float
keyspace_hits                   integer
keyspace_misses                 integer
latest_fork_usec                integer
lazyfree_pending_objects        integer
loading                         integer
lru_clock                       integer
master_repl_offset              integer
maxmemory                       integer
maxmemory_policy                string
mem_aof_buffer                  integer
mem_clients_normal              integer
mem_clients_slaves              integer
mem_fragmentation_bytes         integer
mem_fragmentation_ratio         float
mem_not_counted_for_evict       integer
mem_replication_backlog         integer
migrate_cached_sockets          integer
number_of_cached_scripts        integer
pubsub_channels                 integer
pubsub_patterns                 integer
rdb_bgsave_in_progress          integer
rdb_changes_since_last_save     integer
rdb_current_bgsave_time_sec     integer
rdb_last_bgsave_status          string
rdb_last_bgsave_time_sec        integer
rdb_last_cow_size               integer
rdb_last_save_time              integer
rdb_last_save_time_elapsed      integer
redis_version                   string
rejected_connections            integer
repl_backlog_active             integer
repl_backlog_first_byte_offset  integer
repl_backlog_histlen            integer
repl_backlog_size               integer
rss_overhead_bytes              integer
rss_overhead_ratio              float
second_repl_offset              integer
slave_expires_tracked_keys      integer
sync_full                       integer
sync_partial_err                integer
sync_partial_ok                 integer
total_commands_processed        integer
total_connections_received      integer
total_net_input_bytes           integer
total_net_output_bytes          integer
total_system_memory             integer
uptime                          integer
used_cpu_sys                    float
used_cpu_sys_children           float
used_cpu_user                   float
used_cpu_user_children          float
used_memory                     integer
used_memory_dataset             integer
used_memory_dataset_perc        float
used_memory_lua                 integer
used_memory_overhead            integer
used_memory_peak                integer
used_memory_peak_perc           float
used_memory_rss                 integer
used_memory_scripts             integer
used_memory_startup             integer
root@node5:~#                                      
Grafana
接下来就可以安装 Grafana 以 InfluxDb 为数据源，进行基于时间序列的数据展示和分析，并添加相应的报警规则
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana
访问  http://10.20.30.40:3000/

默认密码是 admin/admin, 登录之后第一件事是配置数据源
Grafana 已经有许多现成的仪表盘可以借用，例如

https://grafana.com/grafana/dashboards/6908

就是一个Grafana 对于 telegraf 所收集到的度量数据所建立的仪表盘。
当然你也可以创建一个适合自己的图表，例如