111,222均部署keepalived,magent,memcached

keepalived 111为主机，222为备机

其中，111上magent以本地memcache为主，222为备用

222上magen以本地memcache为主，111为备用

 

 

安装memcached

安装：yum -y install memcached查看配置：cat /etc/sysconfig/memcached, 通过ip+下图port可远程连接

　　PORT="11211"

　　USER="memcached"

　　MAXCONN="1024"

　　CACHESIZE="64"

　　OPTIONS=""

启动： systemctl start memcached停止： systemctl stop memcached重启： systemctl restart memcached

连接memcached测试

cmd命令窗口：telnet 10.0.0.111 11211(回车)

ERROR

add aaa 0 0 4

1111

STORED

get aaa

VALUE aaa 0 4

1111

END

安装libevent(略，下次装时补充)

安装magent

mkdir /opt/magentcd magentwget https://files.cnblogs.com/files/lurenjia1994/magent.tar.gztar xzvf magent.tar.gzmake进行安装

可能错误1：

magent.c: 在函数‘writev_list’中:magent.c:729:17: 错误：‘SSIZE_MAX’未声明(在此函数内第一次使用)    if (toSend > SSIZE_MAX ||                 ^magent.c:729:17: 附注：每个未声明的标识符在其出现的函数内只报告一次make: *** [magent.o] 错误 1

解决方法：

vi ketama.h在头部加入以下代码#ifndef SSIZE_MAX#define SSIZE_MAX      32767#endif

可能错误2：

gcc -Wall -g -O2 -I/usr/local/include -m64 -c -o magent.o magent.cgcc -Wall -g -O2 -I/usr/local/include -m64 -c -o ketama.o ketama.cgcc -Wall -g -O2 -I/usr/local/include -m64 -o magent magent.o ketama.o /usr/lib64/libevent.a /usr/lib64/libm.a gcc: 错误：/usr/lib64/libevent.a：没有那个文件或目录gcc: 错误：/usr/lib64/libm.a：没有那个文件或目录make: *** [magent] 错误 1

解决方法：

ln -s /usr/lib64/libm.so /usr/lib64/libm.aln -s /usr/lib64/libevent.so /usr/lib64/libevent.a

检查是否安装成功

[root@localhost magent]# cp magent /usr/bin/[root@localhost magent]# magentplease provide -s "ip:port" argumentmemcached agent v0.6 Build-Date: Aug 22 2018 15:40:29Usage:  -h this message  -u uid  -g gid  -p port, default is 11211. (0 to disable tcp support)  -s ip:port, set memcached server ip and port  -b ip:port, set backup memcached server ip and port  -l ip, local bind ip address, default is 0.0.0.0  -n number, set max connections, default is 4096  -D don't go to background  -k use ketama key allocation algorithm  -f file, unix socket path to listen on. default is off  -i number, set max keep alive connections for one memcached server, default is 20  -v verbose

 

启动magent

[root@centos7 magent]# magent -u root -n 51200 -l 10.0.0.111 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211[root@centos7 magent]# ps -aux|grep magentroot     32007  0.0  0.0  11948   616 ?        Ss   15:47   0:00 magent -u root -n 51200 -l 10.0.0.111 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211root     32466  0.0  0.0 112720   972 pts/1    S+   15:48   0:00 grep --color=auto magent

测试magent

1.连接magent节点

现象：通过magent节点可以取到所有主节点的数据

telnet 10.0.0.111 12000UNSUPPORTED COMMANDget aaa 前面在测试memcache连接设置的aaa,这里取值成功VALUE aaa 0 41111ENDset b 0 0 41234STORED

2.连接主节点

现象：可以取到所有magent节点存入的数据

ERRORget aaaVALUE aaa 0 41111ENDget bVALUE b 0 41234ENDset c 0 0 4ccccSTORED

3.连接备份节点

现象：可以取到所有magent节点存入的数据

ERRORget aaaENDget b #只能取到在magent节点设置的值，其他取不到VALUE b 0 41234ENDget cEND

4.关掉主节点的memcache进程

现象：通过magent节点可以取到所有备份节点的数据

[root@centos7 magent]# systemctl stop memcached[root@centos7 magent]# ps -aux|grep memcachedroot     30827  0.0  0.0 112720   972 pts/1    S+   16:05   0:00 grep --color=auto memcachedcmd连接magent节点telnet 10.0.0.111 12000UNSUPPORTED COMMANDget aaaENDget bVALUE b 0 41234ENDget cEND

5.再开启主节点memcached

现象：取不到前面设置的所有值，因为重启以后，主节点数据全部清空，虽然备份节点有数据，但是主节点在备份节点不会生效

UNSUPPORTED COMMANDget aENDget aaaENDget bENDget cEND

安装keepalived(当10.0.0.111上的magent挂掉时，自动切换到10.0.0.222的magent)

yum install keepalived

修改keepalived配置

本文配置为在这个基础上添加的配置，请忽略多余部分

keepalived配置

! Configuration File for keepalivedglobal_defs {   notification_email {     #acassen@firewall.loc     #failover@firewall.loc     #sysadmin@firewall.loc   }   #notification_email_from Alexandre.Cassen@firewall.loc   #smtp_server 192.168.200.1   #smtp_connect_timeout 30   router_id LVS_DEVEL   vrrp_skip_check_adv_addr   #vrrp_strict   vrrp_garp_interval 0   vrrp_gna_interval 0}vrrp_script chk_nginx {             script "/etc/keepalived/chk_nginx.sh"       interval 2             }vrrp_script chk_magent {             script "/etc/keepalived/chk_magent.sh"       interval 2             }vrrp_instance VI_1 {    state MASTER    interface ens192    virtual_router_id 51    priority 50    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.0.0.233    }    notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111成为主机！ 报！111成为主机！"    notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111已挂！ 10.0.0.111故障！请尽快修复！"    track_script {      chk_nginx                     }}vrrp_instance VI_2 {    state MASTER    interface ens192    virtual_router_id 52    priority 50    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.0.0.234    }    track_script {      chk_magent                     }    }

其中/etc/keepalived目录下的chk_magent.sh

#!/bin/bashrun=`ps -C magent --no-header | wc -l`if [ $run -eq 0 ]; then    magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211    sleep 3    if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then        systemctl stop keepalived    fifi

遇到问题1

绑定不了虚拟vip排查，没有发现虚拟ip 10.0.0.234

只有vrid 51的组播，没有52的组播[root@centos7 keepalived]# tcpdump -nn -c 20 -i any host 224.0.0.18tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes10:46:43.008689 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:44.009046 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:45.009174 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:46.009534 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:47.009703 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:48.010830 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:49.010960 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:50.012050 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:51.012178 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:52.012727 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:53.012828 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:54.013334 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:55.013481 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:56.013858 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:57.014023 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:58.014457 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:46:59.014604 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:47:00.015002 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:47:01.015815 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2010:47:02.016914 IP 10.0.0.111 > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 50, authtype simple, intvl 1s, length 2020 packets captured20 packets received by filter0 packets dropped by kernel

查看日志：cat /var/log/message

似乎是因为脚本导致vip2启动失败

Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233Aug 23 11:17:52 centos7 Keepalived_vrrp[8399]: Sending gratuitous ARP on ens192 for 10.0.0.233Aug 23 11:17:53 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:17:55 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:17:57 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:17:59 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:01 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:03 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:05 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:07 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:09 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:11 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:13 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:15 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:17 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:19 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:21 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:23 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:25 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:27 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:29 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:31 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 11:18:33 centos7 Keepalived_vrrp[8399]: /etc/keepalived/chk_magent.sh exited due to signal 15

难道他是先运行脚本，再开启vip2?

于是把脚本中的10.0.0.234改成10.0.0.111，再重启keepalived，再查看ip，发现已经有10.0.0.234了

[root@centos7 keepalived]# ip addr1: lo: 
      
        mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever2: ens192: 
       
         mtu 1500 qdisc mq state UP group default qlen 1000    link/ether 00:0c:29:c8:03:3f brd ff:ff:ff:ff:ff:ff    inet 10.0.0.111/24 brd 10.0.0.255 scope global noprefixroute ens192       valid_lft forever preferred_lft forever    inet 10.0.0.233/32 scope global ens192       valid_lft forever preferred_lft forever    inet 10.0.0.234/32 scope global ens192       valid_lft forever preferred_lft forever    inet6 fe80::7e90:403b:465:db6c/64 scope link noprefixroute        valid_lft forever preferred_lft forever3: virbr0: 
        
          mtu 1500 qdisc noqueue state DOWN group default qlen 1000    link/ether 52:54:00:70:99:88 brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever4: virbr0-nic: 
         
           mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000    link/ether 52:54:00:70:99:88 brd ff:ff:ff:ff:ff:ff
         
        
       
      

于是在vip1中设置两个虚拟ip：10.0.0.233，10.0.0.234，vip2中不设置虚拟ip，检测脚本启动magent时绑定10.0.0.234，经测试10.0.0.111上部署成功

遇到问题2

10.0.0.222启动keepalived时，magent似乎启动失败

排查：

1.关掉10.0.0.111上的keepalived

2.给111上的keepalived.conf跟chk_magent.sh覆盖掉222对应文件

3.启动

systemctl start keepalived

4.检查地址

[root@localhost keepalived]# ip addr1: lo: 
      
        mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever2: ens192: 
       
         mtu 1500 qdisc mq state UP group default qlen 1000    link/ether 00:0c:29:59:17:90 brd ff:ff:ff:ff:ff:ff    inet 10.0.0.222/24 brd 10.0.0.255 scope global noprefixroute ens192       valid_lft forever preferred_lft forever    inet 10.0.0.233/32 scope global ens192       valid_lft forever preferred_lft forever    inet 10.0.0.234/32 scope global ens192       valid_lft forever preferred_lft forever    inet6 fe80::5e69:554d:d28e:dbb5/64 scope link noprefixroute        valid_lft forever preferred_lft forever3: virbr0: 
        
          mtu 1500 qdisc noqueue state DOWN group default qlen 1000    link/ether 52:54:00:59:38:b4 brd ff:ff:ff:ff:ff:ff    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0       valid_lft forever preferred_lft forever4: virbr0-nic: 
         
           mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000    link/ether 52:54:00:59:38:b4 brd ff:ff:ff:ff:ff:ff
         
        
       
      

5.检查magent服务

[root@localhost keepalived]# ps -aux|grep magentroot     23443  0.0  0.0 115296  1476 ?        S    15:21   0:00 /bin/bash /etc/keepalived/chk_magent.shroot     23457  0.0  0.0 112720   968 pts/0    S+   15:21   0:00 grep --color=auto magent通过./chk_magent.sh可以正确启动正确启动应该是这样[root@localhost keepalived]# ./chk_magent.sh [root@localhost keepalived]# ps -aux|grep magentroot     24142  0.0  0.0  11948   612 ?        Ss   15:22   0:00 magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211root     24185  0.0  0.0 112720   972 pts/0    S+   15:22   0:00 grep --color=auto magent

6.检查系统日志

Aug 23 15:22:19 localhost Keepalived_vrrp[22337]: /etc/keepalived/chk_magent.sh exited due to signal 15Aug 23 15:22:19 localhost setroubleshoot: failed to retrieve rpm info for /usr/bin/magentAug 23 15:22:19 localhost setroubleshoot: SELinux is preventing /usr/bin/magent from name_bind access on the tcp_socket port 12000. For complete SELinux messages run: sealert -l 7a3a5f2e-5328-4dd4-88d6-bdefea289425Aug 23 15:22:19 localhost python: SELinux is preventing /usr/bin/magent from name_bind access on the tcp_socket port 12000.#012#012*****  Plugin bind_ports (92.2 confidence) suggests   ************************#012脚本启动失败了，似乎是防火墙

7.关掉SELinux

[root@localhost keepalived]# setenforce 0[root@localhost keepalived]# getenforcePermissive

8.重启keepalived

[root@localhost keepalived]# systemctl stop keepalived[root@localhost keepalived]# pkill magent[root@localhost keepalived]# ps -aux|grep magentroot     27598  0.0  0.0 112720   972 pts/0    S+   15:32   0:00 grep --color=auto magent[root@localhost keepalived]# systemctl start keepalived[root@localhost keepalived]# ps -aux|grep magentroot     27647  0.0  0.0  11948   616 ?        Ss   15:32   0:00 magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211root     27699  0.0  0.0 112720   972 pts/0    S+   15:32   0:00 grep --color=auto magent果然是防火墙！

9.SELinux放出magent

比较复杂，以后再补

 

 

 

 

最终配置

 10.0.0.111上的keepalived.conf

! Configuration File for keepalivedglobal_defs {   notification_email {     #acassen@firewall.loc     #failover@firewall.loc     #sysadmin@firewall.loc   }   #notification_email_from Alexandre.Cassen@firewall.loc   #smtp_server 192.168.200.1   #smtp_connect_timeout 30   router_id LVS_DEVEL   vrrp_skip_check_adv_addr   #vrrp_strict   vrrp_garp_interval 0   vrrp_gna_interval 0}vrrp_script chk_nginx {             script "/etc/keepalived/chk_nginx.sh"       interval 2             }vrrp_script chk_magent {             script "/etc/keepalived/chk_magent.sh"       interval 2             }vrrp_instance VI_1 {    state MASTER    interface ens192    virtual_router_id 51    priority 50    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.0.0.233        10.0.0.234    }    notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111成为主机！ 报！111成为主机！"    notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！111已挂！ 10.0.0.111故障！请尽快修复！"    track_script {      chk_nginx                     }}vrrp_instance VI_2 {    state MASTER    interface ens192    virtual_router_id 52    priority 50    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        # 10.0.0.234    }    track_script {      chk_magent                     }    }

10.0.0.111上的chk_magent.sh

#!/bin/bashrun=`ps -C magent --no-header | wc -l`if [ $run -eq 0 ]; then    magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.111:11211 -b 10.0.0.222:11211    sleep 3    if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then        systemctl stop keepalived    fifi

10.0.0.222上的keepalived.conf

! Configuration File for keepalivedglobal_defs {   notification_email {     #acassen@firewall.loc     #failover@firewall.loc     #sysadmin@firewall.loc   }   #notification_email_from Alexandre.Cassen@firewall.loc   #smtp_server 192.168.200.1   #smtp_connect_timeout 30   router_id LVS_DEVEL   vrrp_skip_check_adv_addr   #vrrp_strict   vrrp_garp_interval 0   vrrp_gna_interval 0}vrrp_script chk_nginx {             script "/etc/keepalived/chk_nginx.sh"       interval 2             }vrrp_script chk_magent {             script "/etc/keepalived/chk_magent.sh"       interval 2             }vrrp_instance VI_1 {    state BACKUP    interface ens192    virtual_router_id 51    priority 30    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        10.0.0.233        10.0.0.234    }    notify_master "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！222成为主机！ 报！222成为主机！"    notify_fault "/usr/bin/python2.7 /etc/keepalived/send.py 1499418300@qq.com 报！222已挂！ 10.0.0.222故障！请尽快修复！"    track_script {      chk_nginx                     }    }vrrp_instance VI_2 {    state BACKUP    interface ens192    virtual_router_id 52    priority 30    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        # 10.0.0.234    }    track_script {      chk_magent                     }    }

10.0.0.222上的chk_magent.sh

#!/bin/bashrun=`ps -C magent --no-header | wc -l`if [ $run -eq 0 ]; then    magent -u root -n 51200 -l 10.0.0.234 -p 12000 -s 10.0.0.222:11211 -b 10.0.0.111:11211    sleep 3    if [ `ps -C magent --no-header | wc -l` -eq 0 ]; then        systemctl stop keepalived    fifi

 

 

 

参考文档