linux-haheartbeat高可用的安装配置（http服务高可用）
heartbeat工作原理：本次实验，使用广播地址225.0.0.11 694作为heartbeat的网络心跳地址，heartbeat通过网络心跳监测对端服务是否正常，
通过/etc/ha.d/ha.cf 文件的deadtime参数判断对端服务中断，通过参数auto_failback参数控制资源失败自动接管，从而完成httpd服务的双机互备。
两台主机（一台是rhel6.4 arbiter，一台是centos6.8 mymongo1）
一、两台主机的基本信息，重点是hosts文件的IP地址主机名解析
[root@mymongo1 ~]# uname -a
Linux mymongo1 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@mymongo1 ~]# hostname
mymongo1
[root@mymongo1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
#::1 localhost6.localdomain6 localhost6
192.168.144.111 arbiter
192.168.144.132 mymongo1
[root@mymongo1 ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0C:29:40:94:8E
inet addr:192.168.199.131 Bcast:192.168.199.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe40:948e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:46 errors:0 dropped:0 overruns:0 frame:0
TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4219 (4.1 KiB) TX bytes:3299 (3.2 KiB)
eth2 Link encap:Ethernet HWaddr 00:0C:29:40:94:98
inet addr:192.168.144.132 Bcast:192.168.144.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe40:9498/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1890 errors:0 dropped:0 overruns:0 frame:0
TX packets:1800 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:253127 (247.1 KiB) TX bytes:252326 (246.4 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:576 (576.0 b) TX bytes:576 (576.0 b)
[root@mymongo1 ~]#
[root@arbiter ~]# hostname
arbiter
[root@arbiter ~]# uname -a
Linux arbiter 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@arbiter ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
#::1 localhost6.localdomain6 localhost6
192.168.144.111 arbiter
192.168.144.132 mymongo1
[root@arbiter ~]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:50:56:3E:FE:CF
inet addr:192.168.199.128 Bcast:192.168.199.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe3e:fecf/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:85 errors:0 dropped:0 overruns:0 frame:0
TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9437 (9.2 KiB) TX bytes:2719 (2.6 KiB)
eth2 Link encap:Ethernet HWaddr 00:50:56:21:CF:3E
inet addr:192.168.144.111 Bcast:192.168.144.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fe21:cf3e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2992 errors:0 dropped:0 overruns:0 frame:0
TX packets:3054 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:348131 (339.9 KiB) TX bytes:423727 (413.7 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:240 (240.0 b) TX bytes:240 (240.0 b)
[root@arbiter ~]#
二、heartbeat的安装，rhel和centos安装不一样，先安装centos机器的heartbeat
注意：最少保证2台主机中的centos主机能够上网，centos安装Heartbeats，通过yum方式网络安装
1、首先下载epel-release-6-8.noarch.rpm
wget http://mirrors.sohu.com/fedora-epel/6/x86_64/epel-release-6-8.noarch.rpm
[root@mymongo1 ~]# ls
amoeba Documents install.log MySQL-5.6.36.tar.gz Pictures Videos
anaconda-ks.cfg Downloads install.log.syslog mysql-proxy-0.8.2 Public
arbiter epel-release-6-8.noarch.rpmlua-5.1.4 mysql-proxy-0.8.2.tar.gz Templates
cmake-2.8.10.2.tar.gz glib-2.22.5 lua-5.1.4.tar.gz mysql-proxy-0.8.5 test1.sh
Desktop glib-2.22.5.tar Music mysql-proxy-0.8.5.tar.gz test.sh
[root@mymongo1 ~]#
--安装epel，其实是安装yum源
rpm -ivh epel-release-6-8.noarch.rpm
[root@mymongo1 yum.repos.d]# pwd
/etc/yum.repos.d
[root@mymongo1 yum.repos.d]# ls
CentOS-Base.repo CentOS-Debuginfo.repo CentOS-fasttrack.repo CentOS-Media.repo CentOS-Vault.repo epel.repo epel-testing.repo
[root@mymongo1 yum.repos.d]#
[root@mymongo1 yum.repos.d]# cat epel.repo
[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 6 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
[root@mymongo1 yum.repos.d]#
[root@mymongo1 yum.repos.d]# cat epel-testing.repo
[epel-testing]
name=Extra Packages for Enterprise Linux 6 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-epel6&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 6 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/6/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
[epel-testing-source]
name=Extra Packages for Enterprise Linux 6 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
[root@mymongo1 yum.repos.d]#
--调整yum的配置文件，缓存本次heartbeat安装需要的依赖包
[root@mymongo1 etc]# cat yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=19&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release
[root@mymongo1 etc]#
--使用yum工具安装heartbeat
yum -y install heartbeat
安装完成后heartbeat的缓存目录
[root@mymongo1 ~]# find / -name heartbeat*.rpm
/var/cache/yum/x86_64/6/epel/packages/heartbeat-3.0.4-2.el6.x86_64.rpm
/var/cache/yum/x86_64/6/epel/packages/heartbeat-libs-3.0.4-2.el6.x86_64.rpm
[root@mymongo1 ~]#
[root@mymongo1 ~]# cd /var/cache/yum/x86_64/6/base/packages/
[root@mymongo1 packages]# ll
total 2060
-rw-r--r--. 1 root root 66344 Jul 24 2015 cifs-utils-4.8.1-20.el6.x86_64.rpm
-rw-r--r--. 1 root root 72744 Jun 24 2012 cluster-glue-1.0.5-6.el6.x86_64.rpm
-rw-r--r--. 1 root root 119096 Jun 24 2012 cluster-glue-libs-1.0.5-6.el6.x86_64.rpm
-rw-r--r--. 1 root root 40152 Oct 17 2014 keyutils-1.4-5.el6.x86_64.rpm
-rw-r--r--. 1 root root 23488 Sep 25 2011 libgssglue-0.1-11.el6.x86_64.rpm
-rw-r--r--. 1 root root 343900 Mar 23 08:00 nfs-utils-1.2.3-75.el6.x86_64.rpm
-rw-r--r--. 1 root root 72268 Mar 23 08:02 nfs-utils-lib-1.1.5-13.el6.x86_64.rpm
-rw-r--r--. 1 root root 38264 Oct 17 2014 perl-TimeDate-1.16-13.el6.noarch.rpm
-rw-r--r--. 1 root root 913840 Jul 2 2011 PyXML-0.8.4-19.el6.x86_64.rpm
-rw-r--r--. 1 root root 398016 Mar 24 06:25 resource-agents-3.9.5-46.el6.x86_64.rpm
[root@mymongo1 packages]#
2、在rhel机器上arbiter安装heartbeat，方法是将centos机器mymongo1上的heartbeat包及其依赖包scp到arbiter机器上，
使用rpm -ivh方式安装，安装过程（安装过程中需要其他依赖的rpm包，可以挂载centos cd1镜像到arbiter上找到安装即可）：
[root@arbiter heartbeat]# ls
cifs-utils-4.8.1-20.el6.x86_64.rpm keyutils-libs-1.4-5.el6.x86_64.rpm perl-TimeDate-1.16-13.el6.noarch.rpm
cluster-glue-1.0.5-6.el6.x86_64.rpm libevent-1.4.13-4.el6.x86_64.rpm python-argparse-1.2.1-2.1.el6.noarch.rpm
cluster-glue-libs-1.0.5-6.el6.x86_64.rpm libgssglue-0.1-11.el6.x86_64.rpm PyXML-0.8.4-19.el6.x86_64.rpm
heartbeat-3.0.4-2.el6.x86_64.rpm libtirpc-0.2.1-11.el6.x86_64.rpm resource-agents-3.9.5-46.el6.x86_64.rpm
heartbeat-libs-3.0.4-2.el6.x86_64.rpm nfs-utils-1.2.3-75.el6.x86_64.rpm rpcbind-0.2.0-12.el6.x86_64.rpm
keyutils-1.4-5.el6.x86_64.rpm nfs-utils-lib-1.1.5-13.el6.x86_64.rpm
[root@arbiter heartbeat]# rpm -ivh PyXML-0.8.4-19.el6.x86_64.rpm
warning: PyXML-0.8.4-19.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:PyXML ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh perl-TimeDate-1.16-13.el6.noarch.rpm
warning: perl-TimeDate-1.16-13.el6.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:perl-TimeDate ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh cifs-utils-4.8.1-20.el6.x86_64.rpm keyutils-1.4-5.el6.x86_64.rpm keyutils-libs-1.4-5.el6.x86_64.rpm
warning: cifs-utils-4.8.1-20.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
file /lib64/libkeyutils.so.1.3 from install of keyutils-libs-1.4-5.el6.x86_64 conflicts with file from package keyutils-libs-1.4-4.el6.x86_64
[root@arbiter heartbeat]# rpm -qa|grep keyutils-libs
keyutils-libs-1.4-4.el6.x86_64
[root@arbiter heartbeat]# rpm -e keyutils-libs
error: Failed dependencies:
libkeyutils.so.1()(64bit) is needed by (installed) krb5-libs-1.10.3-10.el6.x86_64
libkeyutils.so.1(KEYUTILS_0.3)(64bit) is needed by (installed) krb5-libs-1.10.3-10.el6.x86_64
[root@arbiter heartbeat]# rpm -e --nodeps keyutils-libs
[root@arbiter heartbeat]# rpm -qa|grep keyutils-libs
[root@arbiter heartbeat]# rpm -ivh cifs-utils-4.8.1-20.el6.x86_64.rpm keyutils-1.4-5.el6.x86_64.rpm keyutils-libs-1.4-5.el6.x86_64.rpm
warning: cifs-utils-4.8.1-20.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:keyutils-libs ########################################### [ 33%]
2:keyutils ########################################### [ 67%]
3:cifs-utils ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh cluster-glue-*
warning: cluster-glue-1.0.5-6.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:cluster-glue-libs ########################################### [ 50%]
2:cluster-glue ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh libgssglue-0.1-11.el6.x86_64.rpm
warning: libgssglue-0.1-11.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:libgssglue ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh libgssglue-0.1-11.el6.x86_64.rpm
warning: libgssglue-0.1-11.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
package libgssglue-0.1-11.el6.x86_64 is already installed
[root@arbiter heartbeat]# rpm -ivh libevent-1.4.13-4.el6.x86_64.rpm
warning: libevent-1.4.13-4.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:libevent ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh python-argparse-1.2.1-2.1.el6.noarch.rpm
warning: python-argparse-1.2.1-2.1.el6.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:python-argparse ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh libtirpc-0.2.1-11.el6.x86_64.rpm
warning: libtirpc-0.2.1-11.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:libtirpc ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh rpcbind-0.2.0-12.el6.x86_64.rpm
warning: rpcbind-0.2.0-12.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:rpcbind ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh nfs-utils-*
warning: nfs-utils-1.2.3-75.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:nfs-utils-lib ########################################### [ 50%]
2:nfs-utils ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh heartbeat-*
warning: heartbeat-3.0.4-2.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
error: Failed dependencies:
resource-agents is needed by heartbeat-3.0.4-2.el6.x86_64
[root@arbiter heartbeat]# rpm -ivh resource-agents-3.9.5-46.el6.x86_64.rpm
warning: resource-agents-3.9.5-46.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing... ########################################### [100%]
1:resource-agents ########################################### [100%]
[root@arbiter heartbeat]# rpm -ivh heartbeat-*
warning: heartbeat-3.0.4-2.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing... ########################################### [100%]
1:heartbeat-libs ########################################### [ 50%]
2:heartbeat ########################################### [100%]
[root@arbiter heartbeat]#
3、heartbeat安装完成后，配置以下三个配置文件
[root@mymongo1 ~]# ll /etc/ha.d/
total 32
-rw-------. 1 root root 37 Aug 16 23:53 authkeys
-rw-r--r--. 1 root root 239 Aug 17 00:01 ha.cf
-rwxr-xr-x. 1 root root 745 Dec 2 2013 harc
-rw-r--r--. 1 root root 206 Aug 17 01:01 haresources
drwxr-xr-x. 2 root root 4096 Aug 16 22:42 rc.d
-rw-r--r--. 1 root root 692 Dec 2 2013 README.config
drwxr-xr-x. 2 root root 4096 Aug 16 22:42 resource.d
-rw-r--r--. 1 root root 2082 Mar 24 06:01 shellfuncs
[root@mymongo1 ~]#
heartbeat三个配置文件的所在目录
[root@mymongo1 ~]# rpm -q heartbeat -d
/usr/share/doc/heartbeat-3.0.4/AUTHORS
/usr/share/doc/heartbeat-3.0.4/COPYING
/usr/share/doc/heartbeat-3.0.4/COPYING.LGPL
/usr/share/doc/heartbeat-3.0.4/ChangeLog
/usr/share/doc/heartbeat-3.0.4/README
/usr/share/doc/heartbeat-3.0.4/apphbd.cf
/usr/share/doc/heartbeat-3.0.4/authkeys
/usr/share/doc/heartbeat-3.0.4/ha.cf
/usr/share/doc/heartbeat-3.0.4/haresources
/usr/share/man/man1/cl_status.1.gz
/usr/share/man/man1/hb_addnode.1.gz
/usr/share/man/man1/hb_delnode.1.gz
/usr/share/man/man1/hb_standby.1.gz
/usr/share/man/man1/hb_takeover.1.gz
/usr/share/man/man5/authkeys.5.gz
/usr/share/man/man5/ha.cf.5.gz
/usr/share/man/man8/apphbd.8.gz
/usr/share/man/man8/heartbeat.8.gz
[root@mymongo1 ~]#
heartbeat的ha.cf配置
[root@mymongo1 ~]# cat /etc/ha.d/ha.cf
#heartbeat controlfile
debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local1
keepalive 2
deadtime 30
warntime 10
initdead 60
#bcast eth2，使用广播方式做Heartbeat的心跳
mcast eth2 225.0.0.11 694 1 0
auto_failback on
node arbiter
node mymongo1
crm no
[root@mymongo1 ~]#
heartbeat的authkeys配置
sha1加密字符串随意，但是要求权限是600
[root@mymongo1 ~]# cat /etc/ha.d/authkeys
auth 1
1 sha1 adf220fnasdfinvs02adsf
[root@mymongo1 ~]#
heartbeart的haresources配置
这里注意：配置的资源是浮动vip的地址，vip绑定的网络端口要保持一致，用eth2都用eth2，用eth0都用eth0
并且eht1和eth0要用不同的IP网段，网口后的服务httpd代表资源启动时会连带启动httpd，没有则不启动
[root@mymongo1 ~]# cat /etc/ha.d/haresources
arbiter IPaddr::192.168.144.121/24/eth2 httpd
mymongo1 IPaddr::192.168.144.122/24/eth2
#mysql master
#arbiter IPaddr::192.168.144.11/24/eth2 mysqld
#mysql salve
#mymongo1 IPaddr::192.168.144.12/24/eth2
[root@mymongo1 ~]#
三、本次实验使用http做测试
2台主机分别编辑index.html
[root@mymongo1 ~]# cat /var/www/html/index.html
this is mymongo1,its ip is 192.168.144.132
[root@mymongo1 ~]#
[root@arbiter ~]# cat /var/www/html/index.html
this is arbiter ,its ip is 192.168.144.111!
[root@arbiter ~]#
2台主机均启动http服务
[root@arbiter ~]# service httpd status
httpd (pid 4574) is running...
[root@arbiter ~]#
[root@mymongo1 ~]# service httpd status
httpd (pid 5714) is running...
[root@mymongo1 ~]#
2台主机均启动heartbeat
[root@mymongo1 ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
INFO: Resource is stopped
Done.
[root@mymongo1 ~]#
稍等1分钟左右观察浮动IP会启动
[root@mymongo1 ~]# ip add|grep 192.168.144
inet 192.168.144.132/24 brd 192.168.144.255 scope global eth2
inet 192.168.144.122/24 brd 192.168.144.255 scope global secondary eth2
[root@mymongo1 ~]#
[root@arbiter ~]# /etc/init.d/heartbeat start
Starting High-Availability services: INFO: Resource is stopped
INFO: Resource is stopped
Done.
[root@arbiter ~]# ip add|grep 192.168.144
inet 192.168.144.111/24 brd 192.168.144.255 scope global eth2
inet 192.168.144.121/24 brd 192.168.144.255 scope global secondary eth2
[root@arbiter ~]#
网页查看
到此实验结束！
分享题目：linux-haheartbeat高可用的安装配置（http服务高可用）
当前URL：http://scyanting.com/article/jpcpis.html
