• Pcs：Pacemaker/Corosync構成ツール
  コマンドラインインターフェースによるpacemaker/corosyncの制御や設定が可能です。
  Pacemakerのクラスタについて、作成・表示・変更などを設定できます。
• Pacemaker：リソース制御機能
  • アプリケーション監視・制御機能
    Apache, nginx, Tomcat, JBoss, PostgreSQL, Oracle, MySQL, ファイルシステム制御、仮想IPアドレス制御、等、多数のリソースエージェント(RA)を同梱しています。
    また、RAを自作すればどんなアプリケーションでも監視可能です。
  • ネットワーク監視・制御機能
    定期的に指定された宛先へpingを送信することでネットワーク接続の正常性を監視できます。
  • ノード監視機能
    定期的に互いにハートビート通信を行いノード監視をします。
    また、STONITH機能により通信不可となったノードの電源を強制的に停止し、両系稼働状態（スプリットブレイン）を回避できます。
  • 自己監視機能
    Pacemaker関連プロセスの停止時は影響度合いに応じ適宜、プロセス再起動、またはフェイルオーバを実施します。
    また、watchdog機能を併用し、メインプロセス停止時は自動的にOS再起動（およびフェイルオーバ）を実行します。
  • ディスク監視・制御機能
    指定されたディスクの読み込みを定期的に実施し、ディスクアクセスの正常性を監視します。
• Corosync：クラスタ制御機能
• Stonith：排他制御機能
  制御が利かなくなったノードの電源を強制的に停止してクラスタから「強制的に離脱（Fencing)」させる機能のことです。
  
  • フェンシング（電源断）制御
    ipmi（IPMIデバイス用）
    libvirt（KVM,Xen等仮想マシン制御用）
    ec2（AmazonEC2用）
    等
    
  • サーバ生死確認、相撃ち防止
    stonith-helper

• 仮想共有ディスクのvmdkファイルを置く場所
  /vmfs/volumes/datastore2/com_datastore
• 仮想共有ディスクファイル名
  com1.vmdk
• 仮想共有ディスクの容量
  5GB

# ssh vmware
password:
[root@vmware:~] mkdir /vmfs/volumes/datastore2/com_datastore
[root@vmware:~] vmkfstools -d eagerzeroedthick -c 5G /vmfs/volumes/datastore2/com_datastore/com1.vmdk

# ssh vmware
password:
[root@vmware:~] mkdir /vmfs/volumes/datastore2/com_datastore

• 新たにSCSIコントローラを作成します。
  SCSIバスの共有：仮想
  ※同一のESXiに存在するノード間でディスクを共有したい場合は「仮想」を選択し、複数のESXiをまたいでディスクを共有したい場合は「物理」を選択する。
  
• ハードディスクを追加します。
  「ハードディスクの追加」　－　「新規標準ハードディスク」
  「新規ハードディスク」の右側の▶をクリックし展開します。
  以下は設定した箇所
  容量：5GB
  場所：[datastore2] com_datastore/com1.vmdk
  ディスク プロビジョニング：シック プロビジョニング (Eager Zeroed)
  シェア：標準
  制限-IOPs：制限なし
  コントローラの場所：SCSIコントローラ1　SCSI (1:0)
  ディスクモード：独立型：通常
  共有：マルチライターの共有
  右下の「保存」をクリックすると共有ディスクが作成されます。
  
  2つ目のノード（centos7-2）で上記で作成した共有ディスクを利用してSCSIコントローラ等を作成します。
  
  
• 新たにSCSIコントローラを作成します。
  SCSIバスの共有：仮想
  
• ハードディスクを追加します。
  「ハードディスクの追加」　－　「既存のハードディスク」で「[datastore2] com_datastore/com1.vmdk」を選択します。
  タイプが「シック プロビジョニング (Lazy Zeroed)」になっていても、この状態のまま一度保存します。
  再度、該当VMの設定を開きます。
  すると、タイプが「シック プロビジョニング (Eager Zeroed) 」として表示されるはずです。
  以下を設定します。
  シェア：標準
  制限-IOPs：制限なし
  コントローラの場所：SCSIコントローラ1　SCSI (1:0)
  ディスクモード：独立型：通常
  共有：マルチライターの共有
  右下の「保存」をクリックします。
  

■centos7-1で確認
[root@centos7-1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O サイズ (最小 / 推奨): 512 バイト / 512 バイト

■centos7-2で確認
[root@centos7-2 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O サイズ (最小 / 推奨): 512 バイト / 512 バイト

[root@centos7-1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x631b680e.
コマンド (m でヘルプ): m
コマンドの動作
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)
現時点でのパーティション情報を表示させます。
コマンド (m でヘルプ): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O サイズ (最小 / 推奨): 512 バイト / 512 バイト
Disk label type: dos
ディスク識別子: 0x631b680e
デバイス ブート      始点        終点     ブロック   Id  システム
パーティションの設定作業未実施のため、何も表示されません。
パーティションを設定します。
コマンド (m でヘルプ): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
プライマリパーティションとして作成します。
サイズは全領域を指定します。
Select (default p): p
パーティション番号 (1-4, default 1): （リターン）
最初 sector (2048-10485759, 初期値 2048): （リターン）
初期値 2048 を使います
Last sector, +sectors or +size{K,M,G} (2048-10485759, 初期値 10485759): （リターン）
初期値 10485759 を使います
Partition 1 of type Linux and of size 5 GiB is set
ファイルシステムが作成されているか確認します。
コマンド (m でヘルプ): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O サイズ (最小 / 推奨): 512 バイト / 512 バイト
Disk label type: dos
ディスク識別子: 0x631b680e
デバイス ブート  始点     終点        ブロック    Id  システム
/dev/sdb1        2048     10485759    5241856     83  Linux
コマンド (m でヘルプ): w               
パーティションテーブルは変更されました！

[root@centos7-1 ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310464 blocks
65523 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@centos7-1 ~]# mount /dev/sdb1 /mnt

# dnf repolist all
repo id                               repo の名前                                                                       状態
appstream                             CentOS Stream 8 - AppStream                                                       有効化
baseos                                CentOS Stream 8 - BaseOS                                                          有効化
debuginfo                             CentOS Stream 8 - Debuginfo                                                       無効化
epel                                  Extra Packages for Enterprise Linux 8 - x86_64                                    無効化
epel-debuginfo                        Extra Packages for Enterprise Linux 8 - x86_64 - Debug                            無効化
epel-modular                          Extra Packages for Enterprise Linux Modular 8 - x86_64                            有効化
epel-modular-debuginfo                Extra Packages for Enterprise Linux Modular 8 - x86_64 - Debug                    無効化
epel-modular-source                   Extra Packages for Enterprise Linux Modular 8 - x86_64 - Source                   無効化
epel-playground                       Extra Packages for Enterprise Linux 8 - Playground - x86_64                       無効化
epel-playground-debuginfo             Extra Packages for Enterprise Linux 8 - Playground - x86_64 - Debug               無効化
epel-playground-source                Extra Packages for Enterprise Linux 8 - Playground - x86_64 - Source              無効化
epel-source                           Extra Packages for Enterprise Linux 8 - x86_64 - Source                           無効化
epel-testing                          Extra Packages for Enterprise Linux 8 - Testing - x86_64                          無効化
epel-testing-debuginfo                Extra Packages for Enterprise Linux 8 - Testing - x86_64 - Debug                  無効化
epel-testing-modular                  Extra Packages for Enterprise Linux Modular 8 - Testing - x86_64                  無効化
epel-testing-modular-debuginfo        Extra Packages for Enterprise Linux Modular 8 - Testing - x86_64 - Debug          無効化
epel-testing-modular-source           Extra Packages for Enterprise Linux Modular 8 - Testing - x86_64 - Source         無効化
epel-testing-source                   Extra Packages for Enterprise Linux 8 - Testing - x86_64 - Source                 無効化
extras                                CentOS Stream 8 - Extras                                                          有効化
ha                                    CentOS Stream 8 - HighAvailability                                                無効化
media-appstream                       CentOS Stream 8 - Media - AppStream                                               無効化
media-baseos                          CentOS Stream 8 - Media - BaseOS                                                  無効化
powertools                            CentOS Stream 8 - PowerTools                                                      無効化
rt                                    CentOS Stream 8 - RealTime                                                        無効化

※両ノードで実施
# dnf --enablerepo=ha install -y pacemaker pcs fence-agents-all

※両ノードで実施
[root@centos8-1 ~]# cat /etc/passwd | grep hacluster
hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin

[root@centos8-2 ~]# cat /etc/passwd | grep hacluster
hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin

※両ノードで実施
# passwd hacluster
ユーザ hacluster のパスワードを変更。
新しいパスワード:
新しいパスワードを再入力してください:
passwd: すべての認証トークンが正しく更新できました。

# vi /etc/hosts
10.0.0.41    centos8-str1
10.0.0.42    centos8-str2
192.168.0.41    centos8-str1c
192.168.0.42    centos8-str2c

# systemctl enable --now pcsd

or

# systemctl start pcsd.service
# systemctl enable pcsd.service

※両ノードで実施
# firewall-cmd --add-service=high-availability --permanent
success
# firewall-cmd --reload
success

[root@centos8-str1 ~]# pcs host auth centos8-str1 centos8-str2
Username: hacluster
Password: 
centos8-str2: Authorized
centos8-str1: Authorized

[root@centos8-str1 ~]# pcs cluster setup --start bigbang centos8-str1 addr=10.0.0.41 addr=192.168.0.41 \
                                                         centos8-str2 addr=10.0.0.42 addr=192.168.0.42
Destroying cluster on hosts: 'centos8-str1', 'centos8-str2'...
centos8-str1: Successfully destroyed cluster
centos8-str2: Successfully destroyed cluster
Requesting remove 'pcsd settings' from 'centos8-str1', 'centos8-str2'
centos8-str1: successful removal of the file 'pcsd settings'
centos8-str2: successful removal of the file 'pcsd settings'
Sending 'corosync authkey', 'pacemaker authkey' to 'centos8-str1', 'centos8-str2'
centos8-str1: successful distribution of the file 'corosync authkey'
centos8-str1: successful distribution of the file 'pacemaker authkey'
centos8-str2: successful distribution of the file 'corosync authkey'
centos8-str2: successful distribution of the file 'pacemaker authkey'
Sending 'corosync.conf' to 'centos8-str1', 'centos8-str2'
centos8-str1: successful distribution of the file 'corosync.conf'
centos8-str2: successful distribution of the file 'corosync.conf'
Cluster has been successfully set up.
Starting cluster on hosts: 'centos8-str1', 'centos8-str2'...

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang

WARNINGS:
No stonith devices and stonith-enabled is not false

Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:33:04 2021
  * Last change:  Fri Mar  5 14:32:07 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * No resources

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# corosync-cfgtool -s
Printing link status.
Local node ID 1
LINK ID 0
	addr	= 10.0.0.41
	status:
		nodeid  1:	localhost
		nodeid  2:	connected
LINK ID 1
	addr	= 192.168.0.41
	status:
		nodeid  1:	localhost
		nodeid  2:	connected

• no-quorum-policy: クラスタがquorumを持っていないときの動作を定義します。デフォルトはstop（全リソース停止）です。
  2台構成の場合は、片方が停止しても動作を続行できるよう、ignore（全リソースはそのまま動作を続行）を指定します。
• stonith-enabled: フェンシング（STONITH）を有効にするかどうかを指定します。デフォルトはtrue（有効）です。
  これが有効、かつSTONITHリソースが未設定の場合リソースが起動できません。
• cluster-recheck-interval: クラスタのチェック間隔を指定します。デフォルトは15分です。
  failure-timeout（failcountを自動クリアするまでの時間）など、時間ベースの動作が反映されるまでの時間に影響します。

[root@centos8-str1 ~]# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-8.el8-ba59be7122
 have-watchdog: false
# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore
# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-8.el8-ba59be7122
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

• ocf:heartbeat:IPaddr2 : 仮想IPアドレスを制御
• ocf:heartbeat:Filesystem : ファイルシステムのマウントを制御
• systemd:mariadb : MariaDBを制御

# pcs resource describe <リソースエージェント>

※片方のノードで実施
[root@centos8-str1 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.140 cidr_netmask=24 nic=ens192 op monitor interval=30s

• IPaddr
  　manages virtual IPv4 addresses (portable version)
• IPaddr2
  　manages virtual IPv4 addresses (Linux specific version)

※片方のノードで実施
[root@centos8-str1 ~]# pcs resource create ShareDir ocf:heartbeat:Filesystem device=/dev/sdb1 directory=/mnt fstype=ext4

※片方のノードで実施
[root@centos8-str1 ~]# pcs resource create MariaDB systemd:mariadb

※片方のノードで実施
[root@centos8-str1 ~]# pcs resource group add rg01 VirtualIP ShareDir MariaDB

※片方のノードで実施
[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:38:16 2021
  * Last change:  Fri Mar  5 14:38:07 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource config
 Group: rg01
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.140 nic=ens192
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)

[root@centos8-str1 ~]# pcs cluster start --all
centos8-str1: Starting Cluster...
centos8-str2: Starting Cluster...

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:45:03 2021
  * Last change:  Fri Mar  5 14:38:07 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource status
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

[root@centos8-str1 ~]# pcs resource move rg01 centos8-str2
[root@centos8-str1 ~]# pcs resource move rg01
[root@centos8-str1 ~]# pcs resource clear rg01

[root@centos8-str1 ~]# pcs resource status
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2

[[root@centos8-str1 ~]# pcs resource move rg01 centos8-str1
[root@centos8-str1 ~]# pcs resource move rg01
[root@centos8-str1 ~]# pcs resource clear rg01

[root@centos8-str1 ~]# pcs resource status
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:48:22 2021
  * Last change:  Fri Mar  5 14:47:55 2021 by root via crm_resource on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs node standby centos8-str1

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:49:16 2021
  * Last change:  Fri Mar  5 14:49:04 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Node centos8-str1: standby
  * Online: [ centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs node unstandby centos8-str1
[root@centos8-str1 ~]# pcs node standby centos8-str2
[root@centos8-str1 ~]# pcs node unstandby centos8-str2

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:50:19 2021
  * Last change:  Fri Mar  5 14:50:14 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# shutdown -r now
Connection to centos8-str1 closed by remote host.
Connection to centos8-str1 closed.

[root@centos8-str2 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:56:47 2021
  * Last change:  Fri Mar  5 14:50:14 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str2 ]
  * OFFLINE: [ centos8-str1 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# pcs cluster start centos8-str1
centos8-str1: Starting Cluster...

[root@centos8-str2 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 14:57:50 2021
  * Last change:  Fri Mar  5 14:50:14 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource update VirtualIP op monitor on-fail=standby
[root@centos8-str1 ~]# pcs resource config
 Group: rg01
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.140 nic=ens192
   Operations: monitor interval=60s on-fail=standby (VirtualIP-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)

[root@centos8-str2 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 15:05:45 2021
  * Last change:  Fri Mar  5 14:58:41 2021 by root via cibadmin on centos8-str2
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Node centos8-str1: standby (on-fail)
  * Online: [ centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Starting centos8-str2

Failed Resource Actions:
  * VirtualIP_monitor_60000 on centos8-str1 'not running' (7): call=20, status='complete', exitreason='', \
    last-rc-change='2021-03-05 15:05:42 +09:00', queued=0ms, exec=0ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Mar  5 15:10:57 2021
  * Last change:  Fri Mar  5 15:10:36 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 3 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# dnf install httpd php php-fpm php-mbstring
# vi /etc/php.ini
date.timezone ="Asia/Tokyo"

# dnf install httpd

# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore

# vi /etc/selinux/config
SELINUX=disabled

# dnf install mariadb mariadb-server
# rpm -Uvh https://repo.zabbix.com/zabbix/5.0/rhel/8/x86_64/zabbix-release-5.0-1.el8.noarch.rpm
# rpm -Uvh https://repo.zabbix.com/zabbix/5.4/rhel/8/x86_64/zabbix-release-5.4-1.el8.noarch.rpm
# dnf clean all
# dnf install zabbix-server-mysql zabbix-web-mysql zabbix-apache-conf zabbix-agent
# dnf install zabbix-server-mysql zabbix-web-service zabbix-sql-scripts zabbix-agent
# dnf install zabbix-server-mysql zabbix-web-service zabbix-sql-scripts zabbix-agent zabbix-web-mysql zabbix-apache-conf zabbix-web-japanese zabbix-get

[root@centos8-str1 ~]# systemctl start mariadb
[root@centos8-str1 ~]# systemctl status mariadb
[root@centos8-str1 ~]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

[root@centos8-str1 ~]# mysql -uroot -p
Enter password: 
MariaDB [(none)]> create database zabbix character set utf8 collate utf8_bin;
MariaDB [(none)]> create user zabbix@localhost identified by 'password';
MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@localhost;
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> quit

[root@centos8-str1 ~]# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix
[root@centos8-str1 ~]# zcat /usr/share/doc/zabbix-sql-scripts/mysql/create.sql.gz | mysql -uzabbix zabbix -p
Enter password: 

or

[root@centos8-str1 ~]# zcat /usr/share/doc/zabbix-sql-scripts/mysql/create.sql.gz | mysql -uzabbix -p zabbix
Enter password: 

# vi /etc/zabbix/zabbix_server.conf
DBSocket=/mnt2/drbd0/mysql/mysql.sock
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=yourpassword

# vi /etc/zabbix/web/zabbix.conf.php
$DB['SERVER'] = 'localhost';　←　変更前
  ↓
$DB['SERVER'] = '127.0.0.1';　←　変更後
※このファイルをscpで他方のホストにコピーする場合、所有者に注意してください。
　[root@centos8-str2 ~]# ls -la /etc/zabbix/web/zabbix.conf.php
　-rw------- 1 apache apache 1482  3月 26 18:55 /etc/zabbix/web/zabbix.conf.php
　アクセス権がapacheでない場合、webフロントエンドで
　  configuration file error
　    permission denied
　と表示されてログインできなくなります。

# vi /etc/php-fpm.d/zabbix.conf
php_value[date.timezone] = "Asia/Tokyo"

# firewall-cmd --add-port=10051/tcp --permanent
# firewall-cmd --add-service=http --permanent
# systemctl restart firewalld

[root@centos8-str1 ~]# systemctl restart zabbix-server zabbix-agent httpd php-fpm

• DBHost=localhost
• DBName=zabbix
• DBUser=zabbix
• DBPassword=yourpassword

• ユーザ名：Admin
• パスワード：zabbix

[root@centos8-str1 ~]# scp /etc/zabbix/web/zabbix.conf.php centos8-str2:/etc/zabbix/web/zabbix.conf.php
root@centos8-str2's password: 

# vi /etc/zabbix/zabbix_server.conf
DBSocket=/mnt2/drbd0/mysql/mysql.sock
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=yourpassword

# vi /etc/zabbix/web/zabbix.conf.php
$DB['SERVER'] = 'localhost';　←　変更前
  ↓
$DB['SERVER'] = '127.0.0.1';　←　変更後
※このファイルをscpで他方のホストにコピーする場合、所有者に注意してください。
　[root@centos8-str2 ~]# ls -la /etc/zabbix/web/zabbix.conf.php
　-rw------- 1 apache apache 1482  3月 26 18:55 /etc/zabbix/web/zabbix.conf.php
　アクセス権がapacheでない場合、webフロントエンドで
　  configuration file error
　    permission denied
　と表示されてログインできなくなります。

# vi /etc/php-fpm.d/zabbix.conf
php_value[date.timezone] = "Asia/Tokyo"

[root@centos8-str1 ~]# systemctl stop httpd
[root@centos8-str1 ~]# systemctl stop zabbix-server

[root@centos8-str1 ~]# pcs resource create httpd systemd:httpd --group rg01
[root@centos8-str1 ~]# pcs resource create zabbix-server systemd:zabbix-server --group rg01

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Thu Mar 25 15:47:26 2021
  * Last change:  Thu Mar 25 15:47:08 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
  * Clone Set: DRBD-clone [DRBD] (promotable):
    * Masters: [ centos8-str1 ]
    * Slaves: [ centos8-str2 ]

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str1 centos8-str2
Pacemaker Nodes:
 centos8-str1 centos8-str2

Resources:
 Group: rg01
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.140 nic=ens192
   Operations: monitor interval=60s on-fail=standby (VirtualIP-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt2/drbd0 fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: httpd (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (httpd-monitor-interval-60)
               start interval=0s timeout=100 (httpd-start-interval-0s)
               stop interval=0s timeout=100 (httpd-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=60 timeout=100 (zabbix-server-monitor-interval-60)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
 Clone: DRBD-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD-reload-interval-0s)
               start interval=0s timeout=240 (DRBD-start-interval-0s)
               stop interval=0s timeout=100 (DRBD-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
Ordering Constraints:
  promote DRBD-clone then start rg01 (kind:Mandatory) (id:order-DRBD-clone-rg01-mandatory)
  promote DRBD-clone then start FS_DRBD0 (kind:Mandatory) (id:order-DRBD-clone-FS_DRBD0-mandatory)
Colocation Constraints:
  DRBD-clone with rg01 (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD-clone-rg01-INFINITY)
  DRBD-clone with FS_DRBD0 (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD-clone-FS_DRBD0-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  No defaults set
Operations Defaults:
  No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-8.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1616654708
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

• DRBDr0
  DRBDリソースr0をMaster/Slaveで設定
• FS_DRBD0
  /dev/drbd0をxfs形式で/mnt2/drbd0ディレクトリへマウント。DRBDr0のMaster側で稼働
• ShareDir
  /dev/sdb1をxfs形式で/mntディレクトリへマウント
• MariaDB
  FS_DRBD0でマウントされたデータベースを読み込んでMariaDBを起動。Active/Standby。DRBDr0のMaster側で稼働
• VirtualIP
  仮想IPの割り当て。DRBDr0のMaster側で稼働
• Apache
  httpdサービスを起動。Active/Standby。DRBDr0のMaster側で稼働
• Zabbix-Server
  zabbix-serverサービスを起動。Active/Standby。DRBDr0のMaster側で稼働

[root@centos8-str1 ~]# pcs resource remove zabbix-server
Attempting to stop: zabbix-server... Stopped
[root@centos8-str1 ~]# pcs resource remove httpd
Attempting to stop: httpd... Stopped
[root@centos8-str1 ~]# pcs resource remove VirtualIP
Attempting to stop: VirtualIP... Stopped
[root@centos8-str1 ~]# pcs resource remove MariaDB
Attempting to stop: MariaDB... Stopped
[root@centos8-str1 ~]# pcs resource remove ShareDir
Attempting to stop: ShareDir... Stopped
[root@centos8-str1 ~]# pcs resource remove FS_DRBD0
Attempting to stop: FS_DRBD0... Stopped
[root@centos8-str1 ~]# pcs resource remove DRBD-clone
Attempting to stop: DRBD... Stopped

# pcs resource defaults migration-threshold=1

[root@centos8-str1 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str1 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
　優先的にcentos8-str1がMasterとして起動するようにします。
　スコア値が高いほど優先的に起動するようです。
　-INFINITY<-100<0<100<INFINITY
[root@centos8-str1 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str1=100
[root@centos8-str1 ~]# pcs resource cleanup DRBD_r0
[root@centos8-str1 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem \
                           device=/dev/drbd0 directory=/mnt2/drbd0 fstype=xfs --group zabbix-group
[root@centos8-str1 ~]# pcs resource create MariaDB systemd:mariadb --group zabbix-group
[root@centos8-str1 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.140 \
                       cidr_netmask=24 nic=ens192 op monitor interval=30s --group zabbix-group
[root@centos8-str1 ~]# pcs resource create httpd systemd:httpd --group zabbix-group
[root@centos8-str1 ~]# pcs resource create zabbix-server systemd:zabbix-server --group zabbix-group
[root@centos8-str1 ~]# pcs resource create ShareDir ocf:heartbeat:Filesystem \
                           device=/dev/sdb1 directory=/mnt fstype=ext4 --group zabbix-group

## DRBD_r0がMaster側のノードでFS_DRBD0リソースを起動するように設定
[root@centos8-str1 ~]# pcs constraint colocation add DRBD_r0-clone with Master FS_DRBD0

## DRBD_r0起動後にFS_DRBD0リソースを起動
[root@centos8-str1 ~]# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0
Adding DRBD_r0-clone FS_DRBD0 (kind: Mandatory) (Options: first-action=promote then-action=start)

## zabbix-groupはDRBD_r0のMasterと同じノードで起動
[root@centos8-str1 ~]# pcs constraint colocation add zabbix-group with Master DRBD_r0-clone INFINITY

## MariaDBはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0 with MariaDB INFINITY --force
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with MariaDB INFINITY

## VirtualIPはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0 with VirtualIP INFINITY --force
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP INFINITY

## httpdはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0 with httpd INFINITY --force
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with httpd INFINITY

## zabbix-serverはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0 with zabbix-server INFINITY --force
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-server INFINITY

## ShareDirはDRBD_r0のMasterと同じノードで起動（異常検知で必要）：後日追加
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0 with ShareDir INFINITY --force
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with ShareDir INFINITY
[root@centos8-str1 ~]# pcs constraint colocation add master DRBD_r0-clone with ShareDir INFINITY

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 11:17:19 2021
  * Last change:  Mon Mar 29 11:10:05 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str1 ]
    * Slaves: [ centos8-str2 ]
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str1 centos8-str2
Pacemaker Nodes:
 centos8-str1 centos8-str2

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Group: zabbix-group
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt2/drbd0 fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.140 nic=ens192
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: httpd (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (httpd-monitor-interval-60)
               start interval=0s timeout=100 (httpd-start-interval-0s)
               stop interval=0s timeout=100 (httpd-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=60 timeout=100 (zabbix-server-monitor-interval-60)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
Ordering Constraints:
  promote DRBD_r0-clone then start FS_DRBD0 (kind:Mandatory) (id:order-DRBD_r0-clone-FS_DRBD0-mandatory)
Colocation Constraints:
  DRBD_r0-clone with FS_DRBD0 (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD_r0-clone-FS_DRBD0-INFINITY)
  FS_DRBD0 with zabbix-group (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-FS_DRBD0-zabbix-group-INFINITY)
  DRBD_r0 with MariaDB (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-MariaDB-INFINITY)
  DRBD_r0 with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-VirtualIP-INFINITY)
  DRBD_r0 with httpd (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-httpd-INFINITY)
  DRBD_r0 with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-zabbix-server-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
Operations Defaults:
  No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-8.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1616983434
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str1 ~]# ps axu | grep http[d]
（30秒ほど待ちます。）
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 13:42:44 2021
  * Last change:  Mon Mar 29 13:24:34 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Resource Group: rg01:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1　← pacemakerによって新たに起動されています。
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
  * Clone Set: DRBD-clone [DRBD] (promotable):
    * DRBD	(ocf::linbit:drbd):	 Slave centos8-str2
    * DRBD	(ocf::linbit:drbd):	 Master centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD                     	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD                     	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * httpd: migration-threshold=1000000 fail-count=1 last-failure='Mon Mar 29 13:42:10 2021'
　　　　↑　centos8-str1でhttpdにfail-countが加算されています。

Failed Resource Actions:　← centos8-str1でhttpdが起動していなかった事が記録されています。
  * httpd_monitor_60000 on centos8-str1 'not running' (7): call=256, \
        status='complete', exitreason='', last-rc-change='2021-03-29 13:42:10 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup httpd

[root@centos8-str1 ~]# pcs resource defaults migration-threshold=1
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str1 ~]# ps axu | grep http[d]

[root@centos8-str2 ~]# pkill httpd
[root@centos8-str2 ~]# ps axu | grep http[d]
（30秒ほど待ちます。）
[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 13:49:59 2021
  * Last change:  Mon Mar 29 13:47:34 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * httpd: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 13:49:25 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str2 'not running' (7): call=134, status='complete', exitreason='', last-rc-change='2021-03-29 13:49:24 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup httpd

[root@centos8-str1 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str1 ~]# kill -kill `pgrep -f mysqld`
[root@centos8-str1 ~]# ps axu | grep mysqld
（30秒ほど待ちます。）
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Tue Mar 30 09:40:05 2021
  * Last change:  Mon Mar 29 18:08:20 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * MariaDB: migration-threshold=1 fail-count=1 last-failure='Tue Mar 30 09:39:21 2021'

Failed Resource Actions:
  * MariaDB_monitor_60000 on centos8-str1 'not running' (7): call=40, status='complete', exitreason='', last-rc-change='2021-03-30 09:39:21 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# pcs resource cleanup MariaDB
[root@centos8-str2 ~]# kill -kill `pgrep -f mysqld`
[root@centos8-str2 ~]# ps axu | grep mysqld
（30秒ほど待ちます。）
[root@centos8-str2 ~]# pcs status --full;date
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Tue Mar 30 10:02:50 2021
  * Last change:  Tue Mar 30 09:42:57 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured (1 BLOCKED from further action due to failure)

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 FAILED centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Stopped
    * httpd	(systemd:httpd):	 Stopped
    * zabbix-server	(systemd:zabbix-server):	 FAILED centos8-str2 (blocked)
    * ShareDir	(ocf::heartbeat:Filesystem):	 Stopped

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * MariaDB: migration-threshold=1 fail-count=1 last-failure='Tue Mar 30 09:44:32 2021'
    * zabbix-server: migration-threshold=1 fail-count=1000000 last-failure='Tue Mar 30 09:46:14 2021'

Failed Resource Actions:
  * MariaDB_monitor_60000 on centos8-str2 'not running' (7): call=46, status='complete', exitreason='', last-rc-change='2021-03-30 09:44:32 +09:00', queued=0ms, exec=0ms
  * zabbix-server_stop_0 on centos8-str2 'OCF_TIMEOUT' (198): call=64, status='Timed Out', exitreason='', last-rc-change='2021-03-30 09:46:14 +09:00', queued=0ms, exec=99987ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# systemctl stop zabbix-server
^C

[root@centos8-str2 ~]# kill -kill `pgrep -f zabbix_server`

[root@centos8-str2 ~]# pcs resource cleanup MariaDB
[root@centos8-str2 ~]# pcs resource cleanup zabbix-server
[root@centos8-str2 ~]# pcs constraint colocation add MariaDB with zabbix-server INFINITY
[root@centos8-str2 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str1 centos8-str2
Pacemaker Nodes:
 centos8-str1 centos8-str2

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Group: zabbix-group
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt2/drbd0 fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.140 nic=ens192
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: httpd (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (httpd-monitor-interval-60)
               start interval=0s timeout=100 (httpd-start-interval-0s)
               stop interval=0s timeout=100 (httpd-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=60 timeout=100 (zabbix-server-monitor-interval-60)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
Ordering Constraints:
  promote DRBD_r0-clone then start FS_DRBD0 (kind:Mandatory) (id:order-DRBD_r0-clone-FS_DRBD0-mandatory)
Colocation Constraints:
  DRBD_r0-clone with FS_DRBD0 (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD_r0-clone-FS_DRBD0-INFINITY)
  FS_DRBD0 with zabbix-group (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-FS_DRBD0-zabbix-group-INFINITY)
  DRBD_r0 with MariaDB (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-MariaDB-INFINITY)
  DRBD_r0 with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-VirtualIP-INFINITY)
  DRBD_r0 with httpd (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-httpd-INFINITY)
  DRBD_r0 with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-zabbix-server-INFINITY)
  DRBD_r0 with ShareDir (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-ShareDir-INFINITY)
  MariaDB with zabbix-server (score:INFINITY) (id:colocation-MariaDB-zabbix-server-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
Operations Defaults:
  No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-8.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1617067515
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

[root@centos8-str1 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str1 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str1 ~]# ps axu | grep zabbix_server
（30秒ほど待ちます。）
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 17:41:43 2021
  * Last change:  Mon Mar 29 17:39:12 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 17:41:02 2021'

Failed Resource Actions:
  * zabbix-server_monitor_60000 on centos8-str1 'not running' (7): call=298, status='complete', exitreason='', last-rc-change='2021-03-29 17:41:02 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# pcs resource cleanup zabbix-server
[root@centos8-str2 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str2 ~]# ps axu | grep zabbix_server
（30秒ほど待ちます。）
[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 17:46:45 2021
  * Last change:  Mon Mar 29 17:45:27 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 17:46:14 2021'

Failed Resource Actions:
  * zabbix-server_monitor_60000 on centos8-str2 'not running' (7): call=304, status='complete', exitreason='', last-rc-change='2021-03-29 17:46:14 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled/

[root@centos8-str2 ~]# pcs resource cleanup zabbix-server

[root@centos8-str1 ~]# pcs resource defaults migration-threshold=1

（30秒ほど待ちます。）
[root@centos8-str2 ~]# pcs status --full;date
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 14:56:56 2021
  * Last change:  Mon Mar 29 14:42:29 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * VirtualIP: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 14:55:16 2021'

Failed Resource Actions:
  * VirtualIP_monitor_30000 on centos8-str1 'not running' (7): call=173, status='complete', exitreason='', last-rc-change='2021-03-29 14:55:16 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Offline
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# pcs resource cleanup VirtualIP

（30秒ほど待ちます。）
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 15:14:32 2021
  * Last change:  Mon Mar 29 15:07:40 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * VirtualIP: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 15:14:02 2021'

Failed Resource Actions:
  * VirtualIP_monitor_30000 on centos8-str2 'not running' (7): call=179, status='complete', exitreason='', last-rc-change='2021-03-29 15:14:02 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Offline

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup VirtualIP

[root@centos8-str1 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str1 ~]# umount /mnt
（30秒ほど待ちます。）
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 16:28:17 2021
  * Last change:  Mon Mar 29 16:25:19 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * ShareDir: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 16:26:48 2021'

Failed Resource Actions:
  * ShareDir_monitor_20000 on centos8-str1 'not running' (7): call=218, status='complete', exitreason='', last-rc-change='2021-03-29 16:26:48 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup ShareDir

[root@centos8-str2 ~]# umount /mnt
（30秒ほど待ちます。）
[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar 29 16:40:45 2021
  * Last change:  Mon Mar 29 16:37:13 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * ShareDir: migration-threshold=1 fail-count=1 last-failure='Mon Mar 29 16:40:25 2021'

Failed Resource Actions:
  * ShareDir_monitor_20000 on centos8-str2 'not running' (7): call=224, status='complete', exitreason='', last-rc-change='2021-03-29 16:40:25 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup ShareDir

# dnf install httpd php php-fpm php-mbstring

# vi /etc/php.ini
date.timezone ="Asia/Tokyo"

# vi /etc/selinux/config
SELINUX=disabled

# dnf install mariadb mariadb-server
# rpm -Uvh https://repo.zabbix.com/zabbix/5.0/rhel/8/x86_64/zabbix-release-5.0-1.el8.noarch.rpm
# rpm -Uvh https://repo.zabbix.com/zabbix/5.4/rhel/8/x86_64/zabbix-release-5.4-1.el8.noarch.rpm
# dnf clean all
# dnf install zabbix-server-mysql zabbix-web-mysql zabbix-apache-conf zabbix-agent
# dnf install zabbix-server-mysql zabbix-web-service zabbix-sql-scripts zabbix-agent
# dnf install zabbix-server-mysql zabbix-web-service zabbix-sql-scripts zabbix-agent zabbix-web-mysql zabbix-apache-conf zabbix-web-japanese zabbix-get

[root@centos8-str3 ~]# systemctl start mariadb
[root@centos8-str3 ~]# systemctl status mariadb
[root@centos8-str3 ~]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

[root@centos8-str3 ~]# mysql -uroot -p
Enter password: 
MariaDB [(none)]> create database zabbix character set utf8 collate utf8_bin;
MariaDB [(none)]> create user zabbix@localhost identified by 'password';
MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@localhost;
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> quit
[root@centos8-str3 ~]# zcat /usr/share/doc/zabbix-server-mysql*/create.sql.gz | mysql -uzabbix -p zabbix
[root@centos8-str3 ~]# zcat /usr/share/doc/zabbix-sql-scripts/mysql/create.sql.gz | mysql -uzabbix zabbix -p
Enter password: 

# vi /etc/zabbix/zabbix_server.conf
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=yourpassword

# vi /etc/zabbix/web/zabbix.conf.php
$DB['SERVER'] = 'localhost';　←　変更前
  ↓
$DB['SERVER'] = '127.0.0.1';　←　変更後
※このファイルをscpで他方のホストにコピーする場合、所有者に注意してください。
　[root@centos8-str4 ~]# ls -la /etc/zabbix/web/zabbix.conf.php
　-rw------- 1 apache apache 1482  3月 26 18:55 /etc/zabbix/web/zabbix.conf.php
　アクセス権がapacheでない場合、webフロントエンドで
　  configuration file error
　    permission denied
　と表示されてログインできなくなります。

# vi /etc/php-fpm.d/zabbix.conf
php_value[date.timezone] = "Asia/Tokyo"

# firewall-cmd --add-port=10051/tcp --permanent
# firewall-cmd --add-service=http --permanent
# systemctl restart firewalld

[root@centos8-str3 ~]# systemctl start mariadb httpd php-fpm zabbix-server

• DBHost=localhost
• DBName=zabbix
• DBUser=zabbix
• DBPassword=yourpassword

• ユーザ名：Admin
• パスワード：zabbix

# systemctl stop zabbix-server php-fpm httpd mariadb
# systemctl disable mariadb httpd zabbix-server

[root@centos8-str3 ~]# systemctl stop zabbix-server
[root@centos8-str3 ~]# systemctl stop mariadb
[root@centos8-str3 ~]# systemctl stop php-fpm
[root@centos8-str3 ~]# systemctl stop httpd
[root@centos8-str3 ~]# systemctl disable zabbix-server
[root@centos8-str3 ~]# systemctl disable mariadb
[root@centos8-str3 ~]# systemctl disable php-fpm
[root@centos8-str3 ~]# systemctl disable httpd

[root@centos8-str3 ~]# systemctl enable --now pcsd
[root@centos8-str3 ~]# pcs cluster setup --start bigbang centos8-str3 addr=192.168.0.43 addr=10.0.0.43 centos8-str4 addr=192.168.0.44 addr=10.0.0.44
Destroying cluster on hosts: 'centos8-str3', 'centos8-str4'...
centos8-str3: Successfully destroyed cluster
centos8-str4: Successfully destroyed cluster
Requesting remove 'pcsd settings' from 'centos8-str3', 'centos8-str4'
centos8-str3: successful removal of the file 'pcsd settings'
centos8-str4: successful removal of the file 'pcsd settings'
Sending 'corosync authkey', 'pacemaker authkey' to 'centos8-str3', 'centos8-str4'
centos8-str3: successful distribution of the file 'corosync authkey'
centos8-str3: successful distribution of the file 'pacemaker authkey'
centos8-str4: successful distribution of the file 'corosync authkey'
centos8-str4: successful distribution of the file 'pacemaker authkey'
Sending 'corosync.conf' to 'centos8-str3', 'centos8-str4'
centos8-str3: successful distribution of the file 'corosync.conf'
centos8-str4: successful distribution of the file 'corosync.conf'
Cluster has been successfully set up.
Starting cluster on hosts: 'centos8-str3', 'centos8-str4'...

[root@centos8-str3 ~]# pcs property set stonith-enabled=false
[root@centos8-str3 ~]# pcs property set no-quorum-policy=ignore

[root@centos8-str3 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs node standby centos8-str4

• MariaDB
  各ホストに保存されたデータベースを読み込んでMariaDBを起動。Active/Standby。Master側で稼働
• Apache
  httpdサービスを起動。Active/Standby。Master側で稼働
• php-fpm
  php-fpmサービスを起動。Master側で稼働
• Zabbix-Server
  zabbix-serverサービスを起動。Active/Standby。Master側で稼働
• VirtualIP
  仮想IPの割り当て。Master側で稼働

[root@centos8-str3 ~]# pcs resource create MariaDB ocf:heartbeat:mysql binary=/usr/bin/mysqld_safe \
datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_user=repl \
replication_passwd=[パスワード] op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# ps aux|grep -v grep|grep mysql

[root@centos8-str3 ~]# pcs resource promotable MariaDB master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

[root@centos8-str3 ~]# pcs constraint location MariaDB-clone prefers centos8-str3=100

[root@centos8-str3 ~]# pcs resource create zabbix-server systemd:zabbix-server op monitor interval="10s" timeout="20s"

[root@centos8-str3 ~]# ps aux|grep -v grep|grep zabbix_server

# vi /etc/httpd/conf.d/status.conf
ExtendedStatus On
 
<Location /server-status>
    SetHandler server-status
    Require local
</Location>

[root@centos8-str3 ~]# pcs resource create Apache ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf \
statusurl="http://localhost/server-status" op monitor interval=10s timeout=10s

Failed Resource Actions:
  * Apache_start_0 on centos8-str3 'error' (1): call=35, status='Timed Out', exitreason='', last-rc-change='2021-04-06 16:51:57 +09:00', queued=0ms, exec=40001ms

上記記載方法だとクリーンアップしてもエラーになるので、リソースの追加方法を下記に変更します。
[root@centos8-str3 ~]# pcs resource create Apache systemd:httpd op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# ps aux|grep -v grep|grep httpd

[root@centos8-str3 ~]# pcs resource create php-fpm systemd:php-fpm op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# ps aux|grep -v grep|grep php-fpm

[root@centos8-str3 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.40 cidr_netmask=24 nic=ens192 op monitor interval=10s timeout=10s
[root@centos8-str3 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.0.40 cidr_netmask=24 nic=ens224 op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource group add zabbix-group Apache php-fpm zabbix-server VirtualIP VirtualIP2

# MariaDB起動後にzabbix-groupリソースを起動
[root@centos8-str3 ~]# pcs constraint order promote MariaDB-clone then start zabbix-group
Adding MariaDB-clone zabbix-group (kind: Mandatory) (Options: first-action=promote then-action=start)

# zabbix-groupはMariaDBのMasterと同じノードで起動
[root@centos8-str3 ~]# pcs constraint colocation add zabbix-group with master MariaDB-clone INFINITY

# zabbix-serverはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with zabbix-server INFINITY

# ApacheはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with Apache INFINITY

# php-fpmはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with php-fpm INFINITY

# VirtualIPはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with VirtualIP INFINITY

# VirtualIP2はMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with VirtualIP2 INFINITY

[root@centos8-str3 ~]# pcs node unstandby centos8-str4
[root@centos8-str3 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Wed Apr  7 17:19:13 2021
  * Last change:  Wed Apr  7 17:16:09 2021 by root via cibadmin on centos8-str3
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 centos8-str4 ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * Masters: [ centos8-str3 ]
    * Slaves: [ centos8-str4 ]
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str3 centos8-str4
Pacemaker Nodes:
 centos8-str3 centos8-str4

Resources:
 Clone: MariaDB-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: MariaDB (class=ocf provider=heartbeat type=mysql)
   Attributes: binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_passwd=[パスワード] replication_user=repl
   Operations: demote interval=0s timeout=120s (MariaDB-demote-interval-0s)
               monitor interval=10s timeout=10s (MariaDB-monitor-interval-10s)
               notify interval=0s timeout=90s (MariaDB-notify-interval-0s)
               promote interval=0s timeout=120s (MariaDB-promote-interval-0s)
               start interval=0s timeout=120s (MariaDB-start-interval-0s)
               stop interval=0s timeout=120s (MariaDB-stop-interval-0s)
 Group: zabbix-group
  Resource: Apache (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status
   Operations: monitor interval=10s timeout=10s (Apache-monitor-interval-10s)
               start interval=0s timeout=40s (Apache-start-interval-0s)
               stop interval=0s timeout=60s (Apache-stop-interval-0s)
  Resource: php-fpm (class=systemd type=php-fpm)
   Operations: monitor interval=10s timeout=10s (php-fpm-monitor-interval-10s)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=10s timeout=20s (zabbix-server-monitor-interval-10s)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.40 nic=ens192
   Operations: monitor interval=10s timeout=10s (VirtualIP-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.40 nic=ens224
   Operations: monitor interval=10s timeout=10s (VirtualIP2-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: MariaDB-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-MariaDB-clone-centos8-str3-100)
Ordering Constraints:
  promote MariaDB-clone then start zabbix-group (kind:Mandatory) (id:order-MariaDB-clone-zabbix-group-mandatory)
Colocation Constraints:
  zabbix-group with MariaDB-clone (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-zabbix-group-MariaDB-clone-INFINITY)
  MariaDB-clone with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-zabbix-server-INFINITY)
  MariaDB-clone with Apache (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-Apache-INFINITY)
  MariaDB-clone with php-fpm (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-php-fpm-INFINITY)
  MariaDB-clone with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP-INFINITY)
  MariaDB-clone with VirtualIP2 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP2-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
    resource-stickiness=INFINITY
Operations Defaults:
  No defaults set

Cluster Properties:
 MariaDB_REPL_INFO: centos8-str3.bigbang.mydns.jp|mysql-bin.000352|342
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-9.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1617778306
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str3 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str3 ~]# ps axu | grep http[d]
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 10:34:17 2021
  * Last change:  Fri Apr  9 10:32:45 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 1         
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 10:32:42 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str3 'not running' (7): call=29, status='complete', exitreason='', last-rc-change='2021-04-09 10:32:42 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str4 ~]# ps axu | grep http[d]
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 10:41:48 2021
  * Last change:  Fri Apr  9 10:41:22 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 10:41:18 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str4 'not running' (7): call=35, status='complete', exitreason='', last-rc-change='2021-04-09 10:41:19 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Offline
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str3 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str3 ~]# ps axu | grep -v grep | grep zabbix_server
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 10:51:46 2021
  * Last change:  Fri Apr  9 10:50:59 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 1         
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 10:50:54 2021'

Failed Resource Actions:
  * zabbix-server_monitor_10000 on centos8-str3 'not running' (7): call=64, status='complete', exitreason='', last-rc-change='2021-04-09 10:50:54 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup zabbix-server
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str4 ~]# ps axu | grep -v grep | grep zabbix_server
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 10:57:12 2021
  * Last change:  Fri Apr  9 10:56:58 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 10:56:52 2021'

Failed Resource Actions:
  * zabbix-server_monitor_10000 on centos8-str4 'not running' (7): call=70, status='complete', exitreason='', last-rc-change='2021-04-09 10:56:52 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Offline
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup zabbix-server

[root@centos8-str3 ~]# pcs resource update VirtualIP op monitor on-fail=standby
[root@centos8-str3 ~]# pcs resource update VirtualIP2 op monitor on-fail=standby
[root@centos8-str3 ~]# pcs resource config

 Clone: MariaDB-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: MariaDB (class=ocf provider=heartbeat type=mysql)
   Attributes: binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_passwd=[パスワード] replication_user=repl
   Operations: demote interval=0s timeout=120s (MariaDB-demote-interval-0s)
               monitor interval=10s timeout=10s (MariaDB-monitor-interval-10s)
               notify interval=0s timeout=90s (MariaDB-notify-interval-0s)
               promote interval=0s timeout=120s (MariaDB-promote-interval-0s)
               start interval=0s timeout=120s (MariaDB-start-interval-0s)
               stop interval=0s timeout=120s (MariaDB-stop-interval-0s)
 Group: zabbix-group
  Resource: Apache (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status
   Operations: monitor interval=10s timeout=10s (Apache-monitor-interval-10s)
               start interval=0s timeout=40s (Apache-start-interval-0s)
               stop interval=0s timeout=60s (Apache-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=10s timeout=20s (zabbix-server-monitor-interval-10s)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.40 nic=ens192
   Operations: monitor interval=60s on-fail=standby (VirtualIP-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.40 nic=ens224
   Operations: monitor interval=60s on-fail=standby (VirtualIP2-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)

[root@centos8-str4 ~]# pcs status --full

Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 11:25:19 2021
  * Last change:  Fri Apr  9 11:24:46 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Node centos8-str3 (1): standby (on-fail)
  * Online: [ centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Stopped
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4

Node Attributes:
  * Node: centos8-str3 (1):
    * readable                        	: 0         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * VirtualIP: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 11:24:39 2021'

Failed Resource Actions:
  * VirtualIP_monitor_60000 on centos8-str3 'not running' (7): call=102, status='complete', exitreason='', last-rc-change='2021-04-09 11:24:39 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Offline
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup VirtualIP

[root@centos8-str4 ~]# pcs status --full

Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 11:32:53 2021
  * Last change:  Fri Apr  9 11:32:30 2021 by hacluster via crmd on centos8-str3
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 1         
    * readable                        	: 0         
  * Node: centos8-str4 (2):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:

Tickets:

PCSD Status:
  centos8-str3: Offline
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Apr  9 11:42:17 2021
  * Last change:  Fri Apr  9 11:41:57 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Node centos8-str4 (2): standby (on-fail)
  * Online: [ centos8-str3 (1) ]

Full List of Resources:
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Stopped
  * Resource Group: zabbix-group:
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * readable                        	: 0         

Migration Summary:
  * Node: centos8-str4 (2):
    * VirtualIP2: migration-threshold=1 fail-count=1 last-failure='Fri Apr  9 11:41:50 2021'

Failed Resource Actions:
  * VirtualIP2_monitor_60000 on centos8-str4 'not running' (7): call=109, status='complete', exitreason='', last-rc-change='2021-04-09 11:41:50 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs resource cleanup VirtualIP2

# pcs host auth alma8-1 alma8-2
# pcs cluster setup --start bigbang alma8-1 addr=10.0.0.151 addr=192.168.10.151 alma8-2 addr=10.0.0.152 addr=192.168.10.152

• no-quorum-policy: ignore　←　2台構成のため
• stonith-enabled: false　←　2台構成のため
• failure-timeout=300s　←　障害検知後、その後300秒間障害がない場合、障害記録（fail-count）をクリア
• migration-threshold=3　←　いずれかのリソースで（上記の場合、300秒以内に）3回障害検知時にフェールオーバー発生
• resource-stickiness=INFINITY　←　自動フェールバック無効

# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore
# pcs resource defaults update failure-timeout=300s
# pcs resource defaults migration-threshold=3
# pcs resource defaults resource-stickiness=INFINITY

• DRBD（マスター・スレーブ方式）・・・①
• ファイルシステムマウント（DRBDマスタ側）・・・②
　DRBD　→　マウント　の順で動作するように設定
• バーチャルIP（マスタのみ）・・・③
• Apache（マスタのみ）・・・④
　②③④：同一リソースグループに設定し、②③④の順に起動
　②③④：このリソースグループはDRBDマスタ側で必ず起動

# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs constraint location DRBD_r0-clone prefers alma8-1=100
# pcs resource cleanup DRBD_r0
# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs --group cluster-group
# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.150 cidr_netmask=24 nic=ens192 op monitor interval=30s --group cluster-group
# pcs resource create httpd systemd:httpd --group cluster-group
# pcs constraint colocation add DRBD_r0-clone with Master FS_DRBD0
# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0
# pcs constraint colocation add cluster-group with Master DRBD_r0 INFINITY --force

• DRBD（マスター・スレーブ方式）・・・①
• ファイルシステムマウント（DRBDマスタ側）・・・②
　DRBD　→　マウント　の順で動作するように設定
• バーチャルIP（マスタのみ）・・・③
• Apache（マスタのみ）・・・④
• MariaDB（マスター・スレーブ方式）・・・⑤
• Zabbix Server（マスタのみ）・・・⑥
　②③④：同一リソースグループに設定し、②③④の順に起動
　　　⑤：マスター・スレーブ方式（各サーバのデータベースはデフォルトフォルダに保存）
　　　⑥：単独リソースとして追加
　②③④：このリソースグループはDRBDマスタ側で必ず起動
　　　⑤：このリソースはDRBDのマスタ側で必ず起動
　　　⑥：このリソースはMariaDBのマスタ側で必ず起動
　　　　　この設定を入れないとZabbix ServerがMariaDBのスレーブ側で起動してしまう

# pcs resource create MariaDB ocf:heartbeat:mysql binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log \
    pid=/run/mariadb/mariadb.pid replication_user=repl replication_passwd=******** op monitor interval=10s timeout=10s
# pcs resource promotable MariaDB master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs constraint colocation add master MariaDB-clone with master DRBD_r0 INFINITY --force
# pcs resource create zabbix-server systemd:zabbix-server
# pcs constraint colocation add zabbix-server with master MariaDB-clone INFINITY

• DRBD（マスター・スレーブ方式）・・・①
• ファイルシステムマウント（DRBDマスタ側）・・・②
　DRBD　→　マウント　の順で動作するように設定
• バーチャルIP（マスタのみ）・・・③
• Apache（マスタのみ）・・・④
• MariaDB（マスター・スレーブ方式）・・・⑤
• Zabbix Server（マスタのみ）・・・⑥
　②③④：同一リソースグループに設定し、②③④の順に起動
　　　⑤：単独リソースとして追加
　②③④：このリソースグループはDRBDマスタ側で必ず起動
　　　⑤：このリソースはDRBDのマスタ側で必ず起動
　　　⑥：このリソースはMariaDBのマスタ側で必ず起動
　　　⑦：「リソース設定（その2）」の設定後、リソースhttpd（または、リソースcluster-group（②③④））が起動してからリソースzabbix-serverを起動する順序を設定

# pcs constraint order promote cluster-group then start zabbix-server
  or
# pcs constraint order promote httpd then start zabbix-server

• MariaDBのインストール及びMaster/Slaveの構築
  「初期設定（CentOS 8）」参照
  レプリケーションの設定の「バージョン10.3系」参照
　
• Zabbix Serverのインストール及び設定
  「pacemakerでZabbix-Server及びMariaDBのMaster/Slaveを制御する」参照
  
　
• DRBDによる共有ディスクの設定
  「DRBDの構築について」参照
  「DRBDをpacemakerの現在のリソースに追加する」参照

[root@centos8-str3 ~]# pcs property set stonith-enabled=false
[root@centos8-str3 ~]# pcs property set no-quorum-policy=ignore

[root@centos8-str3 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs node standby centos8-str4

• DRBD
  DRBDを設定しブロックデバイスをリアルタイムにレプリケート。
  Primary/Secondaryで構築。
• FS_DRBD0
  DRBDのPrimary側でブロックデバイスを/mntにマウントする。
• MariaDB
  各ホストに保存されたデータベースを読み込んでMariaDBを起動。
  Master/Slaveで構築。
• VirtualIP
  仮想IP（サービス側）の割り当て。MariaDBのMaster側で稼働。
  zabbix-groupに追加。
• VirtualIP2
  仮想IP（DRDB同期用）の割り当て。MariaDBのMaster側で稼働。
  zabbix-groupに追加。
• Apache
  httpdサービスを起動。Active/Standby。MariaDBのMaster側で稼働
  zabbix-groupに追加。
• php-fpm
  php-fpmサービスを起動。MariaDBのMaster側で稼働。
  zabbix-groupに追加。
• Zabbix-Server
  zabbix-serverサービスを起動。Active/Standby。MariaDBのMaster側で稼働。
  zabbix-groupに追加。

[root@centos8-str3 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str3 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

[root@centos8-str3 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str3=100
[root@centos8-str3 ~]# pcs resource cleanup DRBD_r0
[root@centos8-str3 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs

[root@centos8-str3 ~]# pcs resource create MariaDB ocf:heartbeat:mysql binary=/usr/bin/mysqld_safe \
datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_user=repl \
replication_passwd=[パスワード] op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource promotable MariaDB master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

[root@centos8-str3 ~]# pcs constraint location MariaDB-clone prefers centos8-str3=100

[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with FS_DRBD0 INFINITY

[root@centos8-str3 ~]# pcs constraint colocation add FS_DRBD0 with MariaDB-clone INFINITY with-rsc-role=Master

[root@centos8-str3 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.40 cidr_netmask=24 nic=ens192 op monitor interval=10s timeout=10s
[root@centos8-str3 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.0.40 cidr_netmask=24 nic=ens224 op monitor interval=10s timeout=10s

# vi /etc/httpd/conf.d/status.conf
ExtendedStatus On
 
<Location /server-status>
    SetHandler server-status
    Require local
</Location>

[root@centos8-str3 ~]# pcs resource create Apache ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf \
statusurl="http://localhost/server-status" op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource create php-fpm systemd:php-fpm op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource create zabbix-server systemd:zabbix-server op monitor interval="10s" timeout="20s"

[root@centos8-str3 ~]# pcs resource group add zabbix-group VirtualIP VirtualIP2 Apache php-fpm zabbix-server

# VirtualIPはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP INFINITY

# VirtualIP2はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP2 INFINITY

# DRBD_r0起動後にzabbix-groupリソースを起動
[root@centos8-str3 ~]# pcs constraint order promote DRBD_r0-clone then start zabbix-group
Adding MariaDB-clone zabbix-group (kind: Mandatory) (Options: first-action=promote then-action=start)

# zabbix-groupはDRBD_r0のMasterと同じノードで起動
[root@centos8-str3 ~]# pcs constraint colocation add zabbix-group with master DRBD_r0-clone INFINITY

# zabbix-serverはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-server INFINITY

# ApacheはMariaDBのDRBD_r0と同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with Apache INFINITY

# php-fpmはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with php-fpm INFINITY

# zabbix-groupはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-group INFINITY

[root@centos8-str3 ~]# pcs node unstandby centos8-str4
[root@centos8-str3 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:04:08 2021
  * Last change:  Thu Apr 22 16:45:44 2021 by root via cibadmin on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 centos8-str4 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str3 ]
    * Slaves: [ centos8-str4 ]
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * Masters: [ centos8-str3 ]
    * Slaves: [ centos8-str4 ]

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str3 centos8-str4
Pacemaker Nodes:
 centos8-str3 centos8-str4

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Group: zabbix-group
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: Apache (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status
   Operations: monitor interval=10s timeout=10s (Apache-monitor-interval-10s)
               start interval=0s timeout=40s (Apache-start-interval-0s)
               stop interval=0s timeout=60s (Apache-stop-interval-0s)
  Resource: php-fpm (class=systemd type=php-fpm)
   Operations: monitor interval=10s timeout=10s (php-fpm-monitor-interval-10s)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=10s timeout=20s (zabbix-server-monitor-interval-10s)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.40 nic=ens192
   Operations: monitor interval=10s timeout=10s (VirtualIP-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.40 nic=ens224
   Operations: monitor interval=10s timeout=10s (VirtualIP2-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)
 Clone: MariaDB-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: MariaDB (class=ocf provider=heartbeat type=mysql)
   Attributes: binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log \
               pid=/run/mariadb/mariadb.pid replication_passwd=[パスワード] replication_user=repl
   Operations: demote interval=0s timeout=120s (MariaDB-demote-interval-0s)
               monitor interval=10s timeout=10s (MariaDB-monitor-interval-10s)
               notify interval=0s timeout=90s (MariaDB-notify-interval-0s)
               promote interval=0s timeout=120s (MariaDB-promote-interval-0s)
               start interval=0s timeout=120s (MariaDB-start-interval-0s)
               stop interval=0s timeout=120s (MariaDB-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: DRBD_r0-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-DRBD_r0-clone-centos8-str3-100)
  Resource: MariaDB-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-MariaDB-clone-centos8-str3-100)
Ordering Constraints:
  promote MariaDB-clone then start zabbix-group (kind:Mandatory) (id:order-MariaDB-clone-zabbix-group-mandatory)
Colocation Constraints:
  zabbix-group with MariaDB-clone (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-zabbix-group-MariaDB-clone-INFINITY)
  MariaDB-clone with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-zabbix-server-INFINITY)
  MariaDB-clone with Apache (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-Apache-INFINITY)
  MariaDB-clone with php-fpm (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-php-fpm-INFINITY)
  MariaDB-clone with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP-INFINITY)
  MariaDB-clone with VirtualIP2 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP2-INFINITY)
  DRBD_r0-clone with zabbix-group (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-zabbix-group-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
    resource-stickiness=INFINITY
Operations Defaults:
  No defaults set

Cluster Properties:
 MariaDB_REPL_INFO: centos8-str3.bigbang.mydns.jp|mysql-bin.000009|342
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-9.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1619074041
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str3 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str3 ~]# ps axu | grep http[d]
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:37:59 2021
  * Last change:  Mon Apr 26 11:37:37 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 11:37:30 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str3 'not running' (7): call=377, status='complete', exitreason='', last-rc-change='2021-04-26 11:37:30 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str4 ~]# ps axu | grep http[d]
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:49:00 2021
  * Last change:  Mon Apr 26 11:48:48 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 11:48:41 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str4 'not running' (7): call=375, status='complete', exitreason='', last-rc-change='2021-04-26 11:48:42 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)

• DRBDの起動（リソース「DRBD_r0」）
  IPアドレスが小さい方で優先的にMasterとして起動するように設定
• FileSystem（DRBDをマウント）の設定（リソース「FS_DRBD0」）
  DRBDのMaster側が有効化されてからマウントするように設定
• MariaDBの設定（リソース「MariaDB」を）及び当該リソースをリソースグループ「zabbix-group」に追加
  DRBDのMaster側で起動
• VirtualIP（2つ）の設定（リソース「VirtualIP」、「VirtualIP2」）及び当該リソースをリソースグループ「zabbix-group」に追加
  DRBDのMaster側で起動
• Apacheの設定（リソース「httpd」）及び当該リソースをリソースグループ「zabbix-group」に追加
  DRBDのMaster側で起動
• PHP-FPMの設定（リソース「php-fpm」）及び当該リソースをリソースグループ「zabbix-group」に追加
  DRBDのMaster側で起動
• Zabix-Serverの設定（リソース「zabbix-server」）及び当該リソースをリソースグループ「zabbix-group」に追加
  DRBDのMaster側で起動
• リソースグループ「zabbix-group」に追加はFS_DRBD0リソース起動してから起動

※centos8-str4はクラスタ構成から削除済です。

[root@centos8-str3 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Mon Jan 24 22:18:57 2022
  * Last change:  Mon Jan 24 18:28:42 2022 by root via cibadmin on centos8-str3
  * 1 node configured
  * 9 resource instances configured

Node List:
  * Online: [ centos8-str3 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str3 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * httpd	(systemd:httpd):	 Started centos8-str3
    * php-fpm	(systemd:httpd):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

※centos8-str4はクラスタ構成から削除済です。

[root@centos8-str3 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str3
Pacemaker Nodes:
 centos8-str3

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
  Attributes: device=/dev/drbd0 directory=/mnt fstype=xfs
  Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
              start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
              stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
 Group: zabbix-group
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=1.0.0.40 nic=ens192
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.111.40 nic=ens224
   Operations: monitor interval=10s timeout=10s (VirtualIP2-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)
  Resource: httpd (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (httpd-monitor-interval-60)
               start interval=0s timeout=100 (httpd-start-interval-0s)
               stop interval=0s timeout=100 (httpd-stop-interval-0s)
  Resource: php-fpm (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (php-fpm-monitor-interval-60)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=60 timeout=100 (zabbix-server-monitor-interval-60)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: DRBD_r0-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-DRBD_r0-clone-centos8-str3-100)
    Disabled on:
      Node: centos8-str4 (score:-INFINITY) (role:Started) (id:cli-ban-DRBD_r0-clone-on-centos8-str4)
Ordering Constraints:
  promote DRBD_r0-clone then start zabbix-group (kind:Mandatory) (id:order-DRBD_r0-clone-zabbix-group-mandatory)
Colocation Constraints:
  DRBD_r0-clone with FS_DRBD0 (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD_r0-clone-FS_DRBD0-INFINITY)
  DRBD_r0-clone with MariaDB (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-MariaDB-INFINITY)
  DRBD_r0-clone with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-VirtualIP-INFINITY)
  DRBD_r0-clone with VirtualIP2 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-VirtualIP2-INFINITY)
  DRBD_r0-clone with httpd (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-httpd-INFINITY)
  DRBD_r0-clone with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-zabbix-server-INFINITY)
  DRBD_r0-clone with php-fpm (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-php-fpm-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    failure-timeout=180s
    migration-threshold=3
    resource-stickiness=INFINITY
Operations Defaults:
  No defaults set

Cluster Properties:
 MariaDB_REPL_INFO: centos8-str4.bigbang.dyndns.org|mysql-bin.000058|342
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-9.el8_4.1-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1643016370
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

※centos8-str7、centos8-str8での作業です。

※両ノードで実行
# vi /etc/drbd.d/r0.res
resource r0 {
  on centos8-str7 {
    device    /dev/drbd0;
    disk      /dev/mapper/pool-data0;
    address   192.168.111.48:7789;
    meta-disk internal;
  }
  on centos8-str8 {
    device    /dev/drbd0;
    disk      /dev/mapper/pool-data0;
    address   192.168.111.49:7789;
    meta-disk internal;
  }
}

※両ノードで実行
DRDBのメタデータを両ノードで初期化
# drbdadm create-md r0
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/mapper/pool-data0 at byte offset 53682892800

Do you really want to overwrite the existing meta-data?
[need to type 'yes' to confirm] yes

md_offset 53682892800
al_offset 53682860032
bm_offset 53681221632

Found xfs filesystem
    52423068 kB data area apparently used
    52423068 kB left usable by current configuration

Even though it looks like this would place the new meta data into
unused space, you still need to confirm, as this is only a guess.

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
initializing bitmap (1600 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

※両ノードで実行
# drbdadm up r0

[root@centos8-str7 ~]# drbdadm status r0
r0 role:Secondary
  disk:Inconsistent
  centos8-str8 role:Secondary
    peer-disk:Inconsistent

centos8-str7をプライマリにし同期させます。
[root@centos8-str7 ~]# drbdadm primary --force r0
[root@centos8-str7 ~]# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  centos8-str8 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:0.88

[root@centos8-str7 ~]# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  centos8-str8 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:82.69

[root@centos8-str7 ~]# drbdadm status r0
r0 role:Primary
  disk:UpToDate
  centos8-str8 role:Secondary
    peer-disk:UpToDate

同期が完了しました。

DRBDを停止します。

[root@centos8-str8 ~]# drbdadm down r0
[root@centos8-str7 ~]# drbdadm down r0

※centos8-str7、centos8-str8での作業です。

root@centos8-str7 ~]# uname -r;rpm -qa|grep corosync|sort;rpm -qa|grep pacemaker|sort;rpm -qa|grep pcs|sort
4.18.0-305.3.1.el8.x86_64
corosync-3.1.0-5.el8.x86_64
corosynclib-3.1.0-5.el8.x86_64
pacemaker-2.0.5-9.el8_4.1.x86_64
pacemaker-cli-2.0.5-9.el8_4.1.x86_64
pacemaker-cluster-libs-2.0.5-9.el8_4.1.x86_64
pacemaker-libs-2.0.5-9.el8_4.1.x86_64
pacemaker-schemas-2.0.5-9.el8_4.1.noarch
pcs-0.10.12-3.el8.x86_64

[root@centos8-str8 ~]# uname -r;rpm -qa|grep corosync|sort;rpm -qa|grep pacemaker|sort;rpm -qa|grep pcs|sort
4.18.0-358.el8.x86_64
corosync-3.1.0-5.el8.x86_64
corosynclib-3.1.0-5.el8.x86_64
pacemaker-2.0.5-9.el8_4.1.x86_64
pacemaker-cli-2.0.5-9.el8_4.1.x86_64
pacemaker-cluster-libs-2.0.5-9.el8_4.1.x86_64
pacemaker-libs-2.0.5-9.el8_4.1.x86_64
pacemaker-schemas-2.0.5-9.el8_4.1.noarch
pcs-0.10.12-3.el8.x86_64

※centos8-str7、centos8-str8での作業です。

[root@centos8-str7 ~]# pcs cluster setup --start bigbang centos8-str7 addr=10.0.0.48 addr=192.168.111.48 centos8-str8 addr=10.0.0.49 addr=192.168.111.49
Error: Hosts do not have the same version of 'pacemaker'; host 'centos8-str8' has version 2.1.2; host 'centos8-str7' has version 2.0.5
Error: Hosts do not have the same version of 'corosync'; host 'centos8-str8' has version 3.1.5; host 'centos8-str7' has version 3.1.0
Error: Errors have occurred, therefore pcs is unable to continue

[root@centos8-str8 ~]# dnf downgrade pacemaker-2.0.5 corosync-3.1.0

下記コマンドをコピペして実行することで、障害等により問題が発生（単純なノード追加で想定しているクラスタ構成が戻らない等）したクラスタの再設定を短時間で設定することが可能です。

現在のクラスタ設定を破棄します。
※両ノードで実行
# pcs cluster stop　--all
# pcs cluster destroy

※両ノードで実行
# systemctl enable --now pcsd

※片方のノードで実行
[root@centos8-str7 ~]# pcs cluster setup --start bigbang centos8-str7 addr=10.0.0.48 addr=192.168.111.48 centos8-str8 addr=10.0.0.49 addr=192.168.111.49
[root@centos8-str7 ~]# pcs property set stonith-enabled=false
[root@centos8-str7 ~]# pcs property set no-quorum-policy=ignore
[root@centos8-str7 ~]# pcs resource defaults failure-timeout=180s
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。
[root@centos8-str7 ~]# pcs resource defaults migration-threshold=3
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。
[root@centos8-str7 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。

[root@centos8-str7 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str7 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
[root@centos8-str7 ~]# pcs resource cleanup DRBD_r0
[root@centos8-str7 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs
[root@centos8-str7 ~]# pcs resource create MariaDB systemd:mariadb --group zabbix-group
[root@centos8-str7 ~]# pcs resource cleanup
[root@centos8-str7 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.39 cidr_netmask=24 nic=ens192 op monitor interval=30s --group zabbix-group
[root@centos8-str7 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.111.39 cidr_netmask=24 nic=ens224 op monitor interval=30s --group zabbix-group
[root@centos8-str7 ~]# pcs resource create httpd systemd:httpd --group zabbix-group
[root@centos8-str7 ~]# pcs resource create php-fpm systemd:php-fpm --group zabbix-group
[root@centos8-str7 ~]# pcs resource create zabbix-server systemd:zabbix-server --group zabbix-group

## DRBD_r0は優先的にcentos8-str7がMasterとして起動するように設定
[root@centos8-str7 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str7=100
## DRBD_r0がMaster側のノードでFS_DRBD0リソースを起動するように設定
[root@centos8-str7 ~]# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0
## FS_DRBD0はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with FS_DRBD0 INFINITY
## MariaDBはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with MariaDB INFINITY
## VirtualIPはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP INFINITY
## VirtualIP2はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP2 INFINITY
## httpdはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with httpd INFINITY
## php-fpmはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with php-fpm INFINITY
## zabbix-serverはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-server INFINITY

## FS_DRBD0が起動してからzabbix-groupを起動するように設定
[root@centos8-str7 ~]# pcs constraint order FS_DRBD0 then start zabbix-group

[root@centos8-str3 ~]# uname -r;rpm -qa|grep corosync|sort;rpm -qa|grep pacemaker|sort;rpm -qa|grep pcs|sort
4.18.0-358.el8.x86_64
corosync-3.1.5-2.el8.x86_64
corosynclib-3.1.5-2.el8.x86_64
pacemaker-2.1.2-2.el8.x86_64
pacemaker-cli-2.1.2-2.el8.x86_64
pacemaker-cluster-libs-2.1.2-2.el8.x86_64
pacemaker-libs-2.1.2-2.el8.x86_64
pacemaker-schemas-2.1.2-2.el8.noarch
pcs-0.10.12-3.el8.x86_64

[root@centos8-str8 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str7 centos8-str8
Pacemaker Nodes:
 centos8-str7 centos8-str8

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
  Attributes: device=/dev/drbd0 directory=/mnt fstype=xfs
  Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
              start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
              stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
 Group: zabbix-group
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.39 nic=ens192
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.111.39 nic=ens224
   Operations: monitor interval=30s (VirtualIP2-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)
  Resource: httpd (class=systemd type=httpd)
   Operations: monitor interval=60 timeout=100 (httpd-monitor-interval-60)
               start interval=0s timeout=100 (httpd-start-interval-0s)
               stop interval=0s timeout=100 (httpd-stop-interval-0s)
  Resource: php-fpm (class=systemd type=php-fpm)
   Operations: monitor interval=60 timeout=100 (php-fpm-monitor-interval-60)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=60 timeout=100 (zabbix-server-monitor-interval-60)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: DRBD_r0-clone
    Enabled on:
      Node: centos8-str7 (score:100) (id:location-DRBD_r0-clone-centos8-str7-100)
Ordering Constraints:
  promote DRBD_r0-clone then start FS_DRBD0 (kind:Mandatory) (id:order-DRBD_r0-clone-FS_DRBD0-mandatory)
  start FS_DRBD0 then start zabbix-group (kind:Mandatory) (id:order-FS_DRBD0-zabbix-group-mandatory)
Colocation Constraints:
  DRBD_r0-clone with FS_DRBD0 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-FS_DRBD0-INFINITY)
  DRBD_r0-clone with MariaDB (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-MariaDB-INFINITY)
  DRBD_r0-clone with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-VirtualIP-INFINITY)
  DRBD_r0-clone with VirtualIP2 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-VirtualIP2-INFINITY)
  DRBD_r0-clone with httpd (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-httpd-INFINITY)
  DRBD_r0-clone with php-fpm (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-php-fpm-INFINITY)
  DRBD_r0-clone with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-zabbix-server-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    failure-timeout=180s
    migration-threshold=3
    resource-stickiness=INFINITY
Operations Defaults:
  No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-9.el8_4.1-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1643265079
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

ーーーーー　動作確認のための作業記録　ーーーーー

※片方のノードで実行
[root@centos8-str7 ~]# pcs cluster setup --start bigbang centos8-str7 addr=10.0.0.48 addr=192.168.111.48 centos8-str8 addr=10.0.0.49 addr=192.168.111.49
[root@centos8-str7 ~]# pcs node standby centos8-str8
[root@centos8-str7 ~]# pcs property set stonith-enabled=false
[root@centos8-str7 ~]# pcs property set no-quorum-policy=ignore
[root@centos8-str7 ~]# pcs resource defaults failure-timeout=300s
[root@centos8-str7 ~]# pcs resource defaults migration-threshold=3
[root@centos8-str7 ~]# pcs resource defaults migration-threshold=1
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。
[root@centos8-str7 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。

[root@centos8-str7 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str7 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
[root@centos8-str7 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str7=100　←　①
　↑　これを設定するとcentos8-str7が復帰した場合、必ずcentos8-str7がMasterとなってしまう。
　↑　centos8-str3・4では設定したが環境が違うためなのか、設定する必要がないようだ。

[root@centos8-str7 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str7=100
　↑　（何回か試験したのだけれど・・・）クラスタをdestroyして、再設定したらcentos8-str3・4と同様の動作をするようになった

[root@centos8-str7 ~]# pcs resource cleanup DRBD_r0

[root@centos8-str7 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs
## DRBD_r0がMaster側のノードでFS_DRBD0リソースを起動するように設定
[root@centos8-str7 ~]# pcs constraint colocation add DRBD_r0-clone with Master FS_DRBD0
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with FS_DRBD0 INFINITY
## DRBD_r0起動後にFS_DRBD0リソースを起動
[root@centos8-str7 ~]# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0　←　②
　↑　これを設定するとDRBD_r0-clone及びFS_DRBD0がStartしない。
　↑　centos8-str3・4では設定したが環境が違うためなのか、設定する必要がないようだ。

[root@centos8-str7 ~]# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0
　↑　（何回か試験したのだけれど・・・）クラスタをdestroyして、再設定したらcentos8-str3・4と同様の動作をするようになった

　①及び②を設定していない場合、クラスタを再起動（pcs cluster stop --all;pcs cluster start --all）するとエラーが発生し、必ず起動しなくなってしまう。
　①及び②を設定していない場合、クラスタを確実に起動するにはいずれか一方のノードを先に起動し、全サービス起動後、他方のノードを起動すると正常にクラスタとして機能する。

[root@centos8-str7 ~]# pcs resource create MariaDB systemd:mariadb --group zabbix-group
[root@centos8-str7 ~]# pcs resource cleanup
## MariaDBはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with MariaDB INFINITY
## DRBD_r0がMaster側のノードでzabbix-groupを起動するように設定
[root@centos8-str7 ~]# pcs constraint order promote DRBD_r0-clone then start zabbix-group
## FS_DRBD0が起動してからzabbix-groupを起動するように設定
[root@centos8-str7 ~]# pcs constraint order FS_DRBD0 then start zabbix-group

[root@centos8-str7 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.39 cidr_netmask=24 nic=ens192 op monitor interval=30s --group zabbix-group
## VirtualIPはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP INFINITY
[root@centos8-str7 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.111.39 cidr_netmask=24 nic=ens224 op monitor interval=30s --group zabbix-group
## VirtualIP2はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP2 INFINITY

[root@centos8-str7 ~]# pcs resource create httpd systemd:httpd --group zabbix-group
## httpdはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with httpd INFINITY

[root@centos8-str7 ~]# pcs resource create php-fpm systemd:php-fpm --group zabbix-group
## php-fpmはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with php-fpm INFINITY

[root@centos8-str7 ~]# pcs resource create zabbix-server systemd:zabbix-server --group zabbix-group
## zabbix-serverはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str7 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-server INFINITY

## zabbix-groupはDRBD_r0のMasterと同じノードで起動
[root@centos8-str1 ~]# pcs constraint colocation add zabbix-group with Master DRBD_r0-clone INFINITY
　↑　この設定をしなくても正常にMaster/Slaveが切り替わります。

[root@centos8-str7 ~]# pcs resource cleanup
Cleaned up all resources on all nodes

※クラスタの状態を確認します。
[root@centos8-str7 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str8 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Tue Jan 25 17:11:00 2022
  * Last change:  Tue Jan 25 17:10:10 2022 by root via crm_resource on centos8-str7
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Online: [ centos8-str7 centos8-str8 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str7 ]
    * Slaves: [ centos8-str8 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str7
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str7
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * httpd	(systemd:httpd):	 Started centos8-str7
    * php-fpm	(systemd:php-fpm):	 Started centos8-str7
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str7

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

下記コマンドをコピペして実行することで、障害等により問題が発生（単純なノード追加で想定しているクラスタ構成が戻らない等）したクラスタの再設定を短時間で設定することが可能です。

現在のクラスタ設定を破棄します。
※両ノードで実行
# pcs cluster stop　--all
# pcs cluster destroy

※両ノードで実行
# systemctl enable --now pcsd

※片方のノードで実行
[root@centos8-str3 ~]# pcs cluster setup --start bigbang centos8-str3 addr=10.0.0.43 addr=192.168.111.43 centos8-str4 addr=10.0.0.44 addr=192.168.111.44
[root@centos8-str3 ~]# pcs property set stonith-enabled=false
[root@centos8-str3 ~]# pcs property set no-quorum-policy=ignore
[root@centos8-str3 ~]# pcs resource defaults failure-timeout=180s
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。
[root@centos8-str3 ~]# pcs resource defaults migration-threshold=3
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。
[root@centos8-str3 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values
※Warningは無視してください。

[root@centos8-str3 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str3 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
[root@centos8-str3 ~]# pcs resource cleanup DRBD_r0
[root@centos8-str3 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs
[root@centos8-str3 ~]# pcs resource create MariaDB systemd:mariadb --group zabbix-group
[root@centos8-str3 ~]# pcs resource cleanup
[root@centos8-str3 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=1.0.0.40 cidr_netmask=24 nic=ens192 op monitor interval=30s --group zabbix-group
[root@centos8-str3 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.111.40 cidr_netmask=24 nic=ens224 op monitor interval=30s --group zabbix-group
[root@centos8-str3 ~]# pcs resource create httpd systemd:httpd --group zabbix-group
[root@centos8-str3 ~]# pcs resource create php-fpm systemd:php-fpm --group zabbix-group
[root@centos8-str3 ~]# pcs resource create zabbix-server systemd:zabbix-server --group zabbix-group

## DRBD_r0は優先的にcentos8-str3がMasterとして起動するように設定
[root@centos8-str3 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str3=100
## DRBD_r0がMaster側のノードでFS_DRBD0リソースを起動するように設定
[root@centos8-str3 ~]# pcs constraint order promote DRBD_r0-clone then start FS_DRBD0
## FS_DRBD0はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with FS_DRBD0 INFINITY
## MariaDBはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with MariaDB INFINITY
## VirtualIPはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP INFINITY
## VirtualIP2はDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with VirtualIP2 INFINITY
## httpdはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with httpd INFINITY
## php-fpmはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with php-fpm INFINITY
## zabbix-serverはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-server INFINITY

## FS_DRBD0が起動してからzabbix-groupを起動するように設定
[root@centos8-str3 ~]# pcs constraint order FS_DRBD0 then start zabbix-group

クラスタの停止
[root@centos8-str7 ~]# pcs cluster stop --all

centos8-str8側でクラスタ設定の破棄
[root@centos8-str8 ~]# pcs cluster destroy
Shutting down pacemaker/corosync services...
Killing any remaining services...
Removing all cluster configuration files...

centos8-str7側でクラスタを起動（centos8-str8の設定が破棄されているため起動しない）
[root@centos8-str7 ~]# pcs cluster start --all
centos8-str8: Error connecting to centos8-str8 - (HTTP error: 400)
centos8-str7: Starting Cluster...
Error: unable to start all nodes

centos8-str8をクラスタから削除
[root@centos8-str7 ~]# pcs cluster node remove centos8-str8
Destroying cluster on hosts: 'centos8-str8'...
centos8-str8: Successfully destroyed cluster
Sending updated corosync.conf to nodes...
centos8-str7: Succeeded
centos8-str7: Corosync configuration reloaded

クラスタの状態を確認
centos8-str7単体で起動している
[root@centos8-str7 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str7 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Sat Jan 29 20:44:21 2022
  * Last change:  Sat Jan 29 20:41:09 2022 by hacluster via crm_node on centos8-str7
  * 1 node configured
  * 9 resource instances configured

Node List:
  * Online: [ centos8-str7 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str7 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str7
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str7
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * httpd	(systemd:httpd):	 Started centos8-str7
    * php-fpm	(systemd:php-fpm):	 Started centos8-str7
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str7

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

centos8-str8をクラスタに追加
[root@centos8-str7 ~]# pcs cluster node add centos8-str8 addr=10.0.0.49 addr=192.168.111.49
Disabling sbd...
centos8-str8: sbd disabled
Sending 'corosync authkey', 'pacemaker authkey' to 'centos8-str8'
centos8-str8: successful distribution of the file 'corosync authkey'
centos8-str8: successful distribution of the file 'pacemaker authkey'
Sending updated corosync.conf to nodes...
centos8-str8: Succeeded
centos8-str7: Succeeded
centos8-str7: Corosync configuration reloaded

centos8-str8のクラスタを起動
[root@centos8-str7 ~]# pcs cluster start centos8-str8
centos8-str8: Starting Cluster...

クラスタの状態を確認
centos8-str8が正常に動作しているように見える
[root@centos8-str7 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str7 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Sat Jan 29 20:52:00 2022
  * Last change:  Sat Jan 29 20:51:27 2022 by hacluster via crmd on centos8-str7
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Online: [ centos8-str7 centos8-str8 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str7 ]
    * Slaves: [ centos8-str8 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str7
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str7
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * httpd	(systemd:httpd):	 Started centos8-str7
    * php-fpm	(systemd:php-fpm):	 Started centos8-str7
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str7

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

centos8-str7をスタンバイにして、centos8-str8にフェールオーバーするか確認します。
[root@centos8-str7 ~]# pcs node standby centos8-str7

正常にフェールオーバーし、centos8-str8でクラスタが動作しています。
[root@centos8-str8 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str7 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Sat Jan 29 20:54:57 2022
  * Last change:  Sat Jan 29 20:54:32 2022 by root via cibadmin on centos8-str7
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Node centos8-str7: standby
  * Online: [ centos8-str8 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str8 ]
    * Stopped: [ centos8-str7 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str8
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str8
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str8
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str8
    * httpd	(systemd:httpd):	 Started centos8-str8
    * php-fpm	(systemd:php-fpm):	 Started centos8-str8
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str8

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

centos8-str8をスタンバイにして、centos8-str7にフェールバックするか確認します。
[root@centos8-str7 ~]# pcs node unstandby centos8-str7

正常にフェールバックしています。
[root@centos8-str8 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str7 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Sat Jan 29 20:58:33 2022
  * Last change:  Sat Jan 29 20:58:00 2022 by root via cibadmin on centos8-str7
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Node centos8-str8: standby
  * Online: [ centos8-str7 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str7 ]
    * Stopped: [ centos8-str8 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str7
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str7
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * httpd	(systemd:httpd):	 Started centos8-str7
    * php-fpm	(systemd:php-fpm):	 Started centos8-str7
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str7

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

centos8-str8をアンスタンバイにし、クラスタに組み込みます。
[root@centos8-str7 ~]# pcs node unstandby centos8-str8

centos8-str8が再度、組み込まれました。
[root@centos8-str8 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str7 (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Sat Jan 29 21:03:26 2022
  * Last change:  Sat Jan 29 21:02:31 2022 by root via cibadmin on centos8-str7
  * 2 nodes configured
  * 9 resource instances configured

Node List:
  * Online: [ centos8-str7 centos8-str8 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str7 ]
    * Slaves: [ centos8-str8 ]
  * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str7
  * Resource Group: zabbix-group:
    * MariaDB	(systemd:mariadb):	 Started centos8-str7
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str7
    * httpd	(systemd:httpd):	 Started centos8-str7
    * php-fpm	(systemd:php-fpm):	 Started centos8-str7
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str7

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

• MariaDBのインストール及びMaster/Slaveの構築
  「初期設定（CentOS 8）」参照
  レプリケーションの設定の「バージョン10.3系」参照
　
• Zabbix Serverのインストール及び設定
  「pacemakerでZabbix-Server及びMariaDBのMaster/Slaveを制御する」参照
  
　
• DRBDによる共有ディスクの設定
  「DRBDの構築について」参照
  「DRBDをpacemakerの現在のリソースに追加する」参照

[root@centos8-str3 ~]# pcs property set stonith-enabled=false
[root@centos8-str3 ~]# pcs property set no-quorum-policy=ignore

[root@centos8-str3 ~]# pcs resource defaults resource-stickiness=INFINITY
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1
Warning: This command is deprecated and will be removed. Please use 'pcs resource defaults update' instead.
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str3 ~]# pcs node standby centos8-str4

• DRBD
  DRBDを設定しブロックデバイスをリアルタイムにレプリケート。Primary/Secondary。MariaDBのMaster側で稼働
• MariaDB
  各ホストに保存されたデータベースを読み込んでMariaDBを起動。Master/Slave。
• Apache
  httpdサービスを起動。Active/Standby。MariaDBのMaster側で稼働
• php-fpm
  php-fpmサービスを起動。MariaDBのMaster側で稼働
• Zabbix-Server
  zabbix-serverサービスを起動。Active/Standby。MariaDBのMaster側で稼働
• VirtualIP
  仮想IP（サービス側）の割り当て。MariaDBのMaster側で稼働
• VirtualIP2
  仮想IP（DRDB同期用）の割り当て。MariaDBのMaster側で稼働

[root@centos8-str3 ~]# pcs resource create DRBD_r0 ocf:linbit:drbd drbd_resource=r0
[root@centos8-str3 ~]# pcs resource promotable DRBD_r0 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

[root@centos8-str3 ~]# pcs constraint location DRBD_r0-clone prefers centos8-str3=100
[root@centos8-str3 ~]# pcs resource cleanup DRBD_r0
[root@centos8-str3 ~]# pcs resource create FS_DRBD0 ocf:heartbeat:Filesystem device=/dev/drbd0 directory=/mnt fstype=xfs --group zabbix-group

[root@centos8-str3 ~]# pcs resource create MariaDB ocf:heartbeat:mysql binary=/usr/bin/mysqld_safe \
datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_user=repl \
replication_passwd=[パスワード] op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource promotable MariaDB master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

[root@centos8-str3 ~]# pcs constraint location MariaDB-clone prefers centos8-str3=100

[root@centos8-str3 ~]# pcs resource create zabbix-server systemd:zabbix-server op monitor interval="10s" timeout="20s"

# vi /etc/httpd/conf.d/status.conf
ExtendedStatus On
 
<Location /server-status>
    SetHandler server-status
    Require local
</Location>

[root@centos8-str3 ~]# pcs resource create Apache ocf:heartbeat:apache configfile=/etc/httpd/conf/httpd.conf \
statusurl="http://localhost/server-status" op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource create php-fpm systemd:php-fpm op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=10.0.0.40 \
cidr_netmask=24 nic=ens192 op monitor interval=10s timeout=10s
[root@centos8-str3 ~]# pcs resource create VirtualIP2 ocf:heartbeat:IPaddr2 ip=192.168.0.40 \
cidr_netmask=24 nic=ens224 op monitor interval=10s timeout=10s

[root@centos8-str3 ~]# pcs resource group add zabbix-group Apache php-fpm zabbix-server VirtualIP VirtualIP2

# MariaDBのcloneにはFS_DRBD0が
[root@centos8-str3 ~]# pcs constraint colocation add FS_DRBD0 with MariaDB-clone INFINITY with-rsc-role=Master

# MariaDB起動後にzabbix-groupリソースを起動
[root@centos8-str3 ~]# pcs constraint order promote MariaDB-clone then start zabbix-group
Adding MariaDB-clone zabbix-group (kind: Mandatory) (Options: first-action=promote then-action=start)

# zabbix-groupはMariaDBのMasterと同じノードで起動
[root@centos8-str3 ~]# pcs constraint colocation add zabbix-group with master MariaDB-clone INFINITY

# zabbix-serverはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with zabbix-server INFINITY

# ApacheはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with Apache INFINITY

# php-fpmはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with php-fpm INFINITY

# VirtualIPはMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with VirtualIP INFINITY

# VirtualIP2はMariaDBのMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master MariaDB-clone with VirtualIP2 INFINITY

# zabbix-groupはDRBD_r0のMasterと同じノードで起動（異常検知で必要）
[root@centos8-str3 ~]# pcs constraint colocation add master DRBD_r0-clone with zabbix-group INFINITY

[root@centos8-str3 ~]# pcs node unstandby centos8-str4
[root@centos8-str3 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:04:08 2021
  * Last change:  Thu Apr 22 16:45:44 2021 by root via cibadmin on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 centos8-str4 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str3 ]
    * Slaves: [ centos8-str4 ]
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * Masters: [ centos8-str3 ]
    * Slaves: [ centos8-str4 ]

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs config show
Cluster Name: bigbang
Corosync Nodes:
 centos8-str3 centos8-str4
Pacemaker Nodes:
 centos8-str3 centos8-str4

Resources:
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Group: zabbix-group
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: Apache (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status
   Operations: monitor interval=10s timeout=10s (Apache-monitor-interval-10s)
               start interval=0s timeout=40s (Apache-start-interval-0s)
               stop interval=0s timeout=60s (Apache-stop-interval-0s)
  Resource: php-fpm (class=systemd type=php-fpm)
   Operations: monitor interval=10s timeout=10s (php-fpm-monitor-interval-10s)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=10s timeout=20s (zabbix-server-monitor-interval-10s)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.40 nic=ens192
   Operations: monitor interval=10s timeout=10s (VirtualIP-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.40 nic=ens224
   Operations: monitor interval=10s timeout=10s (VirtualIP2-monitor-interval-10s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)
 Clone: MariaDB-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: MariaDB (class=ocf provider=heartbeat type=mysql)
   Attributes: binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_passwd=[パスワード] replication_user=repl
   Operations: demote interval=0s timeout=120s (MariaDB-demote-interval-0s)
               monitor interval=10s timeout=10s (MariaDB-monitor-interval-10s)
               notify interval=0s timeout=90s (MariaDB-notify-interval-0s)
               promote interval=0s timeout=120s (MariaDB-promote-interval-0s)
               start interval=0s timeout=120s (MariaDB-start-interval-0s)
               stop interval=0s timeout=120s (MariaDB-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: DRBD_r0-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-DRBD_r0-clone-centos8-str3-100)
  Resource: MariaDB-clone
    Enabled on:
      Node: centos8-str3 (score:100) (id:location-MariaDB-clone-centos8-str3-100)
Ordering Constraints:
  promote MariaDB-clone then start zabbix-group (kind:Mandatory) (id:order-MariaDB-clone-zabbix-group-mandatory)
Colocation Constraints:
  zabbix-group with MariaDB-clone (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-zabbix-group-MariaDB-clone-INFINITY)
  MariaDB-clone with zabbix-server (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-zabbix-server-INFINITY)
  MariaDB-clone with Apache (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-Apache-INFINITY)
  MariaDB-clone with php-fpm (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-php-fpm-INFINITY)
  MariaDB-clone with VirtualIP (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP-INFINITY)
  MariaDB-clone with VirtualIP2 (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-MariaDB-clone-VirtualIP2-INFINITY)
  DRBD_r0-clone with zabbix-group (score:INFINITY) (rsc-role:Master) (with-rsc-role:Started) (id:colocation-DRBD_r0-clone-zabbix-group-INFINITY)
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
    resource-stickiness=INFINITY
Operations Defaults:
  No defaults set

Cluster Properties:
 MariaDB_REPL_INFO: centos8-str3.bigbang.mydns.jp|mysql-bin.000009|342
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 2.0.5-9.el8-ba59be7122
 have-watchdog: false
 last-lrm-refresh: 1619074041
 no-quorum-policy: ignore
 stonith-enabled: false

Tags:
 No tags defined

Quorum:
  Options:

[root@centos8-str3 ~]# pcs resource defaults migration-threshold=1

[root@centos8-str3 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str3 ~]# ps axu | grep http[d]
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:37:59 2021
  * Last change:  Mon Apr 26 11:37:37 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 11:37:30 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str3 'not running' (7): call=377, status='complete', exitreason='', last-rc-change='2021-04-26 11:37:30 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f httpd`
[root@centos8-str4 ~]# ps axu | grep http[d]
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 11:49:00 2021
  * Last change:  Mon Apr 26 11:48:48 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * Apache: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 11:48:41 2021'

Failed Resource Actions:
  * Apache_monitor_10000 on centos8-str4 'not running' (7): call=375, status='complete', exitreason='', last-rc-change='2021-04-26 11:48:42 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup Apache
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)

[root@centos8-str3 ~]# kill -kill `pgrep -f php-fpm`
[root@centos8-str3 ~]# ps axu | grep -v grep | grep php-fpm
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str4 (2) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Tue Apr 27 08:25:01 2021
  * Last change:  Tue Apr 27 08:24:54 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * php-fpm: migration-threshold=1 fail-count=1 last-failure='Tue Apr 27 08:24:44 2021'

Failed Resource Actions:
  * php-fpm_monitor_10000 on centos8-str3 'not running' (7): call=112, status='complete', exitreason='', last-rc-change='2021-04-27 08:24:45 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup php-fpm
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f php-fpm`
[root@centos8-str4 ~]# ps axu | grep -v grep | grep php-fpm
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str4 (2) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Tue Apr 27 08:32:50 2021
  * Last change:  Tue Apr 27 08:32:36 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * php-fpm: migration-threshold=1 fail-count=1 last-failure='Tue Apr 27 08:32:27 2021'

Failed Resource Actions:
  * php-fpm_monitor_10000 on centos8-str4 'not running' (7): call=129, status='complete', exitreason='', last-rc-change='2021-04-27 08:32:27 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup php-fpm
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)

[root@centos8-str3 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str3 ~]# ps axu | grep -v grep | grep zabbix_server
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 15:50:07 2021
  * Last change:  Mon Apr 26 15:49:55 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000
    * master-MariaDB                  	: 1
    * readable                        	: 1
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000
    * master-MariaDB                  	: 3601
    * readable                        	: 1

Migration Summary:
  * Node: centos8-str3 (1):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 15:49:46 2021'

Failed Resource Actions:
  * zabbix-server_monitor_10000 on centos8-str3 'not running' (7): call=440, status='complete', exitreason='', last-rc-change='2021-04-26 15:49:46 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[
root@centos8-str4 ~]# pcs resource cleanup zabbix-server
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# kill -kill `pgrep -f zabbix_server`
[root@centos8-str4 ~]# ps axu | grep -v grep | grep zabbix_server
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 15:59:14 2021
  * Last change:  Mon Apr 26 15:58:50 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str4 (2):
    * zabbix-server: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 15:58:41 2021'

Failed Resource Actions:
  * zabbix-server_monitor_10000 on centos8-str4 'not running' (7): call=434, status='complete', exitreason='', last-rc-change='2021-04-26 15:58:41 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup zabbix-server
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)

[root@centos8-str3 ~]# pcs resource update VirtualIP op monitor on-fail=standby
[root@centos8-str3 ~]# pcs resource update VirtualIP2 op monitor on-fail=standby
[root@centos8-str3 ~]# pcs resource config
 Clone: DRBD_r0-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: DRBD_r0 (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=r0
   Operations: demote interval=0s timeout=90 (DRBD_r0-demote-interval-0s)
               monitor interval=20 role=Slave timeout=20 (DRBD_r0-monitor-interval-20)
               monitor interval=10 role=Master timeout=20 (DRBD_r0-monitor-interval-10)
               notify interval=0s timeout=90 (DRBD_r0-notify-interval-0s)
               promote interval=0s timeout=90 (DRBD_r0-promote-interval-0s)
               reload interval=0s timeout=30 (DRBD_r0-reload-interval-0s)
               start interval=0s timeout=240 (DRBD_r0-start-interval-0s)
               stop interval=0s timeout=100 (DRBD_r0-stop-interval-0s)
 Group: zabbix-group
  Resource: FS_DRBD0 (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd0 directory=/mnt fstype=xfs
   Operations: monitor interval=20s timeout=40s (FS_DRBD0-monitor-interval-20s)
               start interval=0s timeout=60s (FS_DRBD0-start-interval-0s)
               stop interval=0s timeout=60s (FS_DRBD0-stop-interval-0s)
  Resource: Apache (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status
   Operations: monitor interval=10s timeout=10s (Apache-monitor-interval-10s)
               start interval=0s timeout=40s (Apache-start-interval-0s)
               stop interval=0s timeout=60s (Apache-stop-interval-0s)
  Resource: php-fpm (class=systemd type=php-fpm)
   Operations: monitor interval=10s timeout=10s (php-fpm-monitor-interval-10s)
               start interval=0s timeout=100 (php-fpm-start-interval-0s)
               stop interval=0s timeout=100 (php-fpm-stop-interval-0s)
  Resource: zabbix-server (class=systemd type=zabbix-server)
   Operations: monitor interval=10s timeout=20s (zabbix-server-monitor-interval-10s)
               start interval=0s timeout=100 (zabbix-server-start-interval-0s)
               stop interval=0s timeout=100 (zabbix-server-stop-interval-0s)
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=10.0.0.40 nic=ens192
   Operations: monitor interval=60s on-fail=standby (VirtualIP-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: VirtualIP2 (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.40 nic=ens224
   Operations: monitor interval=60s on-fail=standby (VirtualIP2-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP2-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP2-stop-interval-0s)
 Clone: MariaDB-clone
  Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true promotable=true
  Resource: MariaDB (class=ocf provider=heartbeat type=mysql)
   Attributes: binary=/usr/bin/mysqld_safe datadir=/var/lib/mysql log=/var/log/mariadb/mariadb.log pid=/run/mariadb/mariadb.pid replication_passwd=[パスワード] replication_user=repl
   Operations: demote interval=0s timeout=120s (MariaDB-demote-interval-0s)
               monitor interval=10s timeout=10s (MariaDB-monitor-interval-10s)
               notify interval=0s timeout=90s (MariaDB-notify-interval-0s)
               promote interval=0s timeout=120s (MariaDB-promote-interval-0s)
               start interval=0s timeout=120s (MariaDB-start-interval-0s)
               stop interval=0s timeout=120s (MariaDB-stop-interval-0s)

[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 16:34:57 2021
  * Last change:  Mon Apr 26 16:34:36 2021 by root via crm_attribute on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Node centos8-str3 (1): standby (on-fail)
  * Online: [ centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Stopped
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Stopped

Node Attributes:
  * Node: centos8-str3 (1):
    * readable                        	: 0         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:
  * Node: centos8-str3 (1):
    * VirtualIP: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 16:34:24 2021'

Failed Resource Actions:
  * VirtualIP_monitor_60000 on centos8-str3 'not running' (7): call=500, status='complete', exitreason='', last-rc-change='2021-04-26 16:34:24 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Offline
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str4 ~]# pcs resource cleanup VirtualIP
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str4 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str3 (1) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 16:44:15 2021
  * Last change:  Mon Apr 26 16:43:06 2021 by hacluster via crmd on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str4
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str4
    * php-fpm	(systemd:php-fpm):	 Started centos8-str4
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str4
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str4
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 0         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         

Migration Summary:

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str4 (2) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 17:51:47 2021
  * Last change:  Mon Apr 26 17:49:22 2021 by root via crm_attribute on centos8-str3
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Node centos8-str4 (2): standby (on-fail)
  * Online: [ centos8-str3 (1) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
    * DRBD_r0	(ocf::linbit:drbd):	 Stopped
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3
    * MariaDB	(ocf::heartbeat:mysql):	 Stopped

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * readable                        	: 0         

Migration Summary:
  * Node: centos8-str4 (2):
    * VirtualIP: migration-threshold=1 fail-count=1 last-failure='Mon Apr 26 17:49:11 2021'

Failed Resource Actions:
  * VirtualIP_monitor_60000 on centos8-str4 'not running' (7): call=70, status='complete', exitreason='', last-rc-change='2021-04-26 17:49:11 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Offline

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str3 ~]# pcs resource cleanup VirtualIP
Cleaned up FS_DRBD0 on centos8-str4
Cleaned up FS_DRBD0 on centos8-str3
Cleaned up Apache on centos8-str4
Cleaned up Apache on centos8-str3
Cleaned up php-fpm on centos8-str4
Cleaned up php-fpm on centos8-str3
Cleaned up zabbix-server on centos8-str4
Cleaned up zabbix-server on centos8-str3
Cleaned up VirtualIP on centos8-str4
Cleaned up VirtualIP on centos8-str3
Cleaned up VirtualIP2 on centos8-str4
Cleaned up VirtualIP2 on centos8-str3
Waiting for 1 reply from the controller
... got reply (done)
[root@centos8-str3 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str4 (2) (version 2.0.5-9.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Apr 26 17:55:55 2021
  * Last change:  Mon Apr 26 17:55:17 2021 by hacluster via crmd on centos8-str4
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Online: [ centos8-str3 (1) centos8-str4 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str4
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str3
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str3
    * Apache	(ocf::heartbeat:apache):	 Started centos8-str3
    * php-fpm	(systemd:php-fpm):	 Started centos8-str3
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str3
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
    * VirtualIP2	(ocf::heartbeat:IPaddr2):	 Started centos8-str3
  * Clone Set: MariaDB-clone [MariaDB] (promotable):
    * MariaDB	(ocf::heartbeat:mysql):	 Slave centos8-str4
    * MariaDB	(ocf::heartbeat:mysql):	 Master centos8-str3

Node Attributes:
  * Node: centos8-str3 (1):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 3601      
    * readable                        	: 1         
  * Node: centos8-str4 (2):
    * master-DRBD_r0                  	: 10000     
    * master-MariaDB                  	: 1         
    * readable                        	: 0         

Migration Summary:

Tickets:

PCSD Status:
  centos8-str3: Online
  centos8-str4: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

※両ノードで実施
# yum install pacemaker
# yum install pcs
# yum install fence-agents-al

※両ノードで実施
[root@centos7-1 ~]# cat /etc/passwd | grep hacluster
hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin

[root@centos7-2 ~]# cat /etc/passwd | grep hacluster
hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin

※両ノードで実施
# passwd hacluster
ユーザ hacluster のパスワードを変更。
新しいパスワード:
新しいパスワードを再入力してください:
passwd: すべての認証トークンが正しく更新できました。

※両ノードで実施
# systemctl start pcsd.service
# systemctl enable pcsd.service

※両ノードで実施
# systemctl status pcsd.service
● pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
   Active: active (running) since 火 2019-07-02 14:56:38 JST; 35s ago
     Docs: man:pcsd(8)
           man:pcs(8)
 Main PID: 11533 (pcsd)
   CGroup: /system.slice/pcsd.service
           └─11533 /usr/bin/ruby /usr/lib/pcsd/pcsd
 7月 02 14:56:36 centos7-1.bigbang.mydns.jp systemd[1]: Starting PCS GUI and remote configur.....
 7月 02 14:56:38 centos7-1.bigbang.mydns.jp systemd[1]: Started PCS GUI and remote configura...e.
Hint: Some lines were ellipsized, use -l to show in full.

※両ノードで実施
# vi /etc/hosts
------------------------------
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.11 centos7-1c centos7-1c.bigbang.mydns.jp　←　追加
192.168.0.12 centos7-2c centos7-2c.bigbang.mydns.jp　←　追加
192.168.10.11 centos7-1 centos7-1.bigbang.mydns.jp　←　追加
192.168.10.12 centos7-2 centos7-2.bigbang.mydns.jp　←　追加

※両ノードで実施
# firewall-cmd --add-service=high-availability --permanent
success

# firewall-cmd --reload
success

# firewall-cmd --list-service
ssh dhcpv6-client https http high-availability

※片方のノードで実施
※この場合、pcs cluster setup --start・・・で、認証を実施していないノードが含まれていると失敗します。
[root@centos7-1 ~]# pcs cluster auth centos7-1c centos7-2c
Username: hacluster
Password: 
centos7-1c: Authorized
centos7-2c: Authorized

※片方のノードで実施
※今回はスプリットブレインのことは考えず、centos7-1c及びcentos7-2cで設定します。
[root@centos7-1 ~]# pcs cluster setup --start --name bigbang-cluster1 centos7-1,centos7-1c centos7-2,centos7-2c
Error: centos7-1: unable to authenticate to node
Error: centos7-2: unable to authenticate to node
Error: nodes availability check failed, use --force to override. WARNING: This will destroy existing cluster on the nodes.
pcs cluster auth実行時、centos7-1及びcentos7-2を含めていなかったのが原因です。

[root@centos7-1 ~]# pcs cluster setup --start --name bigbang-cluster1 centos7-1c centos7-2c
Destroying cluster on nodes: centos7-1c, centos7-2c...
centos7-1c: Stopping Cluster (pacemaker)...
centos7-2c: Stopping Cluster (pacemaker)...
centos7-1c: Successfully destroyed cluster
centos7-2c: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'centos7-1c', 'centos7-2c'
centos7-1c: successful distribution of the file 'pacemaker_remote authkey'
centos7-2c: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
centos7-1c: Succeeded
centos7-2c: Succeeded
Starting cluster on nodes: centos7-1c, centos7-2c...
centos7-1c: Starting Cluster (corosync)...
centos7-2c: Starting Cluster (corosync)...
centos7-2c: Starting Cluster (pacemaker)...
centos7-1c: Starting Cluster (pacemaker)...
Synchronizing pcsd certificates on nodes centos7-1c, centos7-2c...
centos7-1c: Success
centos7-2c: Success
Restarting pcsd on the nodes in order to reload the certificates...
centos7-1c: Success
centos7-2c: Success

設定確認は以下コマンドで実施します。RING ID 0で1つのIPアドレスが設定されていることがわかります。
# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 192.168.0.11
	status	= ring 0 active with no faults

※今回はスプリットブレイン防止を考慮する方法では、pcs cluster authコマンドで先にcentos7-1,centos7-1c centos7-2,centos7-2cの
設定をしておく必要があります。設定していない場合、下記のようにエラーとなります。
# pcs cluster setup --start --name bigbang-cluster1 centos7-1,centos7-1c centos7-2,centos7-2c
Error: centos7-1: unable to authenticate to node
Error: centos7-2: unable to authenticate to node
Error: nodes availability check failed, use --force to override. WARNING: This will destroy existing cluster on the nodes.

4台のノードを設定する場合は、下記のとおりとなります。
※片方のノードで実施
[root@centos7-1 ~]# pcs cluster auth centos7-1c centos7-2c centos7-1 centos7-2
Username: hacluster
Password: 
centos7-1: Authorized
centos7-1c: Authorized
centos7-2: Authorized
centos7-2c: Authorized

※片方のノードで実施
[root@centos7-1 ~]# pcs cluster setup --start --name bigbang-cluster1 centos7-1,centos7-1c centos7-2,centos7-2c --force
Destroying cluster on nodes: centos7-1, centos7-2...
centos7-1: Stopping Cluster (pacemaker)...
centos7-2: Stopping Cluster (pacemaker)...
centos7-1: Successfully destroyed cluster
centos7-2: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'centos7-1', 'centos7-2'
centos7-1: successful distribution of the file 'pacemaker_remote authkey'
centos7-2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
centos7-1: Succeeded
centos7-2: Succeeded
Starting cluster on nodes: centos7-1, centos7-2...
centos7-1: Starting Cluster (corosync)...
centos7-2: Starting Cluster (corosync)...
centos7-2: Starting Cluster (pacemaker)...
centos7-1: Starting Cluster (pacemaker)...
Synchronizing pcsd certificates on nodes centos7-1, centos7-2...
centos7-1: Success
centos7-2: Success
Restarting pcsd on the nodes in order to reload the certificates...
centos7-1: Success
centos7-2: Success
※「--force」オプションを使用時、これまでの設定が上書きされ、クォーラムやstonishの設定等が消えますので再度設定が必要です。

※片方のノードで実施
設定確認は以下コマンドで実施します。RING ID 0とRING ID 0の2つのIPアドレスが設定されていることがわかります。
[root@centos7-1 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 192.168.10.11
	status	= ring 0 active with no faults
RING ID 1
	id	= 192.168.0.11
	status	= ring 1 active with no faults

[root@centos7-1 ~]# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang-cluster1
 dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
 have-watchdog: false

※片方のノードで実施
# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore
# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang-cluster1
 dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

• ocf:heartbeat:IPaddr2 : 仮想IPアドレスを制御
• ocf:heartbeat:Filesystem : ファイルシステムのマウントを制御
• systemd:mariadb : MariaDBを制御

# pcs resource describe <リソースエージェント>

※片方のノードで実施
# pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.10.1 cidr_netmask=24 nic=ens192 op monitor interval=30s

※片方のノードで実施
# pcs resource create ShareDir ocf:heartbeat:Filesystem device=/dev/sdb1 directory=/mnt fstype=ext4

※片方のノードで実施
# pcs resource create MariaDB systemd:mariadb

※片方のノードで実施
# pcs resource group add rg01 VirtualIP ShareDir MariaDB

※片方のノードで実施
[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1c (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Tue Jul  2 18:19:34 2019
Last change: Tue Jul  2 18:19:17 2019 by root via cibadmin on centos7-1c
2 nodes configured
3 resources configured
Online: [ centos7-1c centos7-2c ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1c
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1c
     MariaDB	(systemd:mariadb):	Started centos7-1c
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs resource show --full

※片方のノードで実施
[root@centos7-1 ~]# pcs cluster start --all
centos7-1c: Starting Cluster (corosync)...
centos7-2c: Starting Cluster (corosync)...
centos7-2c: Starting Cluster (pacemaker)...
centos7-1c: Starting Cluster (pacemaker)...

※片方のノードで実施
[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1c (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Tue Jul  2 18:19:34 2019
Last change: Tue Jul  2 18:19:17 2019 by root via cibadmin on centos7-1c
2 nodes configured
3 resources configured
Online: [ centos7-1c centos7-2c ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1c
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1c
     MariaDB	(systemd:mariadb):	Started centos7-1c
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-2 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 10:50:41 2019
Last change: Wed Jul  3 10:50:24 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-1 centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1

[root@centos7-1 ~]# pcs resource move rg01 centos7-2

[root@centos7-1 ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-2
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-2
     MariaDB	(systemd:mariadb):	Started centos7-2

[root@centos7-1 ~]# pcs resource move rg01 centos7-1

[root@centos7-1 ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 16:32:53 2019
Last change: Wed Jul  3 16:30:07 2019 by root via crm_resource on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-1 centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs cluster standby centos7-1

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 16:33:31 2019
Last change: Wed Jul  3 16:33:26 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Node centos7-1: standby
Online: [ centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-2
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-2
     MariaDB	(systemd:mariadb):	Starting centos7-2
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs cluster unstandby centos7-1
[root@centos7-1 ~]# pcs cluster standby centos7-2
[root@centos7-1 ~]# pcs cluster unstandby centos7-2

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 16:33:59 2019
Last change: Wed Jul  3 16:33:48 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-1 centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-2 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-2 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 16:38:06 2019
Last change: Wed Jul  3 16:33:48 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-2 ]
OFFLINE: [ centos7-1 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-2
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-2
     MariaDB	(systemd:mariadb):	Started centos7-2
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-2 ~]# pcs status
Error: cluster is not currently running on this node

[root@centos7-2 ~]# pcs cluster start centos7-1
centos7-1: Starting Cluster (corosync)...
centos7-1: Starting Cluster (pacemaker)...

[root@centos7-2 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-2 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Jul  3 16:42:50 2019
Last change: Wed Jul  3 16:33:48 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-1 centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Starting centos7-1
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs resource update VirtualIP op monitor on-fail=standby
[root@centos7-1 ~]# pcs resource show --full
 Group: rg01
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.10.1 nic=ens192
   Operations: monitor interval=60s on-fail=standby (VirtualIP-monitor-interval-60s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)
  Resource: ShareDir (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/sdb1 directory=/mnt fstype=ext4
   Operations: monitor interval=20s timeout=40s (ShareDir-monitor-interval-20s)
               notify interval=0s timeout=60s (ShareDir-notify-interval-0s)
               start interval=0s timeout=60s (ShareDir-start-interval-0s)
               stop interval=0s timeout=60s (ShareDir-stop-interval-0s)
  Resource: MariaDB (class=systemd type=mariadb)
   Operations: monitor interval=60 timeout=100 (MariaDB-monitor-interval-60)
               start interval=0s timeout=100 (MariaDB-start-interval-0s)
               stop interval=0s timeout=100 (MariaDB-stop-interval-0s)

root@centos7-2 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Thu Jul  4 13:36:38 2019
Last change: Thu Jul  4 13:29:54 2019 by root via cibadmin on centos7-1
2 nodes configured
3 resources configured
Node centos7-1: standby (on-fail)
Online: [ centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-2
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-2
     MariaDB	(systemd:mariadb):	Started centos7-2
Failed Actions:
* VirtualIP_monitor_60000 on centos7-1 'not running' (7): call=20, status=complete, exitreason='',
    last-rc-change='Thu Jul  4 13:35:55 2019', queued=0ms, exec=0ms
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 1 replies from the CRMd. OK

[root@centos7-1 ~]# pcs status;date
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Thu Jul  4 13:43:31 2019
Last change: Thu Jul  4 13:43:13 2019 by hacluster via crmd on centos7-1
2 nodes configured
3 resources configured
Online: [ centos7-1 centos7-2 ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1
     MariaDB	(systemd:mariadb):	Started centos7-1
Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

[root@centos7-3 ~]# systemctl status pcsd
● pcsd.service - PCS GUI and remote configuration interface
   Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
   Active: active (running) since 木 2019-07-04 15:03:14 JST; 1h 36min ago
     Docs: man:pcsd(8)
           man:pcs(8)
 Main PID: 23964 (pcsd)
    Tasks: 6
   CGroup: /system.slice/pcsd.service
           └─23964 /usr/bin/ruby /usr/lib/pcsd/pcsd
 7月 04 15:03:14 centos7-3.bigbang.mydns.jp systemd[1]: Starting PCS GUI and remote configur.....
 7月 04 15:03:14 centos7-3.bigbang.mydns.jp systemd[1]: Started PCS GUI and remote configura...e.
Hint: Some lines were ellipsized, use -l to show in full.

[root@centos7-1 ~]# pcs cluster auth centos7-1c centos7-2c centos7-3c centos7-1 centos7-2 centos7-3
Username: hacluster
Password: 
centos7-1: Authorized
centos7-3: Authorized
centos7-2: Authorized
centos7-3c: Authorized
centos7-1c: Authorized
centos7-2c: Authorized

[root@centos7-1 ~]# pcs cluster auth centos7-1c centos7-2c centos7-3c centos7-1 centos7-2 centos7-3
centos7-1: Already authorized
centos7-3: Already authorized
centos7-2: Already authorized
centos7-3c: Already authorized
centos7-1c: Already authorized
centos7-2c: Already authorized

[root@centos7-1 ~]# pcs cluster node add centos7-3
Disabling SBD service...
centos7-3: sbd disabled
Sending remote node configuration files to 'centos7-3'
centos7-3: successful distribution of the file 'pacemaker_remote authkey'
centos7-1c: Corosync updated
centos7-2c: Corosync updated
Setting up corosync...
centos7-3: Succeeded
Synchronizing pcsd certificates on nodes centos7-3...
centos7-3: Success
Restarting pcsd on the nodes in order to reload the certificates...
centos7-3: Success


※CentOS Stream 8でVirtualIP、VirtualIP2を設定していた場合、下記のように実行する必要があった
[root@centos8-1 ~]# pcs cluster node add centos8-str4 addr=10.0.0.4 addr=192.168.10.4

※失敗例
[root@centos8-1 ~]# pcs cluster node add centos8-1
No addresses specified for host 'centos8-1', using 'centos8-1'
Error: 2 addresses must be specified for a node, 1 address specified for node 'centos8-1'
Error: Errors have occurred, therefore pcs is unable to continue


centos7-3またはcentos7-3cのいずれかのノード名で実施すればよい。
[root@centos7-1 ~]# pcs cluster node add centos7-3c
Error: Unable to add 'centos7-3c' to cluster: node is already in a cluster

[root@centos7-1 ~]# pcs cluster start centos7-3
centos7-3: Starting Cluster (corosync)...
centos7-3: Starting Cluster (pacemaker)...

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-2c (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Fri Jul  5 11:09:15 2019
Last change: Fri Jul  5 11:09:04 2019 by hacluster via crmd on centos7-2c
3 nodes configured
3 resources configured
Online: [ centos7-1c centos7-2c centos7-3c ]
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1c
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1c
     MariaDB	(systemd:mariadb):	Started centos7-1c
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos7-1 ~]# pcs cluster node remove centos7-3c
centos7-3c: Stopping Cluster (pacemaker)...
centos7-3c: Successfully destroyed cluster
centos7-1c: Corosync updated
centos7-2c: Corosync updated

[root@centos7-1 ~]# pcs status
Cluster name: bigbang-cluster1
Stack: corosync
Current DC: centos7-2c (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Fri Jul  5 11:31:54 2019
Last change: Fri Jul  5 11:31:48 2019 by root via crm_node on centos7-1c
2 nodes configured
3 resources configured
Online: [ centos7-1c centos7-2c ]　←　centos7-3cが消えている
Full list of resources:
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started centos7-1c
     ShareDir	(ocf::heartbeat:Filesystem):	Started centos7-1c
     MariaDB	(systemd:mariadb):	Started centos7-1c
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

# yum -y install pacemaker corosync pcs

# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
# vi /etc/corosync/corosync.conf
compatibility: whitetank
# 以下追記
aisexec {
        user: root
        group: root
}
service {
        name: pacemaker
        ver: 0
        use_mgmtd: yes
}
totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
# インターコネクト用インターフェースのネットワークアドレスを指定
                bindnetaddr: 192.168.20.0
                # Fedora 20では、下記のアドレスに変更した
                amcastaddr: 226.94.1.1　←　不要と判断し削除
                mcastport: 5405
        }
}
logging {
        fileline: off
        to_stderr: no
        to_logfile: yes
        to_syslog: yes
        logfile: /var/log/cluster/corosync.log
        debug: off
        timestamp: on
        logger_subsys {
                # Fedora 20では、下記に変更した
                subsys: AMF
                debug: off
        }
}
# Fedora 20では、下記を追記
amf {
        mode: disabled
}
nodelist {
        node {
                ring0_addr: 192.168.20.1
                nodeid: 1
        }
        node {
                ring0_addr: 192.168.20.2
                nodeid: 2
        }
        node {
                ring0_addr: 192.168.20.3
                nodeid: 3
        }
}
quorum {
        # Enable and configure quorum subsystem (default: off)
        # see also corosync.conf.5 and votequorum.5
        provider: corosync_votequorum
        expected_votes: 3
}
# chown -R hacluster. /var/log/cluster
# /etc/rc.d/init.d/pacemaker start  または　systemctl start pacemaker
# /etc/rc.d/init.d/corosync start  または　systemctl start corosync
　↑　pacemakerが起動すると連動するので削除
# chkconfig pacemaker on  または　systemctl enable pacemaker
# chkconfig corosync on  または　systemctl enable corosync
　↑　pacemakerが起動すると連動するので削除

# yum -y install pacemaker corosync pcs

# vi /etc/hosts
192.168.20.1	server01
192.168.20.2	server02

# systemctl start pcsd
# systemctl enable pcsd

# passwd hacluster
ユーザ hacluster のパスワードを変更。
新しいパスワード:
新しいパスワードを再入力してください:
passwd: すべての認証トークンが正しく更新できました。

# pcs host auth server01 server02
Username: hacluster
Password: 
server02: Authorized
server01: Authorized

# pcs cluster setup --name mycluster server01 server02 --force
Destroying cluster on nodes: server01, server02...
server01: Stopping Cluster (pacemaker)...
server02: Stopping Cluster (pacemaker)...
server02: Successfully destroyed cluster
server01: Successfully destroyed cluster
Sending cluster config files to the nodes...
server01: Succeeded
server02: Succeeded
Synchronizing pcsd certificates on nodes server01, server02...
server02: Success
server01: Success
Restarting pcsd on the nodes in order to reload the certificates...
server02: Success
server01: Success

# pcs cluster stop --all && pcs cluster start --all

# systemctl enable pacemaker
# systemctl enable corosync

# pcs status nodes
Pacemaker Nodes:
 Online: server01 server02
 Standby: 
 Maintenance:
 Offline:
Pacemaker Remote Nodes:
 Online:
 Standby:
 Maintenance:
 Offline:
# corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.20.1) 
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.20.2) 
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
# pcs status corosync
Membership information
----------------------
    Nodeid      Votes Name
         1          1 server01 (local)
         2          1 server02

# pcs cluster stop --all
server01: Stopping Cluster (pacemaker)...
server02: Stopping Cluster (pacemaker)...
server01: Stopping Cluster (corosync)...
　　　　　：
（暫く時間がかかった）
　　　　　：
server02: Stopping Cluster (corosync)...

# pcs cluster start --all
server01: Starting Cluster...
server02: Starting Cluster...
# pcs status nodes
（問題ないことを確認）
# pcs status
（問題ないことを確認）

# pcs property set stonith-enabled=false

# pcs property set no-quorum-policy=ignore

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: mycluster
 dc-version: 1.1.13-3.fc22-44eb2dd
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

# pcs cluster standby server02
# pcs status nodes
Pacemaker Nodes:
 Online: server01
 Standby: server02
 Maintenance:
 Offline:
Pacemaker Remote Nodes:
 Online:
 Standby:
 Maintenance:
 Offline:

# pcs cluster unstandby server02
# pcs status nodes
Pacemaker Nodes:
 Online: server01 server02
 Standby:
 Maintenance:
 Offline:
Pacemaker Remote Nodes:
 Online:
 Standby:
 Maintenance:
 Offline:

STONITHを無効
# pcs property set stonith-enabled=false
スプリットブレインが発生してもクォーラムが特別な動作を行わないように設定
# pcs property set no-quorum-policy=ignore
仮想IP
# pcs resource create dbvip ocf:heartbeat:IPaddr2 ip=192.168.20.10000 cidr_netmask=24 nic="eth1" op monitor interval=30s

仮想IP
# pcs resource create dbvip ocf:heartbeat:IPaddr2 ip=192.168.20.10000 cidr_netmask=24 op monitor interval=30s

# crm_mon -A
Last updated: Tue Jun 30 13:38:44 2015
Last change: Tue Jun 30 13:28:38 2015
Stack: corosync
Current DC: server01.bigbang.mydns.jp (2)	- partition with quorum
Version: 1.1.12-a14efad
2 Nodes configured
1 Resources configured
Online: [ server02.bigbang.mydns.jp server01.bigbang.mydns.jp ]
VIP     (ocf::heartbeat:IPaddr2):	Started server01.bigbang.mydns.jp
Node Attributes:
* Node server02.bigbang.mydns.jp:
* Node server01.bigbang.mydns.jp:

# pcs resource delete IPaddr2

# pcs cluster stop server02
server02: Stopping Cluster (pacemaker)...
server02: Stopping Cluster (corosync)...
# pcs status
Error: cluster is not currently running on this node

# pcs config
Cluster Name: 
Corosync Nodes:
 192.168.20.1 192.168.20.2 192.168.20.3 
Pacemaker Nodes:
 server02.bigbang.mydns.jp server01.bigbang.mydns.jp 
Resources: 
 Resource: VIP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=192.168.20.100 cidr_netmask=24 
  Operations: start interval=0s timeout=20s (VIP-start-timeout-20s)
              stop interval=0s timeout=20s (VIP-stop-timeout-20s)
              monitor interval=10s (VIP-monitor-interval-10s)
Stonith Devices: 
Fencing Levels: 
Location Constraints:
Ordering Constraints:
Colocation Constraints:
Cluster Properties:
 cluster-infrastructure: corosync
 dc-version: 1.1.12-a14efad
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

# pcs cluster cib output.cib

# pcs -f output.cib >command<

# pcs cluster cib-push output.cib

# pcs resource standards

# pcs resource providers

# pcs resource agents ocf:heartbeat

# pcs resource describe ocf:heartbeat:IPaddr2

# crm ra info ocf:heartbeat:IPaddr2

# pcs cluster standby server-name01(サーバ名)
# pcs cluster standby server-name02(サーバ名)

# crm configure erase

# pcs cluster unstandby server-name01(サーバ名)
# pcs cluster unstandby server-name02(サーバ名)

# pcs resource status
# pcs resource show

# pcs constraint order
Ordering Constraints:
  promote ms-mysql then start grp-network
  start res-ping-clone then promote ms-mysql
  start res-chk-vip1 then start res-vip1

# ccs_config_validate
Relax-NG validity error : Extra element cman in interleave
tempfile:23: element cman: Relax-NG validity error : Element cluster failed to validate content
Configuration fails to validate
※上記はエラーが発生している。エラーがでなければOK

# crm_verify -LV

# crm_mon -fA

# pcs cluster report --from="2016-01-07 10:30:00" --to="2016-01-07 11:30:00"

# corosync-cfgtool -s

# ip addr show

# pcs resource failcount show <リソースID>
# pcs resource failcount reset <リソースID>

# pcs status
# pcs resource cleanup <リソースID>

# pcs constraint location rm cli-standby-<RSC_NAME> [NODE]

# pcs cluster standby server1(ホスト名)

# pcs cluster unstandby server1(ホスト名)

# pcs resource move <resource id> [destination node name] ※2台でのクラスタ構成の場合、[destination node name]は不要
# pcs resource clear <resource id>

# pcs resource failcount show res-mysql
No failcounts for res-mysql

# pcs resource failcount reset res-mysql
# pcs resource cleanup res-mysql

# pcs resource start mysql-clone

# pcs resource move ms-mysql
# pcs resource clear ms-mysql
# pcs resource move ms-mysql --master

# pcs cluster cib /tmp/cib.txt

# pcs cluster stop --all
# pcs cluster destroy

# pcs cluster start ホスト名
inu: Unable to connect to ホスト名, try setting higher timeout in \
    --request-timeout option (Failed connect to ホスト名:2224; Connection refused)
inu: Not starting cluster - node is unreachable
Error: unable to start all nodes

# firewall-cmd --list-port

[root@centos7-1 ~]# pcs cluster node add centos7-3
Error: cluster is configured for RRP, you have to specify ring 1 address for the node

[root@centos7-1 ~]# pcs cluster start centos7-3
centos7-3: Error connecting to centos7-3 - (HTTP error: 400)
Error: unable to start all nodes
centos7-3: Error connecting to centos7-3 - (HTTP error: 400)

[root@centos7-1 ~]# pcs cluster setup --start --name bigbang-cluster1 centos7-1,centos7-1c centos7-2,centos7-2c centos7-3,centos7-3c
Error: centos7-1: node is already in a cluster
Error: centos7-2: node is already in a cluster
Error: nodes availability check failed, use --force to override. WARNING: This will destroy existing cluster on the nodes. \
       You should remove the nodes from their clusters instead to keep the clusters working properly.

[root@centos7-1 ~]# pcs cluster setup --start --name bigbang-cluster1 centos7-1,centos7-1c \
                                             centos7-2,centos7-2c centos7-3,centos7-3c --force
Destroying cluster on nodes: centos7-1, centos7-2, centos7-3...
centos7-3: Stopping Cluster (pacemaker)...
centos7-2: Stopping Cluster (pacemaker)...
centos7-1: Stopping Cluster (pacemaker)...
centos7-1: Successfully destroyed cluster
centos7-2: Successfully destroyed cluster
centos7-3: Successfully destroyed cluster

Sending 'pacemaker_remote authkey' to 'centos7-1', 'centos7-2', 'centos7-3'
centos7-1: successful distribution of the file 'pacemaker_remote authkey'
centos7-3: successful distribution of the file 'pacemaker_remote authkey'
centos7-2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
centos7-1: Succeeded
centos7-2: Succeeded
centos7-3: Succeeded

Starting cluster on nodes: centos7-1, centos7-2, centos7-3...
centos7-1: Starting Cluster (corosync)...
centos7-2: Starting Cluster (corosync)...
centos7-3: Starting Cluster (corosync)...
centos7-1: Starting Cluster (pacemaker)...
centos7-2: Starting Cluster (pacemaker)...
centos7-3: Starting Cluster (pacemaker)...

Synchronizing pcsd certificates on nodes centos7-1, centos7-2, centos7-3...
centos7-1: Success
centos7-3: Success
centos7-2: Success
Restarting pcsd on the nodes in order to reload the certificates...
centos7-1: Success
centos7-3: Success
centos7-2: Success
[root@centos7-1 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
	id	= 10.0.0.25
	status	= ring 0 active with no faults
RING ID 1
	id	= 192.168.0.11
	status	= ring 1 active with no faults

[root@centos7-1 ~]# pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang-cluster1
 dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
 have-watchdog: false

[root@サーバB ~]# pcs resource move rg01  ・・・①
Warning: Creating location constraint cli-ban-rg01-on-＜サーバB＞ with a score of \
-INFINITY for resource rg01 on node ＜サーバB＞.
This will prevent rg01 from running on サーバB until the constraint is removed. \
This will be the case even if ＜サーバB＞ is the last node in the cluster.

※結論を先に述べると、①の直後に
# pcs resource clear rg01
を実行するべきでした。

[root@サーバB ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started サーバA

[root@サーバB ~]# pcs resource move rg01  ・・・②
Warning: Creating location constraint cli-ban-rg01-on-＜サーバB＞ with a score of \
-INFINITY for resource rg01 on node ＜サーバB＞.
This will prevent rg01 from running on サーバB until the constraint is removed. \
This will be the case even if ＜サーバB＞ is the last node in the cluster.

※結論を先に述べると、②の実施前（①の実施直後）に
# pcs resource clear rg01
を実行するべきでした。

[root@サーバB ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Stopped

[root@サーバA ~]# pcs config
Cluster Name: bigbang
Corosync Nodes:
 サーバA サーバB
Pacemaker Nodes:
 サーバA サーバB

Resources:
 Group: rg01
  Resource: VirtualIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: cidr_netmask=24 ip=192.168.0.10 nic=eth1
   Operations: monitor interval=30s (VirtualIP-monitor-interval-30s)
               start interval=0s timeout=20s (VirtualIP-start-interval-0s)
               stop interval=0s timeout=20s (VirtualIP-stop-interval-0s)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: VirtualIP
    Enabled on: サーバB (score:INFINITY) (role: Started) (id:cli-prefer-VirtualIP)
  Resource: rg01
    Enabled on: サーバB (score:INFINITY) (role: Started) (id:cli-prefer-rg01)
    Disabled on: サーバB (score:-INFINITY) (role: Started) (id:cli-ban-rg01-on-サーバB)
    Disabled on: サーバA (score:-INFINITY) (role: Started) (id:cli-ban-rg01-on-サーバA)
Ordering Constraints:
Colocation Constraints:
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
 No defaults set
Operations Defaults:
 No defaults set

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: bigbang
 dc-version: 1.1.19-8.el7_6.4-c3c624ea3d
 have-watchdog: false
 no-quorum-policy: ignore
 stonith-enabled: false

Quorum:
  Options:

[root@サーバA ~]# pcs resource clear rg01
[root@サーバA ~]# pcs resource show
 Resource Group: rg01
     VirtualIP	(ocf::heartbeat:IPaddr2):	Started サーバB

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar  8 17:10:39 2021
  * Last change:  Mon Mar  8 17:10:08 2021 by root via crm_resource on centos8-str1
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
  * Clone Set: DRBD-clone [DRBD] (promotable):
    * Masters: [ centos8-str1 ]
    * Slaves: [ centos8-str2 ]

Failed Resource Actions:
  * FS_DRBD0_start_0 on centos8-str2 'error' (1): call=108, status='complete', \
    exitreason='Couldn't mount device [/dev/drbd0] as /mnt2/drdb0', \
    last-rc-change='2021-03-08 17:07:34 +09:00', queued=0ms, exec=2632ms

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 1 reply from the controller
... got reply (done)

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Mon Mar  8 17:11:58 2021
  * Last change:  Mon Mar  8 17:11:51 2021 by hacluster via crmd on centos8-str2
  * 2 nodes configured
  * 6 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Resource Group: rg01:
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
  * Clone Set: DRBD-clone [DRBD] (promotable):
    * Masters: [ centos8-str1 ]
    * Slaves: [ centos8-str2 ]

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs config show
（省略）
Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=1
（省略）

[root@centos8-str1 ~]# pcs resource defaults update migration-threshold=3
Warning: Defaults do not apply to resources which override them with their own defined values
[root@centos8-str1 ~]# pcs config show
（省略）
Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    migration-threshold=3
（省略）

[root@centos8-str1 ~]# pcs status
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 11:37:56 2021
  * Last change:  Fri Aug 27 11:28:03 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 centos8-str2 ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * Masters: [ centos8-str1 ]
    * Slaves: [ centos8-str2 ]
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`

[root@centos8-str1 ~]# pcs status
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:04:12 2021
  * Last change:  Fri Aug 27 13:50:20 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * httpd: migration-threshold=3 fail-count=1. last-failure='Fri Aug 27 14:03:58 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str1 'not running' (7): call=184, status='complete', exitreason='', last-rc-change='2021-08-27 14:03:58 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`

[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:06:20 2021
  * Last change:  Fri Aug 27 13:50:20 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * httpd: migration-threshold=3 fail-count=2 last-failure='Fri Aug 27 14:06:05 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str1 'not running' (7): call=202, status='complete', exitreason='', last-rc-change='2021-08-27 14:06:05 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`

[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:08:31 2021
  * Last change:  Fri Aug 27 13:50:20 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * httpd: migration-threshold=3 fail-count=3 last-failure='Fri Aug 27 14:08:12 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str1 'not running' (7): call=218, status='complete', exitreason='', last-rc-change='2021-08-27 14:08:12 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str1 ~]# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 2 replies from the controller
... got reply
... got reply (done)

※1回目の障害（Apache 1回目の障害）
[root@centos8-str2 ~]# kill -kill `pgrep -f httpd`

[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:14:45 2021
  * Last change:  Fri Aug 27 14:12:41 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * httpd: migration-threshold=3 fail-count=1 last-failure='Fri Aug 27 14:14:27 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str2 'not running' (7): call=197, status='complete', exitreason='', last-rc-change='2021-08-27 14:14:27 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

※2回目の障害（Zabbix Server 1回目の障害）
[root@centos8-str2 ~]# kill -kill `pgrep -f zabbix_server`

[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:16:45 2021
  * Last change:  Fri Aug 27 14:12:41 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * httpd: migration-threshold=3 fail-count=1 last-failure='Fri Aug 27 14:14:27 2021'
    * zabbix-server: migration-threshold=3 fail-count=1 last-failure='Fri Aug 27 14:16:35 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str2 'not running' (7): call=197, status='complete', exitreason='', last-rc-change='2021-08-27 14:14:27 +09:00', queued=0ms, exec=0ms
  * zabbix-server_monitor_60000 on centos8-str2 'not running' (7): call=215, status='complete', exitreason='', last-rc-change='2021-08-27 14:16:35 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

※3回目の障害（Apache 2回目の障害）
[root@centos8-str2 ~]# kill -kill `pgrep -f httpd`

[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:18:43 2021
  * Last change:  Fri Aug 27 14:12:41 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * MariaDB	(systemd:mariadb):	 Started centos8-str2
    * httpd	(systemd:httpd):	 Started centos8-str2
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str2
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str2

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * httpd: migration-threshold=3 fail-count=2 last-failure='Fri Aug 27 14:18:33 2021'
    * zabbix-server: migration-threshold=3 fail-count=1 last-failure='Fri Aug 27 14:16:35 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str2 'not running' (7): call=213, status='complete', exitreason='', last-rc-change='2021-08-27 14:18:33 +09:00', queued=0ms, exec=0ms
  * zabbix-server_monitor_60000 on centos8-str2 'not running' (7): call=215, status='complete', exitreason='', last-rc-change='2021-08-27 14:16:35 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled

[root@centos8-str2 ~]# kill -kill `pgrep -f httpd`

[[root@centos8-str2 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str1 (1) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Fri Aug 27 14:20:15 2021
  * Last change:  Fri Aug 27 14:12:41 2021 by hacluster via crmd on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str2 (2):
    * httpd: migration-threshold=3 fail-count=3 last-failure='Fri Aug 27 14:19:40 2021'
    * zabbix-server: migration-threshold=3 fail-count=1 last-failure='Fri Aug 27 14:16:35 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str2 'not running' (7): call=241, status='complete', exitreason='', last-rc-change='2021-08-27 14:19:40 +09:00', queued=0ms, exec=0ms
  * zabbix-server_monitor_60000 on centos8-str2 'not running' (7): call=215, status='complete', exitreason='', last-rc-change='2021-08-27 14:16:35 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@centos8-str2 ~]# pcs resource cleanup
Cleaned up all resources on all nodes
Waiting for 2 replies from the controller
... got reply
... got reply (done)

[root@centos8-str1 ~]# pcs resource defaults update failure-timeout=60s
Warning: Defaults do not apply to resources which override them with their own defined values

[root@centos8-str1 ~]# pcs config show
（省略）
Resources Defaults:
  Meta Attrs: rsc_defaults-meta_attributes
    failure-timeout=60s
    migration-threshold=3
（省略）

[root@centos8-str1 ~]# kill -kill `pgrep -f httpd`

※障害を検知し、fail-countが「1」になっているかの確認
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (2) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Sat Aug 28 00:14:43 2021
  * Last change:  Sat Aug 28 00:13:12 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: centos8-str1 (1):
    * httpd: migration-threshold=3 fail-count=1 last-failure='Sat Aug 28 00:14:29 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on centos8-str1 'not running' (7): call=40, status='complete', exitreason='', last-rc-change='2021-08-28 00:14:29 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

※60秒経過後にfail-countがクリアしているかの確認
[root@centos8-str1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: centos8-str2 (2) (version 2.0.5-8.el8-ba59be7122) - partition with quorum
  * Last updated: Sat Aug 28 00:15:32 2021
  * Last change:  Sat Aug 28 00:13:12 2021 by root via cibadmin on centos8-str1
  * 2 nodes configured
  * 8 resource instances configured

Node List:
  * Online: [ centos8-str1 (1) centos8-str2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave centos8-str2
    * DRBD_r0	(ocf::linbit:drbd):	 Master centos8-str1
  * Resource Group: zabbix-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * MariaDB	(systemd:mariadb):	 Started centos8-str1
    * httpd	(systemd:httpd):	 Started centos8-str1
    * zabbix-server	(systemd:zabbix-server):	 Started centos8-str1
    * ShareDir	(ocf::heartbeat:Filesystem):	 Started centos8-str1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started centos8-str1

Node Attributes:
  * Node: centos8-str1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: centos8-str2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:

Tickets:

PCSD Status:
  centos8-str1: Online
  centos8-str2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@alma8-1 ~]# kill -kill `pgrep -f httpd`;date
2021年  9月  3日 金曜日 16:55:58 JST

[root@alma8-1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: alma8-2 (2) (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Fri Sep  3 16:55:53 2021
  * Last change:  Wed Sep  1 15:22:27 2021 by root via cibadmin on alma8-2
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ alma8-1 (1) alma8-2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master alma8-1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave alma8-2
  * Resource Group: cluster-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started alma8-1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started alma8-1
    * httpd	(systemd:httpd):	 Started alma8-1

Node Attributes:
  * Node: alma8-1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: alma8-2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: alma8-1 (1):
    * httpd: migration-threshold=3 fail-count=1 last-failure='Fri Sep  3 16:55:26 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on alma8-1 'not running' (7): call=74, status='complete', exitreason='', last-rc-change='2021-09-03 16:55:27 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  alma8-1: Online
  alma8-2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@alma8-1 ~]# kill -kill `pgrep -f httpd`;date
2021年  9月  3日 金曜日 16:55:02 JST

[root@alma8-1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: alma8-2 (2) (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Fri Sep  3 16:56:40 2021
  * Last change:  Wed Sep  1 15:22:27 2021 by root via cibadmin on alma8-2
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ alma8-1 (1) alma8-2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Master alma8-1
    * DRBD_r0	(ocf::linbit:drbd):	 Slave alma8-2
  * Resource Group: cluster-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started alma8-1
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started alma8-1
    * httpd	(systemd:httpd):	 Started alma8-1

Node Attributes:
  * Node: alma8-1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: alma8-2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: alma8-1 (1):
    * httpd: migration-threshold=3 fail-count=2 last-failure='Fri Sep  3 16:56:31 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on alma8-1 'not running' (7): call=78, status='complete', exitreason='', last-rc-change='2021-09-03 16:56:31 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  alma8-1: Online
  alma8-2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@alma8-1 ~]# kill -kill `pgrep -f httpd`;date
2021年  9月  3日 金曜日 16:56:54 JST

[root@alma8-1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: alma8-2 (2) (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Fri Sep  3 16:57:47 2021
  * Last change:  Wed Sep  1 15:22:27 2021 by root via cibadmin on alma8-2
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ alma8-1 (1) alma8-2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave alma8-1
    * DRBD_r0	(ocf::linbit:drbd):	 Master alma8-2
  * Resource Group: cluster-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started alma8-2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started alma8-2
    * httpd	(systemd:httpd):	 Started alma8-2

Node Attributes:
  * Node: alma8-1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: alma8-2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:
  * Node: alma8-1 (1):
    * httpd: migration-threshold=3 fail-count=3 last-failure='Fri Sep  3 16:57:35 2021'

Failed Resource Actions:
  * httpd_monitor_60000 on alma8-1 'not running' (7): call=82, status='complete', exitreason='', last-rc-change='2021-09-03 16:57:35 +09:00', queued=0ms, exec=0ms

Tickets:

PCSD Status:
  alma8-1: Online
  alma8-2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled

[root@alma8-1 ~]# pcs status --full
Cluster name: bigbang
Cluster Summary:
  * Stack: corosync
  * Current DC: alma8-2 (2) (version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum
  * Last updated: Fri Sep  3 17:03:32 2021
  * Last change:  Wed Sep  1 15:22:27 2021 by root via cibadmin on alma8-2
  * 2 nodes configured
  * 5 resource instances configured

Node List:
  * Online: [ alma8-1 (1) alma8-2 (2) ]

Full List of Resources:
  * Clone Set: DRBD_r0-clone [DRBD_r0] (promotable):
    * DRBD_r0	(ocf::linbit:drbd):	 Slave alma8-1
    * DRBD_r0	(ocf::linbit:drbd):	 Master alma8-2
  * Resource Group: cluster-group:
    * FS_DRBD0	(ocf::heartbeat:Filesystem):	 Started alma8-2
    * VirtualIP	(ocf::heartbeat:IPaddr2):	 Started alma8-2
    * httpd	(systemd:httpd):	 Started alma8-2

Node Attributes:
  * Node: alma8-1 (1):
    * master-DRBD_r0                  	: 10000     
  * Node: alma8-2 (2):
    * master-DRBD_r0                  	: 10000     

Migration Summary:

Tickets:

PCSD Status:
  alma8-1: Online
  alma8-2: Online

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled