sap_ha_pacemaker_cluster: Support for VMWare and for ERS2

[rob0d]

Hi all,

Has anyone tried using this to configure ASCS/ERS cluster on VMWare?
I guess the main difference will be the fencing agent, but if anyone did this already it would be really helpful if you can share your experience.

Also the samples and comments in code seem to imply that I can install a combined ASCS/ERS cluster rather than a separate ERS2 cluster. Again, anybody has any view/experience? I need to install two separate clusters. Two node ASCS and two node ERS2.

Any pointers/advice will be greatly appreciated.

[sean-freeman]

@rob0d There is no vars or platform handlers added in the sap_ha_pacemaker_cluster Ansible Role to handle VMware, at this time. The Ansible Role combined with a VMware VM target has been tested by a customer, by using custom variables, but not added into upstream.

The main prohibitor to the aforementioned effort, is the lack of end-to-end flow because of VMware problems that require a deep VMware expert to address. I've got plenty of VMware SDDCs I can access, just cannot get a reliable provision so it breaks automated testing. See:

• notes regarding cloud-init for VMware in community.sap_infrastructure/roles/sap_vm_provision/PLATFORM_GUIDANCE.md
• related GH Issue in vmware.vmware_rest Ansible Collection

The latter question, I will have to defer to @ja9fuchs and @marcelmamula . Would suggest 1 topic per GH Issue/Discussion in future, for ease of discussion

[rob0d]

@sean-freeman

Is this something that you'd be able to share (at least partly) so I don't have to re-invent the wheel?

The Ansible Role combined with a VMware VM target has been tested by a customer, by using custom variables, but not added into upstream.

It seems that these two are more related to VM provisioning, rather than SAP HA install on VMWare?

The main prohibitor to the aforementioned effort, is the lack of end-to-end flow because of VMware problems that require a deep VMware expert to address. I've got plenty of VMware SDDCs I can access, just cannot get a reliable provision so it breaks automated testing. See:

• notes regarding cloud-init for VMware in community.sap_infrastructure/roles/sap_vm_provision/PLATFORM_GUIDANCE.md
• related GH Issue in vmware.vmware_rest Ansible Collection

If I manage the SAP HA part working, do you want me to do a PR to see if we can integrate it back to upstream?

[sean-freeman]

Re. reinvent wheel =
Not for a few weeks, other priorities are calling.

Re. Do-it-faster =
To implement a new default is not that much effort thankfully, just patience required getting the STONITH with fence_vmware_rest Fence Agent parameters correct and thereafter copy/paste/tweak of the IPaddr2 Resource Agent configuration seen elsewhere in the Ansible Role. Thankfully our friends at OVH (thanks @remrozk) already have a very concise version you can follow > https://help.ovhcloud.com/csm/en-gb-sap-hana-configure-suse-cluster?id=kb_article_view&sysparm_article=KB0059460#stonith-resource

Re. provisioning VMware VM =
These issues with cloud-init are provisioning blockers, which is what results in "lack of end-to-end flow" so we cannot perform interative testing during HA development and thereafter regression testing.

[sean-freeman]

@rob0d If you do it, note that you will almost certainly need to use the fence_vmware_rest --filter with the names of each node in the HA cluster.

Unfortunately all of the documentation in the fence_vmware_rest code has been moved and is harder to find, this appears to be the documentation about the API endpoint and using the filters:

https://developer.broadcom.com/xapis/vsphere-automation-api/8.0.3/vcenter/api/vcenter/vm/get/

[marcelmamula]

@rob0d Installation of ASCS/ERS cluster is done following SAP documentation and our certifications. ASCS and ERS is meant to run in one cluster, not spread across independent clusters.

You could potentially:

1. Install ASCS / ERS using sap_swpm role across all those 4 nodes.
2. Setup cluster using directly https://github.com/linux-system-roles/ha_cluster , which would mean you need to prepare all inputs that are now taken care of by sap_ha_pacemaker_cluster role. It means run it once for 2 ASCS nodes, second time for 2 ERS nodes.
3. Execute all post steps manually.

Disclaimer: We do not have any documentation or support for such cluster scenario.

[rob0d]

Hi @marcelmamula,
Combined ASCS/ERS cluster is based on the previous (ENSA1) enqueue server which needed to co-locate ASCS with ERS after a failure. Looking at the current RHEL guides (sorry not seen the SuSe guides), they are still using this setup, but don't create the colocation constraint.
However, it seems that SAP are moving into direction of two separate clusters as per this or 3+ node cluster as per this. I'm not sure how much are they going to push this going forward or if the standard combined ASCS/ERS cluster will remain the default option.

[marcelmamula]

Thank you for providing document link @rob0d Documentation you provided looks like SWPM specific and it lacks any HA Cluster information, including fact that Failover (switchover) groups are not concept in Pacemaker.

SAP ENSA2 clusters are certified only for 2 nodes by default and 3 nodes for special cases described in documentation https://documentation.suse.com/sbp/sap-15/html/SAP-S4HA10-setupguide-simplemount-sle15/index.html#multicluster
HA scenario you shared is not certified by SAP partners, so I cannot comment on validity of that document.

[rob0d]

Hi @marcelmamula,
Yes the links I posted are fairly generic, rather than linked to a specific HA solution. They just show that SAP support multi-node cluster or two two-node clusters. However, I haven't realised that the HA solution certification is strict and doesn't necessarily cover different variant of the cluster setup.
It seems that Redhat are supporting multi-node cluster now.
Thanks for providing the link for Suse documentation, I found a few useful tip there.
It seems that I need to re-think my setup and go back to one two-node cluster.
Thanks for getting back to me.

[ja9fuchs]

@rob0d Coming back to this older question about a combined ASCS/ERS cluster... I actually thought this was and remains the "standard". :D

Yes, with ENSA2 the ASCS instance does not have to fail over to the same node that is running ERS in order to read the replicated data. So now you could have e.g. 3 nodes or more and due to the constraint that tries to keep the instances separate, ASCS would fail over to a suitable node that is not running ERS. The advantage of this setup is that after ASCS has fully started and recovered from ERS, there is no further movement in the cluster. In a 2-node setup the ERS instance would move to the other node (once available) to avoid being with ASCS. This is an extra activity that leaves a gap for missing replication, if the ASCS failed before ERS was back up and in sync.

The same applies to a "cost-optimized" setup, which the role can in fact configure.
I happened to have tested this on a 2-node cluster just recently (and fixed a few minor issues). Additional nodes is a separate story that I will look into later.

The host type can be defined as a list and the combination is evaluated, the following works - an any order:

sap_ha_pacemaker_cluster_host_type:
  - nwas_abap_ascs_ers
  - hana_scaleup_perf

Other than that, all definitions must be provided for HANA, ASCS and ERS, as usual.

(I did realize that the "perf" in the hana type is not quite right in that combination now. But this is only a cosmetical itch.)

[rob0d]

Hi @ja9fuchs,
It seems that combined ASCS/ERS2 is still the standard and what the RHEL/SUSE have certified, but SAP doesn't require this anymore (as of ERS2) as per this

The statement is:

As you know ERS2 allows more flexibility as SAP doesn't really care where the services are running as long as they have virtual IP and underlying clustering SW supports the APIs. However, based on yours and Marcel's comments I've changed the deployment to a combined ASCS/ERS and then added WebDisp (as per PR #974) as a new cluster host type.

For 3-node installation a HA prep run of SWPM will be required (or additional ERS installation and partial removal of local disks) to make the 3rd node ready to host the services.