Solace PubSub HA with DMR and Replication

Table of Contents

Overview

Dev/test environment for Solace event broker high‑availability (HA) with Dynamic Message Routing (DMR) and disaster recovery (DR) using replication. Includes publisher and receiver client use cases. Replication and failover are per‑Message VPN. Only VPNs with replication enabled participate in DR and require a switchover. Non‑replicated VPNs can exist on both sites; clients may connect to either, and DMR routes messages between them without state changes.

Reference: Solace DR Replication Documentation

Hardware and OS Requirements

Component Version/Spec
CPU AMD64
OS Debian GNU/Linux 13 (trixie)
Docker 29.5.3
OpenTofu v1.12.1
stm/stable 0.0.83
pip solace-pubsubplus 1.11.0
Python Samples solace-samples-python

Note: Solace ARM64 docker image exists but fails hardware validation step

Architecture

Static IP assignment on the 172.21.0.0/24 network, ensuring deterministic connectivity between brokers and from clients.

graph TB %% Styles classDef primary fill:#ff9999,stroke:#cc0000,stroke-width:2px,color:#fff classDef backup fill:#99ff99,stroke:#00cc00,stroke-width:2px,color:#000 classDef monitor fill:#9999ff,stroke:#0000cc,stroke-width:2px,color:#fff classDef lb fill:#ffff99,stroke:#cccc00,stroke-width:2px classDef clients fill:#ddd,stroke:#888,stroke-width:2px %% Site 1 (Active) subgraph SITE1 ["<b>Site 1 HA1 (Active)</b>"] direction LR PRIMARY1("primary1<br/>172.21.0.10") BACKUP1("backup1<br/>172.21.0.11") MONITOR1("monitoring1<br/>172.21.0.12") LB1("lb1<br/>172.21.0.20") end %% Site 2 (Standby) subgraph SITE2 ["<b>Site 2 HA2 (Standby)</b>"] direction LR PRIMARY2("primary2<br/>172.21.0.30") BACKUP2("backup2<br/>172.21.0.31") MONITOR2("monitoring2<br/>172.21.0.32") LB2("lb2<br/>172.21.0.40") end %% Clients CLIENTS("Clients") %% Intra‑cluster HA connections PRIMARY1 <-->|Sync| BACKUP1 PRIMARY1 <-->|Check| MONITOR1 BACKUP1 <-->|Check| MONITOR1 PRIMARY2 <-->|Sync| BACKUP2 PRIMARY2 <-->|Check| MONITOR2 BACKUP2 <-->|Check| MONITOR2 %% Cross‑site DMR (thin) and Replication (bold) PRIMARY1 <---->|DMR| PRIMARY2 PRIMARY1 ====|Replication| PRIMARY2 %% Health checks LB1 -.->|Health Check| PRIMARY1 LB1 -.->|Health Check| BACKUP1 LB2 -.->|Health Check| PRIMARY2 LB2 -.->|Health Check| BACKUP2 %% Client messaging traffic CLIENTS <-->|Messaging| LB1 CLIENTS <-->|Messaging| LB2 LB1 -->|Messaging| PRIMARY1 LB2 -->|Messaging| PRIMARY2 %% Apply styles class PRIMARY1,PRIMARY2 primary class BACKUP1,BACKUP2 backup class MONITOR1,MONITOR2 monitor class LB1,LB2 lb class CLIENTS clients

HA Group 1

Two separate HA groups, each with a Primary, Backup, Monitor, and an HAProxy load balancer.

Hostname Role IP Address
primary1 Primary 172.21.0.10
backup1 Backup 172.21.0.11
monitoring1 Monitor 172.21.0.12
lb1 Load Balancer 172.21.0.20

HA Group 2

Hostname Role IP Address
primary2 Primary 172.21.0.30
backup2 Backup 172.21.0.31
monitoring2 Monitor 172.21.0.32
lb2 Load Balancer 172.21.0.40

Docker compose

cd solace-pubsub-ha1 && docker compose up -d; docker compose logs -f
cd solace-pubsub-ha2 && docker compose up -d; docker compose logs -f

Docker containers running

docker container ls
IMAGE                                       COMMAND                PORTS                                                                                          NAMES
haproxy:latest                              "/bin/bash -c 'perl …" 0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp, 0.0.0.0:52555->55555/tcp, [::]:52555->55555/tcp   lb2
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:1222->2222/tcp, [::]:1222->2222/tcp                                                    backup2
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:1122->2222/tcp, [::]:1122->2222/tcp                                                    primary2
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:1322->2222/tcp, [::]:1322->2222/tcp                                                    monitoring2
haproxy:latest                              "/bin/bash -c 'perl …" 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp, 0.0.0.0:51555->55555/tcp, [::]:51555->55555/tcp   lb1
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:3322->2222/tcp, [::]:3322->2222/tcp                                                    monitoring1
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:3222->2222/tcp, [::]:3222->2222/tcp                                                    backup1
solace/solace-pubsub-standard:10.25.0.276   "/usr/sbin/boot.sh"    0.0.0.0:3122->2222/tcp, [::]:3122->2222/tcp                                                    primary1

Ports

Service ports available on both HA group from docker host:

'haproxy_stats',     1936
'semp_in',           8080
'semp_tls_in',       1943
'smf_in',            55555
'smf_compressed_in', 55003
'smf_tls_in',        55443
'web_in',            8008
'web_tls_in',        1443
'mqtt_in',           1883
'mqtt_tls_in',       8883
'mqtt_web_in',       8000
'mqtt_web_tls_in',   8443
'amqp_in',           5672
'amqp_tls_in',       5671
'rest_in',           9000
'rest_tls_in',       9443

Redundancy (within each HA group) confirmed via show redundancy group – all nodes Online.

docker exec -it primary1 /usr/sw/loads/currentload/bin/cli -A

primary1> show redundancy group 
Node Router-Name   Node Type       Address           Status
-----------------  --------------  ----------------  ---------
backup1            Message-Router  backup1           Online
monitoring1        Monitor         monitoring1       Online
primary1*          Message-Router  primary1          Online


primary2> show redundancy group 
Node Router-Name   Node Type       Address           Status
-----------------  --------------  ----------------  ---------
backup2            Message-Router  backup2           Online
monitoring2        Monitor         monitoring2       Online
primary2*          Message-Router  primary2          Online

Initial Setup

TLS Certificates Structure

TLS certificates self‑signed CA and per‑node combined PEM files, mounted into the containers. TLS is used for SEMP (via HAProxy), for SMF messaging (tcps://), and for Solace SSL links.

certs/ca/ca.crt
certs/tls/backup1-combined.pem
certs/tls/backup2-combined.pem
certs/tls/monitoring1-combined.pem
certs/tls/monitoring2-combined.pem
certs/tls/primary1-combined.pem
certs/tls/primary2-combined.pem
certs/tls/testclient-combined.pem

The Common Name (CN) in each certificate must match the broker hostname for server certs, and the client username for client certs (for certificate mapping).

Create API User

On both primary1 and primary2

enable
configure
create username tofudo password TôFûDô cli global-access-level admin

Configure TLS Certificates

On all Solace containers

authentication
  create client-certificate-authority ifri-akl-ca
  certificate file ifri-akl-ca.pem
ssl
  server-certificate <container>-combined.pem

Enable Replication

On primary1

enable
configure
replication mate virtual-router-name v:primary2
replication mate connect-via 172.21.0.30:55443 ssl
replication mate connect-via 172.21.0.31:55443 ssl
replication mate connect-via 172.21.0.30:55003 compressed
replication mate connect-via 172.21.0.31:55003 compressed
replication mate connect-via 172.21.0.30:55555
replication mate connect-via 172.21.0.31:55555
replication config-sync bridge authentication auth-scheme Basic
replication config-sync bridge authentication pre-shared-key key <YOUR-PSK-HERE>
no replication config-sync bridge shutdown

On primary2

enable
configure
replication mate virtual-router-name v:primary1
replication mate connect-via 172.21.0.10:55443 ssl
replication mate connect-via 172.21.0.11:55443 ssl
replication mate connect-via 172.21.0.10:55003 compressed
replication mate connect-via 172.21.0.11:55003 compressed
replication mate connect-via 172.21.0.10:55555
replication mate connect-via 172.21.0.11:55555
replication config-sync bridge authentication auth-scheme Basic
replication config-sync bridge authentication pre-shared-key key <YOUR-PSK-HERE>
no replication config-sync bridge shutdown

Check status

Both show replication outputs show the mate configured correctly and ConfigSync Bridge State: up. This means any VPN details, queue, or topic subscription created on primary1 will be replicated to primary2

primary1> show replication

Replication Interface:             
Replication Mate:                  v:primary2
    Plain Text:                    172.21.0.30:55555
                                   172.21.0.31:55555
    Compressed:                    172.21.0.30:55003
                                   172.21.0.31:55003
    SSL:                           172.21.0.30:55443
                                   172.21.0.31:55443

ConfigSync:
  Bridge:                          
    Admin State:                   Enabled
    State:                         up
    Authentication:
      Pre-Shared Key:              Configured
      Insecure Upgrade Mode:       Disabled
    Compressed:                    No
    Message Spool:                 
      Window Size:                 1024
    Retry Delay:                   3
primary2> show replication

Replication Interface:             
Replication Mate:                  v:primary1
    Plain Text:                    172.21.0.10:55555
                                   172.21.0.11:55555
    Compressed:                    172.21.0.10:55003
                                   172.21.0.11:55003
    SSL:                           172.21.0.10:55443
                                   172.21.0.11:55443

ConfigSync:
  Bridge:                          
    Admin State:                   Enabled
    State:                         up
    Authentication:
      Pre-Shared Key:              Configured
      Insecure Upgrade Mode:       Disabled
    Compressed:                    No
    Message Spool:                 
      Window Size:                 65535
    Retry Delay:                   3

Access Points

All IPs and ports are reachable from docker host

Service HA 1 HA 2
SEMP Web :8080 :8081
SMF TCP :51555 :52555

OpenTofu

Use LB1 and LB2 to manage each HA group

terraform {
  required_providers {
    solacebroker = {
      source  = "SolaceProducts/solacebroker"
      version = "1.3.0"
    }
  }
}

provider "solacebroker" {
  alias    = "ha1"
  username = "tofudo"
  password = "TôFûDô"
  url      = "https://lb1.ifri.akl:1943"   # lb1.ifri.akl listed in SANs
}

provider "solacebroker" {
  alias    = "ha2"
  username = "tofudo"
  password = "TôFûDô"
  url      = "https://lb2.ifri.akl:1943"   # lb2.ifri.akl listed in SANs
}  

Resources

The OpenTofu configuration provisions two Message VPNs:

VPN Replication Use Case
test-infra-vpn ✅ Enabled Disaster recovery (DR) – messages replicated to standby site
test_team_orag ❌ Disabled DMR‑only – clients can connect to either site, DMR routes messages

OpenTofu created resources: VPNs, client profiles, ACLs, client usernames, DMR cluster links, and replicated topic subscription.

solacebroker_dmr_cluster.cluster_ha1
solacebroker_dmr_cluster.cluster_ha2
solacebroker_dmr_cluster_link.link_primary1_to_primary2
solacebroker_dmr_cluster_link.link_primary2_to_primary1
solacebroker_msg_vpn.test_infra_p1
solacebroker_msg_vpn.test_infra_p2
solacebroker_msg_vpn.test_team_orag_p1
solacebroker_msg_vpn.test_team_orag_p2
solacebroker_msg_vpn_acl_profile.test_infra_acl_p1
solacebroker_msg_vpn_acl_profile.test_infra_acl_p2
solacebroker_msg_vpn_acl_profile.test_team_orag_acl_p1
solacebroker_msg_vpn_acl_profile.test_team_orag_acl_p2
solacebroker_msg_vpn_client_profile.test_infra_profile01_p1
solacebroker_msg_vpn_client_profile.test_infra_profile01_p2
solacebroker_msg_vpn_client_profile.test_infra_profile02_p1
solacebroker_msg_vpn_client_profile.test_infra_profile02_p2
solacebroker_msg_vpn_client_profile.test_team_orag_profile01_p1
solacebroker_msg_vpn_client_profile.test_team_orag_profile01_p2
solacebroker_msg_vpn_client_profile.test_team_orag_profile02_p1
solacebroker_msg_vpn_client_profile.test_team_orag_profile02_p2
solacebroker_msg_vpn_client_username.infra_test_bridge_p1
solacebroker_msg_vpn_client_username.infra_test_bridge_p2
solacebroker_msg_vpn_client_username.infra_test_client_p1
solacebroker_msg_vpn_client_username.infra_test_client_p2
solacebroker_msg_vpn_client_username.test_team_orag_bridge_p1
solacebroker_msg_vpn_client_username.test_team_orag_bridge_p2
solacebroker_msg_vpn_client_username.test_team_orag_client_p1
solacebroker_msg_vpn_client_username.test_team_orag_client_p2
solacebroker_msg_vpn_replicated_topic.replicate_all_p1

DMR cluster status

primary1> show cluster mytfcluster link #ACTIVE channel

Cluster Name          : mytfcluster
Node Name             : primary1

Remote Node Name : #ACTIVE

Channel VPN                    Address               Oper Fail Reason / Uptime
------------------------------ --------------------- ---- ---------------------
#cluster                       127.55.55.55:55555    Up   0d 0h 15m 13s

primary1> show cluster mytfcluster link primary2 channel 

Cluster Name          : mytfcluster
Node Name             : primary1

Remote Node Name : primary2

Channel VPN                    Address               Oper Fail Reason / Uptime
------------------------------ --------------------- ---- ---------------------
default                        172.21.0.30:60140     Up   0d 0h 12m 9s
#cluster                       172.21.0.30:40884     Up   0d 0h 14m 42s

Note: Solace Terraform provider is generally available for production services contained within a Message VPN. Use of any resource not contained within a Message VPN is not supported in production.

Tests

stm cli

stm -h
Commands:
  send [options]        Execute a send command
  receive [options]     Execute a receive command
  request [options]     Execute a request command
  reply [options]       Execute a reply command
  config [options]      Manage command configurations
  manage [options]      Manage broker connection and resources
  feed                  Manage event feeds

Examples:

stm receive --topic solace/try/me --config test-infra-config
stm send  --topic solace/try/me -m "Hello World!" --config test-infra-config

Python

Publisher process:

python direct_publisher.py 
Messaging Service connected? True
Direct Publisher ready? True

Send a KeyboardInterrupt to stop publishing

Published message on topic : solace/samples/python/direct/pub/1 
Published message on topic : solace/samples/python/direct/pub/2 
Published message on topic : solace/samples/python/direct/pub/3 
Published message on topic : solace/samples/python/direct/pub/4 
Published message on topic : solace/samples/python/direct/pub/5 

Receiver process:

python direct_receiver.py 
Messaging Service connected? True
Direct Receiver is running? True
Subscribing to: ['solace/samples/python/>']

Message Payload String: this is the body of the msg + 1 


Message Topic: solace/samples/python/direct/pub/1 


Message dump: Destination:                            Topic 'solace/samples/python/direct/pub/1'
ApplicationMessageId:                   NEW 1
Priority:                               4
Class Of Service:                       COS_1
DeliveryMode:                           DIRECT
DMQ Eligible                            
User Property Map:                      
  Key 'application' (STRING) samples
  Key 'language' (STRING) Python
Binary Attachment String:               len=31
  74 68 69 73 20 69 73 20  74 68 65 20 62 6f 64 79      this is    the body
  20 6f 66 20 74 68 65 20  6d 73 67 20 2b 20 31          of the    msg + 1

TLS and cert auth

TLS test using client user testclient created by OpenTofu (testclient.crt/testclient.key)

export SOLACE_VPN=test-infra-vpn
export SOLACE_HOST=tcps://172.21.0.20:55443

python TLS_connection.py 

Message Payload String: {'solace.messaging.transport.host': 'tcps://172.21.0.20:55443', 'solace.messaging.service.vpn-name': 'test-infra-vpn'} 

Messaging Service connected? True
Direct Receiver is running? True
Subscribing to: ['solace/samples/python/dir/sub/>', 'solace/samples/python/dir/sub/v2/>', 'solace/samples/>']


Message Payload String: this is the body of the msg + 1 


Message Topic: solace/samples/python/direct/pub/1 


Message dump: Destination:                            Topic 'solace/samples/python/direct/pub/1'
ApplicationMessageId:                   NEW 1
Priority:                               4
Class Of Service:                       COS_1
DeliveryMode:                           DIRECT
DMQ Eligible                            
User Property Map:                      
  Key 'application' (STRING) samples
  Key 'language' (STRING) Python
Binary Attachment String:               len=31
  74 68 69 73 20 69 73 20  74 68 65 20 62 6f 64 79      this is    the body
  20 6f 66 20 74 68 65 20  6d 73 67 20 2b 20 31          of the    msg + 1
 
Message Payload String: this is the body of the msg + 2