#What it contains, and what it does not

The original document also covers administrative access to the lab, its credentials, and the commitments made to the tenant it runs against. That part is not here, and will not be: it is a map of how to get into an environment that really exists.

References to the production infrastructure the lab reproduces are also left out. The lab grew out of a real environment belonging to a real organisation, and its characteristics — domain, certificates, sizing — are not things to publish.

All internal names are substituted: the Active Directory domain is contoso.lab, addresses belong to generic ranges, people are called Mario Rossi. The substitutions are consistent across every section: if two pieces talk about the same object, they call it the same thing.

#The three levels

Sections carry a label saying who they are for. The filter at the top uses it.

#The origin: three things you could not study anywhere else

L1

The lab grew out of three concrete situations, all with the same shape: a question that could not be answered without taking a risk.

• A TLS/SMTP failure incident about which theories had been formed that nobody could test. They stayed theories because testing them meant touching everyone's mail.
• A local administrator password reset on a perimeter server, performed under pressure. A procedure nobody had ever rehearsed before having to run it.
• The need to test provisioning scripts before running them where every mistake has consequences.

In general: the absence of anywhere you were allowed to be wrong.

#The criterion: architectural fidelity

L1

The stated goal was not "a working Exchange". It was a replica as close as possible to the real architecture, because only then are the observed behaviours transferable.

The difference is concrete. An Exchange on a flat network works beautifully and does not reproduce the phenomenon where the server cannot see the real sender's address. If the incident under study is about exactly that, the lab is worthless.

Every subsequent choice follows from this criterion: wherever a simplification would have erased an observable phenomenon, the simplification was rejected.

#What is reproduced, and what is not

L2

Reproduced faithfully:

Not reproduced, each time for a reason:

#Fictional names are a safety mechanism

L1

Every name in the lab is invented and bears no relation to any real one. This is not modesty: it is a defence.

A command aimed at the lab that names contoso.lab cannot, through inattention, hit production. If the two environments shared names, the only thing separating a test Remove-Mailbox from a disaster would be the care of whoever types it — and care, at eight in the evening, is a weak defence.

The same principle drives the LAB- prefix on every machine name: in the hypervisor's list, the lab is recognisable at a glance.

#The four fences

L2

Protection rests on four independent mechanisms. Independent is the word that matters: if one gives way, the others hold.

1. Network. All machines sit on Linux bridges with no physical ports.
2. Memory and CPU. Fixed memory, explicit limits, reduced scheduling weight.
3. Storage. Thin provisioning with discard, and a threshold beyond which you stop.
4. Conventions. Dedicated pool, tag, reserved ID range, name prefix.

#Bridges with no physical ports

L2

A Linux bridge with no ports attached has no path at all to the host's network interfaces. It is isolated by construction, not by configuration — which is a substantial difference: a firewall rule can be switched off by mistake, a cable that does not exist cannot.

The only point of contact with the outside network is the virtual firewall's WAN interface, operating in NAT: traffic leaves masqueraded behind a single address, and no lab service is reachable from outside except through explicit port forwards.

Checking the isolation:

ip -br a | grep vmbr        # the lab bridges must have no IPv4
bridge link show | grep vmbr9   # no output = no port attached

#Memory and CPU: why 50 and not 512

L2

Every lab machine has fixed memory — no ballooning on the Exchange servers — and two explicit limits:

qm set <vmid> --cpulimit 2     # ceiling of 2 effective cores
qm set <vmid> --cpuunits 50    # half the default weight

The value 50 is deliberate. On cgroup v2 the default weight is 100: setting 50 means that under contention the scheduler favours the pre-existing machines.

There is no headroom to play with on memory: at full tilt the machines ask for more RAM than the host has. The lab is therefore run in power-on groups, never all at once.

#The datacenter rule

L1

No datacenter-level or node-level operation without explicit agreement. Host reboots, changes to the global networking, work on shared storage: these touch everyone, and in a shared environment the only defence is the social rule.

#The segments, and why /29

L2

Five segments behind a virtual firewall, each on a bridge with no physical ports.

The choice of /29 is not cosmetic. In production the balancer and the Exchange servers sit on different, routed networks, and that is the condition that makes SNAT mandatory.

A lab on a flat network would not reproduce the phenomenon. With routed /29s it does, and it is verifiable in the protocol logs.

#SNAT: why Exchange cannot see the real sender

L2

The balancer receives connections on a virtual address and forwards them to the Exchange servers replacing the source address with its own.

This is not a configuration preference: it is a necessity. The balancer cannot present itself with the client's address, because the return traffic would not come back through it — it would go straight to the client, which would not recognise the connection. The price is that Exchange loses sight of the real sender.

The full path of an outbound message:

Application client   (10.20.30.10)
  → balancer VIP     (10.20.20.38)
  → HAProxy w/ SNAT  (source becomes 10.20.20.34)
  → Exchange mailbox (10.20.20.10 / .11)
  → EdgeSync → perimeter (10.20.40.2 / .3)
  → Internet

The proof that the lab is faithful is one line of the protocol log, after generating traffic through the VIP:

2026-08-19T00:23:03.808Z,LAB-MBX02\Default Frontend LAB-MBX02,...,
10.20.20.11:25,10.20.20.34:37058,+,,

The recorded remote address is the balancer's, not the client's. The Exchange servers accept connections from a different, not directly connected /29: exactly the production condition.

#TTL as a diagnostic tool

L1

A detail worth more than many tools. In ping replies, the TTL value distinguishes paths:

Comparing TTLs is the fastest way to tell whether two machines talk directly or via routing. In a segmented environment this is the first question to ask and the last one people think of.

#pfSense: the three boxes people get wrong

L2

Three settings produce faults that look like they have other causes.

Block private networks and Block bogon networks on the WAN. Those filters make sense on an interface facing the real internet, where traffic from private or reserved addresses is anomalous by definition. When the "WAN" is actually a private LAN, administrative traffic arrives from private addresses or from ranges classified as bogon, and gets discarded before any rule. Port forwards end up configured and inert.

The DNS Hostname field in System → General Setup. It serves the resolver's TLS verification and must be left empty. Putting an IP address there — an easy mistake, since the field sits next to the address one — breaks resolution even though routing is correct.

Filter rule association on a port forward. It must stay on Add associated filter rule. With None, pfSense performs the NAT but blocks the traffic, and you spend half an hour looking for the cause elsewhere.

#Adding an interface: not while running

L3

Network interfaces do not hot-attach on FreeBSD. The correct sequence starts with a snapshot:

qm snapshot 1310 pre-change
qm set 1310 -net<N> virtio,bridge=vmbr9X,firewall=1
qm shutdown 1310 && qm start 1310

Then, in the interface: Interfaces → Assignments → Add on the new card, open it, Enable, description, static address, Save and Apply.

#The firewall you reboot, the witness that falls

L2

Every firewall reboot takes the DAG's File Share Witness offline, because the share lives on the domain controller, which sits beyond the firewall.

It does not come back on its own. This is where an apparently harmless piece of network maintenance leaves the cluster degraded without anyone noticing. The remedy is in Quorum and the witness.

#The forest, and why the internal domain is not enough

L2

A single domain controller, domain contoso.lab, Active Directory recycle bin enabled.

The recycle bin is not a detail: it lets you restore an accidentally deleted object with all its attributes. Without it, the deletion propagates to the cloud and recovery becomes laborious.

Get-ADOptionalFeature -Filter * | ft Name,EnabledScopes -Auto

Restoring a deleted object:

Get-ADObject -Filter 'isDeleted -eq $true' -IncludeDeletedObjects -Properties * |
  ft Name,SamAccountName,LastKnownParent,whenChanged -Auto

Get-ADObject -Filter 'SamAccountName -eq "<user>"' -IncludeDeletedObjects |
  Restore-ADObject

An additional UPN suffix on a verified public domain was added to the forest:

Get-ADForest | Set-ADForest -UPNSuffixes @{add="lab.impicciando.it"}

This is essential: contoso.lab is not routable on the internet, and Entra Connect refuses to synchronise non-routable UPNs. Cloud-bound users therefore carry a UPN on the public domain, while the samAccountName and Windows sign-in stay unchanged.

#Organisational units as a fence

L2

Nine organisational units reflecting a complete account lifecycle: operational staff, head office, support, external contractors, function mailboxes, decommissioned, disabled, awaiting deletion — and SYNC.

SYNC serves a different purpose from the rest: it is the synchronisation fence. Entra Connect is configured to consider that unit exclusively, so any object outside it does not, as far as the tenant is concerned, exist.

The choice is protective. As long as the scope stays narrow, a mistake in the synchronisation configuration can affect the test objects at worst.

#The CN must match the samAccountName

L1

This is not an aesthetic preference, and it is the costliest trap in the whole structure.

A user created with -Name "Ufficio Acquisti" and -SamAccountName "ufficio.acquisti" makes the second command fail with a laconic wasn't found, while the first works fine. The result is a script that assigns half the permissions and fails on the other half — without stopping.

Correct creation: the readable name belongs in DisplayName, which is what appears in the address book.

New-ADUser -Name "mario.rossi" -DisplayName "Mario Rossi" -SamAccountName "mario.rossi"

Checking and fixing non-conforming objects:

Get-ADUser -SearchBase "OU=SYNC,DC=contoso,DC=lab" -Filter * -Properties Name,DisplayName |
  Where-Object { $_.Name -ne $_.SamAccountName } |
  ForEach-Object {
    Write-Host ("{0} -> {1}" -f $_.Name, $_.SamAccountName)
    Rename-ADObject -Identity $_.DistinguishedName -NewName $_.SamAccountName
  }

#The time hierarchy

L2

Time is the most delicate point in the whole lab, and the reason is told at length in Nine hours off.

Internet (time.windows.com, pool.ntp.org)
  │
  LAB-DC01  → root of the hierarchy, marked /reliable:yes
  │
  ├── domain members  → domhier mode (automatic)
  └── workgroup machines → manual pointer at the domain controller

On the domain controller:

w32tm /config /manualpeerlist:"time.windows.com,0x8 pool.ntp.org,0x8" /syncfromflags:manual /reliable:yes /update
Set-Service w32time -StartupType Automatic
Restart-Service w32time
w32tm /resync /rediscover

On domain members:

w32tm /config /syncfromflags:domhier /update
Restart-Service w32time
w32tm /query /source    # must answer with the domain controller's name

Every machine must carry the same time zone as the host running it. The reason is not the convenience of reading the clock, but the correctness of UTC after every reboot.

#The edition limit: five databases

L2

The Standard edition allows at most five mailbox databases per server. An evaluation install behaves as Standard.

Going beyond produces:

RcrExceedDbLimitException: ... maximum databases limit of 5

Two details you only discover by walking into them:

• The limit counts databases present on the server, copies included, not active databases. Four databases with cross copies occupy four slots on each node, not two.
• Installing the second mailbox server automatically creates its own default database with a generated name, consuming one of the five slots. Remove or rename it before creating the project's databases.

The resulting architecture — two active databases per node, each copied to the other — stays inside the limit and gives a fully working, balanced DAG.

Checking state:

Get-MailboxDatabaseCopyStatus * |
  ft Name,Status,ActiveCopy,CopyQueueLength,ReplayQueueLength,ContentIndexState -Auto

Mounted marks the active copy, Healthy a passive copy in step. CopyQueueLength is the number of logs still to copy, ReplayQueueLength those copied but not yet applied: both at zero means replication is current.

#The DAG with no IP address

L2

The DAG is configured without an administrative access point — the model recommended from Exchange 2013 onward. The underlying cluster has neither a network name nor an IP address of its own.

The practical consequence is worth knowing, because it looks like a fault and is not: cluster commands do not accept -Cluster. Trying produces an RPC-unavailable error.

Get-ClusterNode | ft Name,State -Auto   # correct: run it on a node
Get-ClusterNode -Cluster DAG1           # error, and expected

#Quorum and the witness beyond the firewall

L2

With two nodes and a witness there are three votes and the majority is two: the lab survives losing a node or the witness, not both.

The witness lives on the domain controller, which sits in a different segment from the Exchange nodes, beyond the firewall. This is a real fragility and it shows up at every firewall maintenance.

Symptom: QuorumGroup fails saying the File Share Witness is offline, while FileShareQuorum reports Passed.

The distinction between the two checks is informative: FileShareQuorum verifies that the witness configuration is correct, QuorumGroup that the resource is online. During a firewall reboot the nodes lose access to the share, the resource goes to Failed and does not come back by itself.

Get-ClusterResource | ft Name,State -Auto
Start-ClusterGroup "Cluster Group"

Replication health:

Test-ReplicationHealth -Identity LAB-MBX01 | Where-Object Result -ne 'Passed'

Test-ReplicationHealth runs nineteen checks per server. If the filtered command prints nothing, the DAG is sound.

#Node maintenance

L3

The sequence must be run from the node that stays active.

Set-ServerComponentState LAB-MBX01 -Component HubTransport -State Draining -Requester Maintenance
Invoke-Command -ComputerName LAB-MBX01 -ScriptBlock { Restart-Service MSExchangeTransport }
Redirect-Message -Server LAB-MBX01 -Target LAB-MBX02.contoso.lab
Suspend-ClusterNode -Name LAB-MBX01
Set-MailboxServer LAB-MBX01 -DatabaseCopyActivationDisabledAndMoveNow $true
Set-MailboxServer LAB-MBX01 -DatabaseCopyAutoActivationPolicy Blocked
Set-ServerComponentState LAB-MBX01 -Component ServerWideOffline -State Inactive -Requester Maintenance

Two steps that sequence does not complete on its own, and which must be verified.

Active databases do not always move: the flag can hit Active Manager's limit of four moves per hour per database. They then have to be moved by hand.

Move-ActiveMailboxDatabase DB01 -ActivateOnServer LAB-MBX02 -Confirm:$false

And Suspend-ClusterNode puts the node into Paused but does not move the cluster group that is already there: the Primary Active Manager stays where it is until you move it.

Move-ClusterGroup -Name "Cluster Group" -Node LAB-MBX02

Verifying an empty node: no Mounted on the node under maintenance, PrimaryActiveManager on the other, node in Paused.

After maintenance the databases stay concentrated on one node. Rebalancing them is not cosmetic: activation preferences reflect capacity decisions, and you are returning to the designed configuration.

#The application connectors: two distinct errors

L2

Two receive connectors reproduce the pair found in production, and the difference between them explains two errors you meet in real logs.

Creating the anonymous connector, in two steps — the second is the one people forget:

New-ReceiveConnector -Name "smtp-app-lab" -TransportRole FrontendTransport -Server LAB-MBX01 `
  -Bindings 0.0.0.0:25 `
  -RemoteIPRanges 10.20.20.10,10.20.20.11,10.20.20.34,10.20.30.10 `
  -PermissionGroups AnonymousUsers -AuthMechanism Tls -Enabled $true

Get-ReceiveConnector "LAB-MBX01\smtp-app-lab" |
  Add-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON" `
    -ExtendedRights "Ms-Exch-SMTP-Accept-Any-Recipient"

The Ms-Exch-SMTP-Accept-Any-Recipient right is what authorises relay to external domains; RemoteIPRanges limits it to the listed addresses. Without that right, an outbound send gets 550 5.7.54.

How Exchange picks a connector. Several connectors can share the same binding; Exchange chooses the one with the most specific source address range. But specificity only applies within the same address family: a connector listening on 0.0.0.0:25 with IPv4 ranges is never selected for an IPv6 connection. The full story is in 550 5.7.54 on the right connector.

Protocol logs are the only place where Exchange states which connector accepted a session and from where:

Get-ReceiveConnector -Server LAB-MBX01 | Set-ReceiveConnector -ProtocolLoggingLevel Verbose

The second field of each line is the connector that accepted the session, the fifth is the source address. Comparing them is the fastest way to understand what happened.

#Perimeter transport and EdgeSync

L3

Two Windows servers outside the domain, in a DMZ, carrying only the perimeter transport role. They never contact Active Directory: they keep a copy of the configuration data in AD LDS, populated by EdgeSync.

That isolation is what makes them suitable for the perimeter — and it is also why, if the local administrator password is lost, there is no domain administrator who can get in.

The three points where this stage fails.

The primary DNS suffix. A workgroup machine does not inherit it from the domain and it must be set by hand in the registry. Without it the full name stays the bare NetBIOS name, and the subscription fails.

[System.Net.Dns]::GetHostEntry($env:COMPUTERNAME).HostName

The clock. EdgeSync uses self-signed certificates evaluated against the current time. A few minutes of skew makes them invalid, and the resulting error talks about connectivity, not time: The LDAP server is unavailable.

The services after subscription. Even with the right time and an open port, EdgeSync may not start until the services that evaluate certificates are restarted.

Restart-Service ADAM_MSExchange -Force
Restart-Service MSExchangeEdgeCredential
Restart-Service MSExchangeTransport

How to read the subscription's status:

A message to a non-existent external domain sitting in the queue in Retry is the expected result: it proves the internal → perimeter path works.

#Direct Trust: why you do not need a CA

L3

The certificates the mailbox and perimeter servers exchange are self-signed, generated by Exchange. Trust does not come from the certification chain: it comes from the certificate being published in the directory — Active Directory on one side, AD LDS on the other — and distributed by EdgeSync.

The practical consequence is that a CA is unnecessary. A certificate issued by an authority would add nothing to the verification.

Renewal, at expiry, is done by redoing the subscription.

Get-ExchangeCertificate | ft Thumbprint,Subject,Issuer,NotAfter,Services -Auto

Where Subject and Issuer match, the certificate is self-signed.

#HAProxy in L4 mode

L2

Twenty-nine lines of configuration, of which two actually matter.

frontend ft_smtp
    bind 10.20.20.38:25
    default_backend bk_smtp

backend bk_smtp
    balance roundrobin
    source 10.20.20.34
    server mbx01 10.20.20.10:25 check
    server mbx02 10.20.20.11:25 check

mode tcp — layer 4, no TLS termination. The handshake crosses the balancer intact and reaches Exchange, exactly as the commercial balancer it reproduces does. In HTTP mode the balancer would terminate the connection, and the TLS phenomenon under study would no longer be observable.

source 10.20.20.34 — forces the source address towards the servers. That is the SNAT.

Check before restarting — no output means the configuration is valid:

haproxy -c -f /etc/haproxy/haproxy.cfg

Testing the full path, which must answer with one of the two Exchange banners:

telnet 10.20.20.38 25

#Certificates: the declared gaps

L2

Two kinds of certificate exist in the lab, and a third is missing.

The auth certificate deserves attention because it expires silently, and its expiry breaks hybrid functions such as free/busy sharing with no obvious message. It is a check worth adding to periodic verification, in production too.

Get-ExchangeCertificate -Thumbprint (Get-AuthConfig).CurrentCertificateThumbprint |
  fl NotBefore,NotAfter,Subject

The service certificate cannot be obtained for a lab domain, so certificates stay self-signed. This must be declared when reading results: external client access tests run with a certificate the client does not trust. That does not prevent verifying the network path — the connection establishes, traffic flows — but "the browser complained" and "it did not work" are different outcomes, and only one of them is a fault.

To check whether a chain is actually verifiable and not merely well-formed:

certutil -verify -urlfetch <file.cer>

-urlfetch really does download CRLs and intermediate certificates from the addresses inside the certificate.

#The tenant, and choosing not to run the HCW

L2

The lab synchronises to a real Microsoft 365 tenant, not a trial one. That choice increases fidelity and imposes caution.

The most important part: the Hybrid Configuration Wizard was not run. It writes organisation-level connectors into the tenant, and that tenant carries real mail. Skipping it means giving up part of the hybrid behaviour; running it would have meant changing the mail configuration of an environment in use.

The lab subdomain was verified automatically, with no TXT record: Microsoft inherits proof of ownership from the parent domain, already verified in the same tenant. When adding it, the services step was deliberately skipped — no MX, CNAME or SPF record — because the lab only needs the domain to be verified, not to route mail.

#Entra Connect: the OU filter

L2

This is the tenant's main protection. Entra Connect considers exclusively one organisational unit: any object outside it does not, for the tenant, exist.

The filter must be re-checked every time the wizard is run again: it presents the current settings but does nothing to stop you changing them by accident.

Set-ADSyncScheduler -SyncCycleEnabled $true

#Exchange hybrid deployment and writeback

L2

Enabled after the initial configuration, and the difference was substantial.

Before enabling it, Exchange Online did not recognise on-premises objects as valid recipients. Afterwards, the same objects appear correctly as MailUser, which is how a hybrid represents a mailbox living on the other side.

The feature also enables writeback of certain attributes from the cloud into Active Directory: archive state, safe and blocked sender lists, public delegates, and addresses created in the cloud.

The effect is observable: on a synchronised object, in Active Directory, an X500 address appears with the prefix /o=ExchangeLabs — the Exchange Online organisation, not the local one. That address was born in the cloud and came back. It preserves the ability to reply to messages sent while the mailbox lived elsewhere, and must not be removed.

After enabling it, a full cycle is mandatory:

Start-ADSyncSyncCycle -PolicyType Initial

#The custom synchronisation rule

L3

It synchronises only those with a given employeeType value, or those carrying an extension attribute with an agreed value. It writes cloudFiltered, the attribute Entra Connect uses to decide whether an object exists for the tenant.

IIF(IsPresent([employeeType]),
    IIF([employeeType]="Interno", False,
        IIF(IsPresent([extensionAttribute1]),
            IIF([extensionAttribute1]="SYNC365", False, True), True)),
    IIF(IsPresent([extensionAttribute1]),
        IIF([extensionAttribute1]="SYNC365", False, True), True))

Truth table:

Four things to know before writing it.

The expression goes in the Source field. There is no separate field: when you set FlowType = Expression, the Source column becomes a free text box. This is where people get stuck looking for a field that does not exist.

Precedence must be below 100. From 100 upwards sit Microsoft's built-in rules; a rule with higher precedence would be overridden and cloudFiltered would never be written.

No boolean operators. The expression language is limited: the nested IIF structure is wordier but safe. IsPresent must always be checked before comparing an attribute that might be absent, otherwise the expression fails at runtime on precisely those objects.

After every change a full cycle is needed. A Delta cycle evaluates only recently changed objects; new rules apply to existing ones only with Initial.

#Staging mode

L2

This is the correct way to develop or change rules. In staging the engine imports, applies the rules and computes every difference, but exports nothing: you can be wrong as many times as you need without a single object in the tenant noticing.

Why it is indispensable, and it is not obvious: an object that leaves the synchronisation scope is indistinguishable, to Entra Connect, from an object that has disappeared. It generates a deletion in the cloud.

This was verified deliberately during the build: removing the admitting attribute from an already-synchronised user made a pending Delete appear in staging. No deletion occurred.

The procedure, and the step that actually protects you:

1. Enable staging mode
2. Change or create the rule in the editor
3. Run an Initial cycle
4. Inspect the pending operations (Pending Export)
5. Correct until no unwanted deletions appear
6. Only then disable staging
7. Run another Initial cycle
8. Verify the objects in the tenant

Step 5 is the rule: you never leave staging with a pending deletion you do not actually want.

#The connector space: why is this user not syncing

L2

The tool that answers the most frequent question in thirty seconds.

Overview — what would be exported: Connectors tab → the tenant connector → Search Connector Space → Scope: Pending Export → tick Add, Modify, Delete → Search.

Detail — why an object is excluded: Connectors tab → the forest connector → Search Connector Space → Scope: DN → the object's distinguished name → select it → Preview → Generate Preview → Import Attribute Flow.

There you see every rule applied to the object, in precedence order, and the final value of every attribute.

#Three principles

L1

Learned from the build, in order of usefulness.

Read the log, not the message on screen. The dialog's message is almost always generic. The real cause is in the event log or the tool's own log file. The most instructive error of the whole project — could not configure application based authentication — had nothing to do with authentication: the event log showed a certificate with inconsistent dates, and the cause was a clock.

Compare two independent measurements. A single figure cannot tell you whether it is right. Two figures that should agree and do not will pinpoint the problem. That is how the time skew was found: two timestamps in the same event, exactly nine hours apart.

Identify which layer is failing. TCP connecting but LDAP not answering means the problem is above the transport, not in the network. Continuing to check firewalls and routing at that point is wasted time.

#Read-only commands

L1

Network:

Test-NetConnection <ip> -Port <port> -InformationLevel Quiet
Resolve-DnsName <name>
Get-NetConnectionProfile

Identity and Kerberos:

nltest /sc_query:contoso.lab
Test-ComputerSecureChannel -Verbose
klist
w32tm /query /status
(Get-Date).ToUniversalTime()

Exchange and the DAG:

Get-ClusterNode | ft Name,State,StatusInformation,DrainStatus -Auto
Get-MailboxDatabaseCopyStatus * | ft Name,Status,ActiveCopy,CopyQueueLength -Auto
Get-Queue -Server <server> | ft Identity,Status,MessageCount,NextHopDomain -Auto
Get-MessageTrackingLog -Server <server> -Sender <address> -Start (Get-Date).AddMinutes(-30) |
  fl Timestamp,EventId,Source,RecipientStatus,Recipients

Entra Connect, where the real cause of failures lives:

Get-WinEvent -LogName Application -MaxEvents 100 |
  Where-Object { $_.ProviderName -match "ADSync|Directory Synchronization|Entra" } |
  Select-Object TimeCreated,ProviderName,LevelDisplayName,Message | fl

#Shadow redundancy is not an anomaly

L2

Tracking logs show HARECEIVE and HAREDIRECT events and Shadow queues. These are not faults: they are the mechanism by which Exchange keeps a copy of every message on the other node until the primary confirms delivery.

A Shadow queue with messages in it is normal. A Shadow queue that grows and never drains points to a node that has stopped confirming deliveries, and that is a symptom worth investigating.

#Small but recurring errors

L1

#The four long stories

L1

Four faults required real diagnosis, and each is told at length elsewhere.

• Nine hours off: the certificate error was the clock — the hypervisor, the guests' time zone, and the collateral damage to Kerberos and the DAG.
• 550 5.7.54 on the right connector: it arrived over IPv6 — why a correctly configured connector was never selected.
• Three assumptions an Exchange script never states — the wrong session, the object's name, the silent prerequisites.
• Who decides a hybrid mailbox's primary address — the policy, the command and Entra contending for the same field.

The common thread across all four: the error message pointed in the wrong direction. Not through bad design — a component can only report what it sees — but because the cause always lay one layer below where the symptom showed.