Certificate should refresh after restart of server. But as I understand you are restarting the server. WF does not cache certificate. Apparently you are not changing file on proper place. Are you changing proper security realm referenced by undertow subsystem? When in domain mode you should change keystore file on all servers. Isn't there load balancer providing certificate?
Your response was very helpful in a few different areas. It stopped me pursuing the caching path and made me look at some other possible causes. And even though we dont have load balancers in our environment, this statement of yours did make me think that the problem might be somewhere on our network on another device. It turns out that we have a web application firewall that still had the old certificate loaded and was providing it to the client. After updating this, the problem went away.
Thanks again for the quick response, really appreciated.