Why My WebSocket Service Was Timing Out (And How I Fixed It)

 We have several projects, and we moved all our domain names to Cloudflare, which uses proxying by default. As a result, the actual IP addresses of our servers are hidden. Everything was running smoothly until I came back from lunch one day and found a task assigned to me: the chat service was responding slowly, with noticeable latency. 

I started investigating the problem: 

1.  First, I checked the API endpoint that retrieves messages it was indeed slow. 
2.  I then checked the database query it was fast. 
3.  Next, I reviewed the code and noticed that it triggers an event in the WebSocket service. 
4.  I had the idea to temporarily prevent this event from firing to see if the latency would improve and it did. This confirmed that the WebSocket service was the source of the problem. 

 I then started debugging the WebSocket service: 

•  I checked the SSL certificate and verified that the service was running everything looked fine. 
•  However, the service logs showed an ETIMEDOUT error, caused by a timeout. I had never seen this error before. 
•  I stopped and restarted the service multiple times, but the error persisted. 
•  I tried connecting to the WebSocket via Postman, but the connection still timed out. 
•  Finally, I used curl (curl -vk tls://domainname:9095) and noticed something important: the IP address I was connecting to was not the servers actual IP. This made sense because Cloudflare hides the servers real IP and does not allow all ports to be proxied. 

 Root cause:  The WebSocket service was running on port 9095, which is not supported by Cloudflare. No wonder it kept timing out it was like trying to sneak through a locked door. 

 Solution:  I switched the WebSocket service to a port supported by Cloudflare, and just like magic, the problem was resolved. The chat service was back to its speedy self, and I could finally go back to thinking about lunch instead of timeouts.