Socket connections dropped

Websocket
1

In this case, I’m talking about the public API, not the private one.

I subscribe to:

  • Aggregated trades for 138 futures.
  • Candles on 8 time frames for 138 futures.

This is done through multiple sockets:

  • 1 socket handles the 138 futures’ aggregated trades.
  • 14 sockets handle 10 futures’ candles on 8 time frames each (the last socket has 8 futures).

The data is read on a 12 cores/64gb/1gbps/NVMe server and each message that arrives is put into a queue and the message is acknowledged right away, so there is no processing delay here.
If I disable everything on this server, but the capture tool, the machine is almost idle while processing this data, so there is no resource issue.

Since the smallest candle is 1 min, I consider data is missing if nothing has been received for 65 seconds. In that case, the socket is closed, re-opened, and the subscription request is sent again.

Problems will arise from a few minutes after start to a few hours, there is no consistency.

  • The first problem:

    Some sockets will not receive any message for the timeout period (65 seconds). They didn’t get a disconnection message. There is just no data in the feed.
    It’s not always the same socket with the same coins, it’s random as well.
    It is rarely one socket, usually 2-3 will have a problem at the same time.

  • The second problem:

    When getting into these cycles of connect/disconnect, the trades socket, which is always the first created and the one with the heaviest data load, will just stop getting any messages. It’s just dropped, without any disconnection message either.

Now it becomes interesting:

  • 1 socket for the trades of 10 coins, and then 10 sockets, with one coin * 8 time frames each.
    This triggers the problem: the trades socket gets dropped without a disconnection message.

  • 1 socket for the trades of 10 coins, and 1 socket for the candles of 10 coins * 8 time frames
    The same problem will happen, but much later.

So the issue seems to happen more quickly if there are more sockets in use.

This is repeatable over and over.

My question here is why do the sockets stop like this? Obviously they get dropped from the Binance side, and they don’t even send a disconnect.

2

Since the smallest candle is 1 min, I consider data is missing if nothing has been received for 65 seconds. In that case, the socket is closed, re-opened, and the subscription request is sent again.

Try increase the threshold to roughly double the current value to diagnose if the issue is latency or disconnection.

It is rarely one socket, usually 2-3 will have a problem at the same time.

This behaviour hints to the cause being on the client side. Do you have the same issue if you distribute the load across more sockets? (Reduce streams per socket)

When getting into these cycles of connect/disconnect, the trades socket, which is always the first created and the one with the heaviest data load, will just stop getting any messages. It’s just dropped, without any disconnection message either.

This is interesting. Do you still receive feedback from the server if you execute commands such as LIST_SUBSCRIPTIONS?

  • 1 socket for the trades of 10 coins, and then 10 sockets, with one coin * 8 time frames each.
    This triggers the problem: the trades socket gets dropped without a disconnection message.
  • 1 socket for the trades of 10 coins, and 1 socket for the candles of 10 coins * 8 time frames
    The same problem will happen, but much later.

Please provide more concrete values instead of much later. How long does it take to disconnect on 1 socket per coin and how long does it take to disconnect on 1 socket for all coins?

My question here is why do the sockets stop like this? Obviously they get dropped from the Binance side, and they don’t even send a disconnect.

There are multiple factors when dealing with websockets, which may arise from both parties. However what you described above is not the expected behaviour.