Avoiding/Detecting stale websocket (user data stream) connections

Websocket
, ,
1

Hi!

Currently, I experience some unexpected behavior in my websocket client for the Binance Spot API. The procedure looks like this:

  • fetch listenKey, establish websocket connection
  • periodically update listenKey (PUT) every 10 minutes (response: 200 OK, body: {})
  • make some trades
    -> user data received correctly
  • no active trading for some time (>> 1h) without any order placement/fills and thus no user data events
  • trades placed, fills happen
    -> no user data received any more, websocket connection seems open but dead

This happens sporadically.

The disconnects from Binance (as stated in docs, may happen every 24h) are registered correctly, so our used websocket implementation should not silently drop close frames or something alike. Since we detect “real” disconnects correctly and our connection stays open for longer periods of time, our used implementation seems to answer Binance’ ping frames correctly.

To be more explicit in my question:

  1. Is there any way to detect such a dead connection? Ping frames from my end seem to be answered correctly.
  2. Is it recommended to always reset the listenKey (DELETE and POST) if we want to reconnect?
  3. It seems that even invalid path/listenKey connections can be established, but won’t deliver any data whatsoever. Is there any way to detect this?

Many thanks in advance!

11

Bump for visibility.

 
We create trading activity drop-copies for several of our clients by using long-running services to capture our clients’ user data streams on their behalf. We’re seeing an issue with listen keys being “silently” dropped.

  • listen keys continue to be successfully refreshed (every 20 minutes) using PUT
  • We have never seen a 'e':'listenKeyExpired' message. Not once over several months of operation.
  • We send periodic pings on all websocket connections, and on pong timeouts, we immediately close and re-establish the connection. We rarely see pong timeouts, if ever.
  • We do detect the 24-hour disconnects from Binance, and immediately reconnect.
  • The volume of our clients’ user data streams over the websockets, which has been normally quite high for several months, began to degrade over a period of several days. We saw a tremendous drop in user data stream volume from 2023-03-09 to 2023-03-10, and has now slowed to essentially a trickle of data. We are definitely missing data, and our clients are getting annoyed.

It looks as other people are having similar issues. Has anyone discovered a solution? It would be much appreciated if you shared here.

 
(Aside: Does anyone know if Binance has plans to provide proper FIX feeds for user data streams and market data?)

2

I’m also very interested in this.
Leaving this comment here to take the thread upper, maybe we get some answers.

3

Hi. Thanks for reminding this missing post @Alex_Soare. Let me respond the question one by one.

(1) Can you try with CLI tool (e.g. websocat) to see if it also happens? If not, probably it’s the library/ programming issue. Please check the source code or find the original developers about this.

(2) It’s not necessary to DELETE the listen key if it’s not yet expired.

(3) So far there is no specific approach to detect the invalid stream name. Please crosscheck the listen key with the API response.

4
  1. ping WebSocket server regularly, if pong messge is received, the connection is alive and healthy.
6

  1. By pinging, did you mean the RestFul PUT listenKey endpoint, or is there a WS ping message we can send?

  2. Why can’t I just keep POSTing to the listenKey endpoint - PUT will risk extending an already expired key but POST will not only extend but also create one if the old key expired.

  3. After running the websocket for a few days, there often occurred silent drop where PUT returns 200 but no more WS updates were received - is there a limit on how many times I can keep a ListenKey alive

5

Hello @kh-dev ,
I’m just facing this issue, did you find any solution?
I’m refreshing the listen key every 30 min, but after some random time (more than 24h, last time it has been alive for 66h) my program stops receiving data from user data stream

I’m using the python-binance library

A weird thing that I’ve seen is that my program keeps refreshing the key with no errors. However, if I take that same key and try to refresh it manually I get the following error:
APIError(code=-1125): This listenKey does not exist.
I don’t know why I’m not receiving this error in my program :confused:

1 Like
7

Same behavior - have you tried to POST every 30 min instead of PUT

8

Hello, most likely your internet connection dropped and you are not handling that error, please check that.

10

The node.js connector (“binance” on npm, since there’s a few) handles this by sending ping frames as heartbeats every few seconds (ping interval):

This starts a race condition (pong timeout):

The healthy winner in this race is either a message OR a pong frame reply before the pong timeout:

If neither happens before the pong timer expires, the connection is assumed to be dead and killed/respawned to replace it with a fresh one:

This is the most resilient way I’ve seen to manage this regardless of network conditions. Doesn’t matter if it’s your network, binance servers, or somewhere in between. Detect a dead connection and enter a reconnection cycle until it succeeds. This needs a bit extra handling for the user data stream (since the listenkey can expire), but the rest is pretty similar. Hope the code references help.

12

Same to me. Let me know if you will found solution

13

Same to me.

I found websocket could not recieve user data update, but is still able to recieve ping and send pong periodically.

14

They get complicated, if your websocket closes simply catch the error and reopen it, if your listenkey is invalid catch the error and generate a new one and subscribe again, if you stop receiving messages in x period of time you close the websocket and you open it again

1 Like