managing websocket buffer

Websocket
1

Regarding the last article : increasing delays on @bookTicker streams

Initial Prob: When i subscribe almost all(60~70) futures BookTicker streams through websocket, i endup getting latency (currentTime-transactionTime) of more than 500000ms within few hours.

I’ve tried several approaches suggested there.

  1. Changing t2.micro to t3.micro to get faster network speed,
  2. subscribing to AllbookTicker stream( !bookTicker) instead of combination of individual streams through ‘/’
  3. spliting tickers to several streams so that single stream has only 20-30 tickers each.
  4. regularly close ws and reconnect.

all of methods worked for only short period of time and got back to the same situation again.
I think the reason would be the piling up of the websocket incoming data. eventHandling speed can’t follow up the inflow so the requests are lining up which makes the delay.

So here’s the problem.

  1. How can i know how many requests are piled up?(i.e. ws property that’s indicating the length of buffered request yet to be processed)
  2. Is there any way that i can clear the buffer so that i can give up the piled data and start processing the new one?
  3. Or any other methods that this wouldn’t happen at all?.. I think many people are using AWS EC2 t3a.micro instance and binance wouldn’t make AllbookTicker stream if that popular intance couldn’t handle this. Maybe there’s a proper way to use the websocket ?..

Using nodejs/ws module/Ubuntu EC2 AWS Tokyo t3a.micro/Elastic IP bound.

2

More streams means more data coming from the connection. Handling network data requires CPU, memory, etc.
If you need the book ticker of all the symbols, it can be a challenge on that client settings.

Spin up a powerful machine or reduce the stream numbers.

3

@lolsm Did you find a solution to this? I have the same problem with the order book depth updates. It happens only with Binance.

4

Hi guys,

yes there is a way. Find the path where unicorn binance websocket api manager is installed. In my case this was C:\Users\Administrator\AppData\Local\Programs\Python\Python39\Lib\site-packages\unicorn_binance_websocket_api.

Then edit file unicorn_binance_websocket_api_manager.py. I created a new method at the bottom of the file. This is the code:

def clear_buffer(self, stream_buffer_name=False):
    #self.stream_buffer_locks[stream_buffer_name] = threading.Lock()
    #self.stream_buffers[stream_buffer_name] = []
    if stream_buffer_name is False:
        try:
            with self.stream_buffer_lock:
                stream_data = self.stream_buffer = []
            return stream_data
        except IndexError:
            return False
    else:
        try:
            with self.stream_buffer_locks[stream_buffer_name]:
                stream_data = self.stream_buffers[stream_buffer_name] = []
            return stream_data
        except IndexError:
            return False
        except KeyError:
            return False

Now when ever you want. Call binance_websocket_api_manager.clear_buffer() in your program and you will be fine :wink:

hope that helps.

1 Like
5

I implemented such a feature to UBWA:
clear_stream_buffer()

The stream buffer isnt a list anymore its now a deque() and supports LIFO and FIFO stacks and now you can also set a maxlenght of the stream_buffer.

More info in the wiki: `stream_buffer` · LUCIT-Systems-and-Development/unicorn-binance-websocket-api Wiki · GitHub