I am working on a binance trading bot using asyncio and the Binance API to stream and process cryptocurrency data. However, I frequently encounter a “Queue overflow” error which suggests that the internal queue handling incoming messages is full, resulting in data being processed too slowly. Here’s the error message I repeatedly see:
Traceback:
Queue overflow for coin (WIFUSDT) !
Sleeping for 30 seconds before retrying...
Unknown message type: {'e': 'error', 'm': 'Queue overflow. Message not filled'}, weight set to 5
This error occurs despite my attempts to manage the queue size and processing rate.
Below is a simplified version of the part of the code that seems to be address the issue:
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
class RateLimiter:
# initialization and request allowance logic
class LOB:
def __init__(self, symbols):
# setup, including asyncio loop and Binance Socket Manager
async def async_stream_data(self, symbol):
while True:
try:
async with self.bsm.futures_multiplex_socket(streams=[f"{symbol.lower()}@depth20@100ms", f"{symbol.lower()}@kline_4h", f"{symbol.lower()}@trade"]) as stream:
while True:
msg = await asyncio.wait_for(stream.recv(), timeout=60)
if self.stream._queue.qsize() < 100:
await self.process_message(msg, symbol)
else:
await asyncio.sleep(10) # sleep if queue is full
except Exception as e:
# handle exceptions and retry logic
async def process_message(self, msg, symbol):
# message processing logic
# Main execution flow setup and entry point
async def main():
async_coins = ["BTCUSDT", "WIFUSDT"]
tasks = []
try:
for coin in async_coins:
loop = asyncio.get_event_loop()
tasks.append(loop.create_task(lob.async_stream_data(coin)))
# ---------------------------------------------------------------
# await all asyns tasks
return await asyncio.gather(*tasks, return_exceptions=True)
except Exception as e:
print(f"An Exception occured in main: {e}")
if __name__ == "__main__":
asyncio.run(main())
Question:
How can I optimize the message processing or manage the queue more efficiently to avoid overflows? Are there better strategies for handling high-frequency data in such an asynchronous environment?
Python 3.11
python-binance 1.0.19
python-asyncio