Hi, I am using binance-connector-java. I am streaming prices up to 5 different tickers sometimes. For each marketData(message) I get, I a thread which handle each message, the thread are running synchronous.
onMessageCallbackMarket = (message) → {
Runnable task = () → {checkMarketData(message);};
executorUserData.execute(task);
};
The reason I create a thread for handling the message is because I dont want the whole checkMarketData(message) logic in the same thread, because the queue might get overload. Right?
Is this good approach to avoid a long queue of incoming message ?
Creating a thread for each message will not scale appropriately when handling high frequency streams.
You should either queue messages and have a defined amount of threaded works to process the queue or have a defined amount of threaded workers, each with individual queues and distribute messages accordingly.
Either approach taken will require that the scaling of memory and cpu resources is proportional to the frequency of messages.
Hi tantialex, I should clarify more. I am not creating a thread a thread for each message. I create one thread, inside that thread I have a pool of tasks. No task is running parallel, the thread runs one task at a time a, starting with the First in First out (FIFO principle). So basically, I create a task for each message. Is this okay ?
Whats the difference between " have a defined amount of threaded works" and “have a defined amount of threaded workers” in your solution ?
I create one thread, inside that thread I have a pool of tasks. No task is running parallel, the thread runs one task at a time a, starting with the First in First out (FIFO principle). So basically, I create a task for each message. Is this okay ?
Yes, this is a good approach.
Whats the difference between " have a defined amount of threaded works" and “have a defined amount of threaded workers” in your solution ?
Typo from my end, inteded to communicate workers instead of works.
1 Like