![]() No direct support for low-latency streaming (can use native Kafka clients to get this, perhaps expose Websocket interface).This is fairly easy to add in a frontend, but means that using this API directly from various services might be a bit ugly. No direct support for regular JSON messages / only supports Avro-flavored JSON.Google Task the confluence proxy seems to cover most of what we need, especially on the producer side.Ī few nitpicks I noticed while browsing the docs:.rebalance periodically based on committed offsets in each partition?.each consumer group maintains its own commit offset, so speed independent.use one consumer group id per pub/sub client. ![]() Goal: Deliver each message to all consumers. once max number of retries reached, place message in separate dead-letter queue for inspection (ideally with some debugging info).dequeue messages from retry queue as well, keep counter of retries.else: place message(s) into retry queue, ack primary queue.message is placed in internal queue & handed out again *once* messages are only committed up to the failed message.If no ack is received within this timeout window:.can be increased by client up to 12 hours (AWS) in poll request or per message ('heartbeat').Clients rebalance partitions on join need more (~2x?) partitions than clients.Offset can be *committed* once acks are received by service clients ( = false).Reading from a partition is separate from committing the offset.Each partition's *commit* offset is stored in zookeeper. ![]() Kafka splits a topic into many partitions. ![]() Goal: deliver each message at least once while avoiding duplicate processing. Fall-back to polling is common in websocket libraries. Client connectionĪ popular and well supported option is to use websockets or long polling. Implementation sketch for a reliable event bus with both task queuing and pub/sub functionality, originally based on this restbase doc. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |