使用kafka-connect时偶尔会报错
报错日志先是connect中两个ERROR
[ERROR] [pool-12-thread-1] connect.runtime.WorkerSourceTask - WorkerSourceTask{id=mgsource_etl_1_20191128-0} Failed to flush, timed out while waiting for producer to flush outstanding 53 messages
[ERROR] [pool-12-thread-1] connect.runtime.SourceTaskOffsetCommitter - WorkerSourceTask{id=mgsource_etl_1_20191128-0} Failed to commit offsets
然后是kafka中两条WARN
[WARN ] [kafka-producer-network-thread | producer-5] producer.internals.Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition Interaction.MainPostLike_6-1 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.. Going to request metadata update now
[WARN ] [kafka-producer-network-thread | producer-5] producer.internals.Sender - [Producer clientId=producer-5] Got error produce response with correlation id 1861556 on topic-partition Interaction.MainPostLike_5-2, retrying (9 attempts left). Error: NETWORK_EXCEPTION
集群没问题,各种压测也做了,很难复现,偶尔几天会有一两次报错。其实数据并没有丢,因为kafka和connect的重试机制,在报错后下次都flush出去了,找不出原因
你的答案