batch capacity flume

flume org.apache.flume.ChannelException: Unable to put batch on required channel

abloz 2013-02-25
2013-02-25

执行flume windows版本时遇到如下错误：

2013-02-25 12:05:37,818 (pool-4-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(Spooling
FileLineReader.java:167)] Last read was never comitted - resetting mark position.
2013-02-25 12:05:41,297 (pool-5-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.r
un(SpoolDirectorySource.java:148)] Uncaught exception in Runnable
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name:
 memch1}
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
        at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:143)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.j
ava:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.
java:181)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205
)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keepin
g up with sources, or the buffer size is too tight
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 10 more
2013-02-25 12:05:41,874 (pool-5-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(Spooling
FileLineReader.java:167)] Last read was never comitted - resetting mark position.

source为spooldir,sink 为avro。 我们在运行flume时将java选项–Xms=200m，还是有该错误。spooldir里面的内容大约有1GB，数10个文件。后面有人说memory channel 的capacity 应为 每秒传输event条数 * 要传输的时间秒数。然后capacity 应为agent1.channels.memch1.transactionCapactiy的10-100倍。我们刚开始将这两个值设为 agent1.channels.memch1.capacity = 10000 agent1.channels.memch1.transactionCapactiy = 100 结果报了上述错误。后面将capacity设为1000000,该错误消失(报了outofmemory错误。因为我们内存设小了。)但程序可以正确运行。

如非注明转载, 均为原创. 本站遵循知识共享CC协议,转载请注明来源

FEATURED TAGS
css vc6 http automake linux make makefile voip 乱码 awk flash vista vi vim javascript pietty putty ssh posix subversion svn windows 删除 编译 多线程 wxwidgets ie ubuntu 开源 c python bash 备份 性能 scp 汉字 log ruby 中文 bug msn nginx php shell wordpress mqueue android eclipse java mac ios html5 js mysql protobuf apache hadoop install iocp twisted centos mapreduce hbase thrift tutorial hive erlang lucene hdfs sqoop utf8 filter 草原 yarn ganglia 恢复 scrapy django fsimage flume tail flume-ng mining scala go kafka gradle cassandra baas spring postgres maven mybatis mongodb https nodejs 镜像 心理学 机器学习 Keras theano anaconda docker spark akka-http json 群论 区块链 加密 抽象代数 离散对数 同余 欧拉函数 扩展欧几里德算法 ES6 node-inspect debug win10 vscode 挖矿
FRIENDS