flume windows agent reports OutOfMemory

abloz 2013-01-29
abloz 2013-01-29

周海汉

2013.1.29

http://abloz.com

when i start the flume node service,the agent is OPENING. I need to gather logs from windows 7, but the flume master and collector is running on linux. I download 0.9.4 exe file from flume official web.

When I start the node from windows, the status is OPENING.

zhouhh.TKOffice.local zhouhh.TKOffice.local zhouhh.TKOffice.local OPENING Fri Jan 25 11:45:01 CST 2013 4

And the log reports OutOfMemoryError.

================ D:Program FilesClouderaFlume 0.9.4logflumenode-stdout.2013-01-25.log

2013-01-25 15:31:03 Commons Daemon procrun stdout initializedListening for transport dt_socket at address: 8888 java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid8100.hprof … Heap dump file created [13270531 bytes in 0.115 secs]

=============== D:Program FilesClouderaFlume 0.9.4logflumenode-stderr.2013-01-25.log … Exception in thread “logicalNode zhouhh.TKOffice.local-25” java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:59) at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845) at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891) at com.cloudera.flume.handlers.hdfs.SeqfileEventSource.next(SeqfileEventSource.java:62) at com.cloudera.flume.core.EventUtil.dumpAll(EventUtil.java:47) at com.cloudera.flume.agent.durability.NaiveFileWALManager.checkAndStripAckFraming(NaiveFileWALManager.java:378) at com.cloudera.flume.agent.durability.NaiveFileWALManager.recoverLog(NaiveFileWALManager.java:288) at com.cloudera.flume.agent.durability.NaiveFileWALManager.recover(NaiveFileWALManager.java:411) at com.cloudera.flume.agent.durability.NaiveFileWALDeco.open(NaiveFileWALDeco.java:240) at com.cloudera.flume.agent.AgentSink.open(AgentSink.java:150) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)

I have 4GB memory, and the source file to tail only 1KB, What can I do to remove this error?

I found that if I change windows agent configure from

tail("c:\test.txt") agentSink("hadoop48",35853) ;

to

tail("c:\test.txt") agentDFOSink("hadoop48",35853) ;

then the agent changes status from OPENING to ACTIVE.

my windows agent is :192.168.20.81, while linux collector is : 192.168.10.48

another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.

 in the flume0.9.4 tray icon menu I could edit the parameter, and edit registry is also ok.

I add two parameter:

-XX:MaxDirectMemorySize=500m

-XX:MaxNewSize=500m

and change JvmMs from 128 to 500(-Xms 500,-Xmx 1024)

increase -Xms and -Xmx values which are shown as “Initial memory pool” and “Maximum memory pool” in “Java” tab of configuration of tray icon.

(HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0 > FlumeNode > Parameters > Java also can change the javavm parameter)

it dosen’t work while using agentSink and report OutOfMemory again after configured agentDFOSink to agentSink. So I removed the WAL dir because there may be currupted due to previous out of memory.

I  removed those directories below of WAL , configured the agent sink to agentSink, and restart flume service,It’s OK!

C:tmpflume-ZHOUHH$agentzhouhh.TKOffice.local>ls

dfo_error   dfo_logged   dfo_writing  error   logged   sent

dfo_import  dfo_sending  done         import  sending  writing

Thanks to Jeong-shik Jang’s help.


如非注明转载, 均为原创. 本站遵循知识共享CC协议,转载请注明来源