flume的source和sink

abloz 2013-01-21
abloz 2013-01-21

周海汉 2013.1.21

http://abloz.com

常用source源:

  • console

  • 标准输入控制台

  • text(“filename”)

  • 单文本文件源,一行一事件

  • tail(“filename”)

  • 和 Unix 的tail -F 类似。一行一事件。一直打开等待数据,会跟踪文件切换。

  • multitail(“file1″[, ”file2”[, …]])

  • 同 tail 源类似,但可以跟踪多文件。

  • asciisynth(msg_count,msg_size)

  • 一个源,用于产生msg_count 个msg_size大小的随机消息,转成可打印 ASCII字符。

  • syslogUdp(port)

  • UDP 端口上的 Syslog,和syslog兼容。

  • syslogTcp(port)

  • TCP 端口上的 Syslog,和syslog-ng兼容。

 常用sink:

null

Null sink. Events are dropped.

console[("format")]

Console sink. Display to console’s stdout. The “format” argument is optional and defaults to the “debug” output format.

text("_txtfile_"[,"format"])

Textfile sink. Write the events to text file _txtfile_ using output format “format”. The default format is “raw” event bodies with no metadata.

dfs("_dfsfile_")

DFS seqfile sink. Write serialized Flume events to a dfs path such as hdfs://namenode/file or file:///file in Hadoop’s seqfile format. Note that because of the HDFS write semantics, no data for this sink write until the sink is closed.

syslogTcp("_host_",_port_)

Syslog TCP sink. Forward to events to host on TCP port port in syslog wire format (syslog-ng compatible), or to other Flume nodes setup to listen for syslogTcp.

附:所有flume的sinks,sources和decorators.

Sinks

accumulator
agentBEChain
agentBESink
agentBestEffortSink
agentDFOChain
agentDFOSink
agentE2EChain
agentE2ESink
agentFailoverSink
agentSink
autoBEChain
autoDFOChain
autoE2EChain
avroSink
collectorSink
console
counter
counterHistory
customdfs
dfs
escapedCustomDfs
escapedFormatDfs
fail
failChain
formatDfs
ganglia
irc
logicalSink
multigrep
multigrepspec
null
regexhisto
regexhistospec
rpcSink
seqfile
syslogTcp
text
thriftSink

Sources

asciisynth
autoCollectorSource
avroSource
collectorSource
console
exec
execPeriodic
execStream
fail
irc
log4jfile
logicalSource
multitail
nonlsynth
null
report
rpcSource
scribe
seqfile
stdin
synth
synthrndsize
syslogTcp
syslogTcp1
syslogUdp
tail
tailDir
text
thriftSource
tpriosource
twitter

Decorators

ackChecker
ackInjector
ackedWriteAhead
batch
benchinject
benchreport
bloomCheck
bloomGen
choke
delay
digest
diskFailover
exDate
flakeyAppend
format
gunzip
gzip
inmem
insistentAppend
insistentOpen
intervalDroppyAppend
intervalFlakeyAppend
intervalSampler
lazyOpen
mask
mult
nullDeco
probSampler
regex
regexAll
reservoirSampler
select
split
stubbornAppend
unbatch
value

如非注明转载, 均为原创. 本站遵循知识共享CC协议,转载请注明来源