11Release Notes - Flume - Version v1.4.0
22
33** New Feature
4- * [FLUME-917] - Thrift Support
54 * [FLUME-924] - Implement a JMS source for Flume NG
65 * [FLUME-997] - Support secure transport mechanism
76 * [FLUME-1170] - Create a framework for transaction listeners
8- * [FLUME-1249] - Add start/stop script
9- * [FLUME-1485] - FileChannel should support checksum
107 * [FLUME-1502] - Support for running simple configurations embedded in host process
118 * [FLUME-1516] - FileChannel Write Dual Checkpoints to avoid replays
129 * [FLUME-1632] - Persist progress on each file in file spooling client/source
13- * [FLUME-1669] - Add support for columnar event serializer in HDFS
14- * [FLUME-1715] - OpenTSDB Sink
10+ * [FLUME-1735] - Add support for a plugins.d directory
1511 * [FLUME-1894] - Implement Thrift RPC
1612 * [FLUME-1917] - FileChannel group commit (coalesce fsync)
17- * [FLUME-1941] - Support defaults or inheritance in configs
18- * [FLUME-1988] - Add Support for Additional Deserializers for SpoolingDirectorySource
19- * [FLUME-1998] - FlumeRESTClient - A HTTP Client for Interacting with REST Servers
20- * [FLUME-1999] - GooglePlusSource - An EventDrivenSource for for Streaming In Data from Google+
21- * [FLUME-2000] - EmailSource - An EventDrivenSource for Streaming In Events from POP/IMAP Mailboxes
22- * [FLUME-2001] - TwitterSource - An EventDrivenSource for Streaming In Data from Twitter
2313 * [FLUME-2004] - Need to capture metrics on the Flume exec source such as events received, rejected, etc.
2414 * [FLUME-2010] - Support Avro records in Log4jAppender and the HDFS Sink
2515 * [FLUME-2048] - Avro container file deserializer
16+ * [FLUME-2070] - Add a Flume Morphline Solr Sink
2617
2718** Improvement
28- * [FLUME-841] - Generate javadoc from NG build
29- * [FLUME-890] - Add unit tests for flume ng channels
30- * [FLUME-1033] - Create system tests for flume
31- * [FLUME-1071] - Evaluate performance of AVRO source with multiple clients
32- * [FLUME-1076] - SInk batch sizes vary wildy
33- * [FLUME-1081] - Load testing tool for Avro source
19+ * [FLUME-1076] - Sink batch sizes vary wildy
3420 * [FLUME-1100] - HDFSWriterFactory and HDFSFormatterFactory should allow extension
35- * [FLUME-1141] - Avro compression unit test should fail if compression does not work
36- * [FLUME-1159] - Add reconfigurability support to components
37- * [FLUME-1178] - Improve performance of SyslogTcpSource
38- * [FLUME-1209] - Support file name extension for HDFS sink
39- * [FLUME-1237] - Refactor configuration and add more tests.
40- * [FLUME-1274] - Clarify semantics of uncaught runtime exceptions thrown from configure() method
41- * [FLUME-1278] - Flume hot-reconfiguration specification
42- * [FLUME-1413] - Document of Channel (especially FileChannel) and Transaction
43- * [FLUME-1497] - Make the TestFileChannel.testInOut() more deterministic
44- * [FLUME-1521] - Document the StressSource
4521 * [FLUME-1571] - Channels should check for positive capacity and transaction capacity values
4622 * [FLUME-1586] - File Channel should support verifying integrity of individual events.
4723 * [FLUME-1652] - Logutils.getLogs could NPE
4824 * [FLUME-1661] - ExecSource cannot execute complex *nix commands
4925 * [FLUME-1677] - Add File-channel dependency to flume-ng-node's pom.xml
5026 * [FLUME-1699] - Make the rename of the meta file platform neutral
5127 * [FLUME-1702] - HDFSEventSink should write to a hidden file as opposed to a .tmp file
52- * [FLUME-1722] - License file corrections
53- * [FLUME-1735] - Add support for a plugins.d directory
5428 * [FLUME-1740] - Remove contrib/ directory from Flume NG
5529 * [FLUME-1745] - FlumeConfiguration Eats Exceptions
5630 * [FLUME-1756] - Avro client should be able to use load balancing RPC
5731 * [FLUME-1757] - Improve configuration of hbase serializers
58- * [FLUME-1759] - make file and dirname of flume log tweakable in log4j
59- * [FLUME-1760] - an extra hdfs. prefix feels redundant in HDFS sink configuration
6032 * [FLUME-1762] - File Channel should recover automatically if the checkpoint is incomplete or bad by deleting the contents of the checkpoint directory
61- * [FLUME-1764] - File_roll type Sink how to achieve according to multiple files dynamically generated directory, and the file on the corresponding directory
6233 * [FLUME-1768] - Multiplexing channel selector should allow optional-only channels
6334 * [FLUME-1769] - Replicating channel selector should support optional channels
6435 * [FLUME-1770] - Flume should have serializer which supports serializer the headers to a simple string
6536 * [FLUME-1777] - AbstractSource does not provide enough implementation for sub-classes
6637 * [FLUME-1790] - Commands in EncryptionTestUtils comments require high encryption pack to be installed
6738 * [FLUME-1794] - FileChannel check for full disks in the background
6839 * [FLUME-1800] - Docs for spooling source durability changes
69- * [FLUME-1801] - Configurable batch size for spooling client
7040 * [FLUME-1808] - ElasticSearchSink is missing log4.properties
7141 * [FLUME-1821] - Support configuration of hbase instances to be used in AsyncHBaseSink from flume config
7242 * [FLUME-1847] - NPE in SourceConfiguration
@@ -83,7 +53,6 @@ Release Notes - Flume - Version v1.4.0
8353 * [FLUME-1926] - Optionally timeout Avro Sink Rpc Clients to avoid stickiness
8454 * [FLUME-1940] - Log a snapshot of Flume metrics on shutdown
8555 * [FLUME-1945] - HBase Serializer allow key from regular expression group
86- * [FLUME-1961] - Enhancements to the Channel interface
8756 * [FLUME-1976] - JMS Source document should provide instruction on JMS implementation jars
8857 * [FLUME-1977] - JMS Source connectionFactory property is not documented
8958 * [FLUME-1992] - ElasticSearch dependency is marked optional
@@ -97,47 +66,24 @@ Release Notes - Flume - Version v1.4.0
9766 * [FLUME-2049] - Compile ElasticSearchSink with elasticsearch 0.90
9867 * [FLUME-2062] - make it possible for HBase sink to deposit event headers into corresponding column qualifiers
9968 * [FLUME-2063] - Add Configurable charset to RegexHbaseEventSerializer
100- * [FLUME-2070 ] - Add a Flume Morphline Solr Sink
69+ * [FLUME-2076 ] - JMX metrics support for HTTP Source
10170 * [FLUME-2093] - binary tarball that is created by flume's assembly shouldn't contain sources
10271 * [FLUME-2100] - Increase default batchSize of Morphline Solr Sink
10372 * [FLUME-2105] - Add docs for MorphlineSolrSink
10473
10574** Bug
106- * [FLUME-981] - Unify sink failure behavior, throwing EventDeliveryException
107- * [FLUME-1000] - Properties validator misses an invalid config
108- * [FLUME-1086] - duplicate lib in Flume's lib dir
109- * [FLUME-1093] - HDFSEventSink gives no indication of failure when Hadoop jars aren't in the classpath
11075 * [FLUME-1110] - HDFS Sink throws IllegalStateException when flume-daemon shuts down
111- * [FLUME-1115] - project "flume-ng-node" should depends on "flume-recoverable-memory-channel"
112- * [FLUME-1152] - flume-ng script has issue with hadoop 0.23 script
11376 * [FLUME-1153] - flume-ng script is missing some agent options in help output
114- * [FLUME-1161] - Logging output says name:null after initiating agent shutdown
115- * [FLUME-1165] - Separate the component factories into pools which hold state and factories which simply create the new components
11677 * [FLUME-1175] - RollingFileSink complains of Bad File Descriptor upon a reconfig event
117- * [FLUME-1203] - Hbase Sink needs to check for partial failures.
118- * [FLUME-1206] - Hbase sink should pull in dependencies based on profiles
119- * [FLUME-1233] - HDFS Sink has problem with %c escape sequence in bucket path
12078 * [FLUME-1262] - Move doc generation to a different profile
121- * [FLUME-1283] - Flume HDFS Sink can't generate file name containing space and : which mentioned by %c
12279 * [FLUME-1285] - FileChannel has a dependency on Hadoop IO classes
123- * [FLUME-1291] - FileChannel log close calls interrupt on current thread and does not kill worker thread immediately
12480 * [FLUME-1296] - Lifecycle supervisor should check if the monitor service is still running before supervising
125- * [FLUME-1308] - HDFS Sink throws DFSOutputStream when exception when maxOpenFiles=1
126- * [FLUME-1321] - BasicTransactionSemantics should never throw from close()
127- * [FLUME-1378] - Sink falls into infinite lifecycle with EventDeliveryException
128- * [FLUME-1403] - File channel throws logFile is null exception in channel.take
129- * [FLUME-1411] - Add average events per second params to MBeans
130- * [FLUME-1481] - Custom component may not support dynamic configuration
13181 * [FLUME-1511] - Scribe-source doesn't handle zero message request correctly.
132- * [FLUME-1551] - Create a home for custom Flume components
133- * [FLUME-1623] - Add more info about failover sink processor
13482 * [FLUME-1676] - ExecSource should provide a configurable charset
13583 * [FLUME-1688] - Bump AsyncHBase version to 1.4.1
13684 * [FLUME-1709] - HDFS CompressedDataStream doesn't support serializer parameter
13785 * [FLUME-1720] - LICENSE file contain entry for protobuf-<version>.jar, however proper artifact name is protobuf-java-<version>.jar
138- * [FLUME-1721] - SpoolDirectorySource stop should shutdown executor
13986 * [FLUME-1731] - SpoolableDirectorySource should have configurable support for deleting files it has already completed instead of renaming
140- * [FLUME-1732] - Build is failing due to netty problems
14187 * [FLUME-1741] - ElasticSearch tests leave directory data/elasticsearch/nodes/ lying around
14288 * [FLUME-1748] - HDFS Sink should check if the thread is interrupted before performing any HDFS operations
14389 * [FLUME-1755] - Load balancing RPC client has issues with downed hosts
@@ -158,7 +104,6 @@ Release Notes - Flume - Version v1.4.0
158104 * [FLUME-1803] - Generated dist tarball is missing flume-ng-embedded-agent
159105 * [FLUME-1804] - JMS source not included in binary dist
160106 * [FLUME-1805] - Embedded agent deps should be specified in dependencyManagement section of pom
161- * [FLUME-1815] - File Channel auto-recovery when checkpoint is bad should be optional
162107 * [FLUME-1818] - Support various layouts in log4jappender
163108 * [FLUME-1819] - ExecSource don't flush the cache if there is no input entries
164109 * [FLUME-1820] - Should not be possible for RPC client to block indefinitely on close()
@@ -190,19 +135,16 @@ Release Notes - Flume - Version v1.4.0
190135 * [FLUME-1911] - Add deprecation back to the legacy thrift code
191136 * [FLUME-1916] - HDFS sink should poll for # of active replicas. If less than required, roll the file.
192137 * [FLUME-1918] - File Channel cannot handle capacity of more than 500 Million events
193- * [FLUME-1919] - Consider adding areYouOK() RPC calls to Avro & Thrift RPC mechanisms
194138 * [FLUME-1922] - HDFS Sink should optionally insert the timestamp at the sink
195139 * [FLUME-1924] - Bug in serializer context parsing in RollingFileSink
196140 * [FLUME-1925] - HDFS timeouts should not starve other threads
197141 * [FLUME-1929] - CheckpointRebuilder main method does not work
198142 * [FLUME-1930] - Inflights should clean up executors on close.
199143 * [FLUME-1931] - HDFS Sink has a commons-lang dependency which is missing in pom
200144 * [FLUME-1932] - no-reload-conf command line param does not work
201- * [FLUME-1936] - Configure max rolls for under-replicated blocks
202145 * [FLUME-1937] - Issue with maxUnderReplication in HDFS sink
203146 * [FLUME-1939] - FlumeEventQueue must check if file is open before setting the length of the file
204147 * [FLUME-1943] - ExecSource tests failing on Jenkins
205- * [FLUME-1944] - parameterising the dependency variables in pom file.
206148 * [FLUME-1948] - plugins.d directory(ies) should be separately overridable, independent of FLUME_HOME
207149 * [FLUME-1949] - Documentation for sink processor lists incorrect default
208150 * [FLUME-1955] - fileSuffix does not work with compressed streams
@@ -233,65 +175,31 @@ Release Notes - Flume - Version v1.4.0
233175 * [FLUME-2054] - Support Version Info on Windows and fix failure of TestVersionInfo
234176 * [FLUME-2057] - Failures in FileChannel's TestEventQueueBackingStoreFactory on Windows
235177 * [FLUME-2060] - Failure in TestLog.testReplaySucceedsWithUnusedEmptyLogMetaDataFastReplay test on Windows
236- * [FLUME-2068] - File Channel issue - recovering from BadCheckpoint exception on Windows
237178 * [FLUME-2072] - JMX metrics support for HBase Sink
238179 * [FLUME-2081] - JMX metrics support for SpoolDir
239180 * [FLUME-2082] - JMX support for Seq Generator Source
240181 * [FLUME-2083] - Avro Source should not start if SSL is enabled and keystore cannot be opened
241182 * [FLUME-2098] - Make Solr sink depend on the CDK version of morphlines
242183
243184** Documentation
244- * [FLUME-1014] - Agent Config docs are out of date
245- * [FLUME-1022] - Update Flume NG Getting Started Guide to mention pulling from trunk instead of flume-728
246- * [FLUME-1144] - Update Flume User Guide with a section on Flume properties such as flume.called.from.service
247- * [FLUME-1592] - Remove appendTimeout from HDFS sink docs
248- * [FLUME-1594] - Document JMX options
249185 * [FLUME-1621] - Document new MemoryChannel parameters in Flume User Guide
250- * [FLUME-1624] - Docs could be more consistent when giving examples of types
251- * [FLUME-1833] - Embedded Agent docs are missing from website
252186 * [FLUME-1910] - Add thrift RPC documentation
253187 * [FLUME-1953] - Fix dev guide error that says sink can read from multiple channels
254188 * [FLUME-1962] - Document proper specification of lzo codec as lzop in Flume User Guide
255189 * [FLUME-1979] - Wrong propname for connection reset interval in avro sink
256190 * [FLUME-2030] - Documentation of Configuration Changes JMSSource, HBaseSink, AsyncHBaseSink and ElasticSearchSink
257191
258192** Task
259- * [FLUME-813] - Build Flume site from Maven
260- * [FLUME-1051] - Configuration System
261- * [FLUME-1618] - Make Flume NG build and tests work with Hadoop 2.0
262193 * [FLUME-1686] - Exclude target directories & Eclipse files from rat checks
263- * [FLUME-2076] - JMX metrics support for HTTP Source
264194 * [FLUME-2094] - Remove the deprecated - Recoverable Memory Channel
265195
266196** Sub-task
267- * [FLUME-731] - Create NG internal diagrams
268- * [FLUME-732] - Create NG developer code tour docs
269- * [FLUME-733] - Define NG master interfaces and classes
270- * [FLUME-738] - Create NG system configuration component
271- * [FLUME-749] - Add support for monitoring APIs
272- * [FLUME-1056] - AvroSource configuration
273- * [FLUME-1057] - AvroSink configuration
274- * [FLUME-1058] - Exec source configuration
275- * [FLUME-1059] - failover sink configuration
276- * [FLUME-1060] - Hdfs sink configuration
277- * [FLUME-1061] - jdbc channel configuration
278- * [FLUME-1062] - Memory Channel configuration
279- * [FLUME-1063] - Miscellanous changes related to imports and minor changes related to config changes
280- * [FLUME-1064] - Channel selector configuration
281- * [FLUME-1065] - netcat source configuration
282- * [FLUME-1066] - PseudoTxnMemoryChannel configuration
283- * [FLUME-1067] - RollingFileSink configuration
284- * [FLUME-1068] - syslog source configuration
285- * [FLUME-1069] - FileChannel configuration
286197 * [FLUME-1626] - Support Hbase security in Hbase sink
287198 * [FLUME-1630] - Flume configuration code could be improved
288199 * [FLUME-1674] - Documentation / Wiki
289200 * [FLUME-1896] - Implement Thrift RpcClient
290201 * [FLUME-1897] - Implement Thrift Sink
291202 * [FLUME-1898] - Implement Thrift Source
292- * [FLUME-1989] - Source Code and Unit Tests for org.apache.flume.serialization.RegexDelimiterDeSerializer
293- * [FLUME-1990] - Source Code and Unit Tests for org.apache.flume.serialization.CharSequenceDelimiterDeSerializer
294- * [FLUME-1991] - Documentation for New Deserializers for SpoolingDirectorySource
295203 * [FLUME-2102] - Update LICENSE file for Flume 1.4.0
296204
297205----
0 commit comments