Git Product home page Git Product logo

apache / logging-flume Goto Github PK

View Code? Open in Web Editor NEW
2.5K 227.0 1.6K 44.05 MB

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log-like data

Home Page: https://flume.apache.org/

License: Apache License 2.0

Shell 0.76% Batchfile 0.03% PowerShell 0.46% Python 0.31% Java 96.48% Thrift 0.06% HTML 0.02% Rich Text Format 1.88%
flume java apache api jvm library

logging-flume's Introduction

Welcome to Apache Flume!

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic application.

The Apache Flume 1.x (NG) code line is a refactoring of the first generation Flume to solve certain known issues and limitations of the original design.

Apache Flume is open-sourced under the Apache Software Foundation License v2.0.

Documentation

Documentation is included in the binary distribution under the docs directory. In source form, it can be found in the flume-ng-doc directory.

The Flume 1.x guide and FAQ are available here:

Contact us!

Bug and Issue tracker.

Compiling Flume

Compiling Flume requires the following tools:

  • Oracle Java JDK 1.8
  • Apache Maven 3.x

Note: The Apache Flume build requires more memory than the default configuration. We recommend you set the following Maven options:

export MAVEN_OPTS="-Xms512m -Xmx1024m"

To compile Flume and build a distribution tarball, run mvn install from the top level directory. The artifacts will be placed under flume-ng-dist/target/.

logging-flume's People

Contributors

adenes avatar annamariakeppler avatar bessbd avatar busbey avatar c-sahuja avatar dependabot[bot] avatar geniusjoe avatar granthenke avatar grobmeier avatar harishreedharan avatar jholoman avatar jmhsieh avatar juhanicc avatar liorze avatar majorendre avatar marcellhegedus avatar mpercy avatar nikita15p avatar peterableda avatar pjfanning avatar pmujumdar avatar ppkarwasz avatar rgoers avatar robin021 avatar smola avatar szaboferee avatar tmgstevens avatar turcsanyip avatar willix avatar xuwenping123 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

logging-flume's Issues

Accessing Flume logs on the Windows 11 platform

My log configuration file:

a1.sources = r1
a1.channels = c1
a1.sinks = k1

Describe/configure the source

a1.sources.r1.type = TAILDIR
#断点续传
#a1.sources.r1.positionFile = D:\flumedev\tail_dir.json

多目录监控

a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = D:\flumedev\source\test2\asdfile.log
a1.sources.r1.filegroups.f2 = D:\flumedev\source\test1\test.log

channal

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

Describe the sink

a1.sinks.k1.type = logger

Bind the source and sink to the channel

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

My startup Flume command:
D:\apache-flume-1.11.0-bin>bin\flume-ng agent --conf conf --conf-file D:\flumedev\flume-taildir-hdfs.conf --name a1

The error logs I received:
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:01,712 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:01,712 ERROR [lifecycleSupervisor-1-4] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:04,724 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:07,728 ERROR [lifecycleSupervisor-1-8] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.