蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎

Related tags

Logging bk-log
Overview

license GitHub release (latest by date) PRs Welcome codecov Web Test

English | 简体中文

重要提示: master 分支在开发过程中可能处于 不稳定或者不可用状态 。 请通过 releases 而非 master 去获取稳定的二进制文件。

蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎,通过蓝鲸智云的专属 Agent 进行日志采集,提供多种场景化的采集、查询功能。

Overview

Features

  • 简单易用的日志采集
  • 可视化的日志字段提取
  • 功能强大的日志查询
  • 实时日志和日志上下文
  • 日志关键字/汇聚告警
  • 支持第三方 ES 接入
  • 分布式跟踪支持
  • 仪表盘能力
  • 在线日志文件提取

Getting Started

  • 安装好MySQL 5.7Python3.6,若同时开发多个项目,请创建Python虚拟环境

  • 创建数据库 CREATE DATABASE bk_log DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

  • 在项目config新建local_settings.py文件,文件内容为数据库配置,如

       DATABASES = {
          'default': {
              'ENGINE': 'django.db.backends.mysql',
              'NAME': 'bk_log',
              'USER': 'root',
              'PASSWORD': '',
              'HOST': '127.0.0.1',
              'PORT': '3306',
          },
      }
  • 编译前端

    cd web
    npm install
    npm run build
  • 配置环境变量

    APP_ID=${APP_ID}
    BK_IAM_V3_INNER_HOST=${BK_IAM_V3_INNER_HOST}
    BK_PAAS_HOST=${BK_PAAS_HOST}
    APP_TOKEN=${APP_TOKEN}
    # BKAPP_REDIS_PASSWORD=${BKAPP_REDIS_PASSWORD}  # 缓存和Celery会使用到redis,如果本地redis有密码需要增加这个环境变量
  • 启动工程 python manage.py runserver 8000

  • 启动celery celery -A worker -l info -c 8

Support

BlueKing Community

  • BK-CMDB:蓝鲸配置平台(蓝鲸 CMDB)是一个面向资产及应用的企业级配置管理平台。
  • BK-CI:蓝鲸持续集成平台是一个开源的持续集成和持续交付系统,可以轻松将你的研发流程呈现到你面前。
  • BK-BCS:蓝鲸容器管理平台是以容器技术为基础,为微服务业务提供编排管理的基础服务平台。
  • BK-BCS-SaaS:蓝鲸容器管理平台 SaaS 基于原生 Kubernetes 和 Mesos 自研的两种模式,提供给用户高度可扩展、灵活易用的容器产品服务。
  • BK-PaaS:蓝鲸 PaaS 平台是一个开放式的开发平台,让开发者可以方便快捷地创建、开发、部署和管理 SaaS 应用。
  • BK-SOPS:标准运维(SOPS)是通过可视化的图形界面进行任务流程编排和执行的系统,是蓝鲸体系中一款轻量级的调度编排类 SaaS 产品。

Contributing

如果你有好的意见或建议,欢迎给我们提 Issues 或 Pull Requests,为蓝鲸开源社区贡献力量。关于 bk-log 分支管理、Issue 以及 PR 规范, 请阅读 Contributing Guide

腾讯开源激励计划 鼓励开发者的参与和贡献,期待你的加入。

License

项目基于 MIT 协议, 详细请参考 LICENSE

You might also like...
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection.

LightLog Introduction LightLog is an open source deep learning based lightweight log analysis tool for log anomaly detection. Function description [BG

A simple CLI to convert snapshots into EAVT log, and EAVT log into SCD.

EAVT helper CLI Simple CLI to convert snapshots into eavt log, and eavt log into slowly changing dimensions Usage Installation Snapshot to EAVT log EA

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

GoAccess What is it? GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal on *nix systems or through y

Yet another Django audit log app, hopefully the simplest one.

django-easy-audit Yet another Django audit log app, hopefully the easiest one. This app allows you to keep track of every action taken by your users.

The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common functions that add additional logs

pandas-log The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common funct

Log and View requests made on Django
Log and View requests made on Django

Django Request Viewer Log and view requests made on your Django App Introduction Recently, @ichtrojan and @toniastro released horus, a request logger

With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials.
With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials.

Django Hijack With Django Hijack, admins can log in and work on behalf of other users without having to know their credentials. Docs See http://django

"Log in as user" for the Django admin.

django-loginas About "Login as user" for the Django admin. loginas supports Python 3 only, as of version 0.4. If you're on 2, use 0.3.6. Installing dj

Django app that enables staff to log in as other users using their own credentials.
Django app that enables staff to log in as other users using their own credentials.

Impostor Impostor is a Django application which allows staff members to login as a different user by using their own username and password. Login Logg

Inject an ID into every log message from a Django request. ASGI compatible, integrates with Sentry, and works with Celery

Django GUID Now with ASGI support! Django GUID attaches a unique correlation ID/request ID to all your log outputs for every request. In other words,

Paper list of log-based anomaly detection

Paper list of log-based anomaly detection

A python application to log QSOs directly to QRZ.com from the command line
A python application to log QSOs directly to QRZ.com from the command line

qrzlogger This script is a QRZ.com command line QSO logger. It does the following: asks the user for a call sign displays available call sign info pul

[Singing Log] Let your program learn to sing!

[Singing Log] Let your program learn to sing! You must have thought this was changelog when you saw the English title, but it's not, it's chànggēlog. What it does is allow your program to print logs and sing at the same time!

Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram.

telegram-logger A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram. Requirements Python 3.6

Summarize LSF job properties by parsing log files.
Summarize LSF job properties by parsing log files.

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Comments
  • 分隔符过滤匹配带有空格数据无法采集到数据

    分隔符过滤匹配带有空格数据无法采集到数据

    数据接入配置日志内容过滤规则,采用分隔符过滤,匹配字段前后带有空格数据无法采集到日志数据:

    • 采集接入配置:采用","分割,匹配第二列包含“ after callback”(字符串前面带有空格),编辑查看详情已将空格自动删除; image image

    • 问题原因

    1. django 序列化器默认将空格删除(默认trim_whitespace=True); image

    2. 下发的采集配置文件已将空格取消,但采集器匹配内容的时候分割日志后是带空格去匹配关键字的,导致匹配不到内容; image image

    • 解决方案
    1. 方案一:修改序列器,增加trim_whitespace=False;
    2. 方案二:修改采集器匹配逻辑,分割完去除前后空格去匹配关键字(个人建议采用该方案,因为空格内容对用户无意义); image
    kind/bug 
    opened by Eli-ZhangLu 3
  • 采集项数据量过大,索引集管理新增采集类型索引集报502错误

    采集项数据量过大,索引集管理新增采集类型索引集报502错误

    1. 问题描述 当业务下“数据接入”建立的采集项过多时,在“索引集管理”-“新建”-“采集接入”-“新增索引” 报502错误;

    2. 版本信息 4.2.653

    3. 报错截图 image

    4. 问题原因 metadata_get_result_table_storage 接口设计不合理,为get请求,参数result_table_list支持传list,当达到一定大小后接口返回502错误

    kind/bug 
    opened by Eli-ZhangLu 1
  • 字段提取功能问题

    字段提取功能问题

    版本信息

    • saas&后台版本:4.2.653

    json提取方式问题

    1. 问题描述:json清洗后带特殊字符的字段重命名后前端校验不通过;
    2. 理想结果:json清洗后带特殊字符的字段重命名,只需要校验重命名字段格式,而不需要校验原始字段;
    3. 问题截图: image
    4. 日志原文:
    {
        "@timestamp": "2021-11-05T22:10:00.000Z",
        "@version": "1",
        "program": "/usr/sbin/cron",
        "host": "127.0.0.1",
        "timestamp": "Nov  6 06:10:00",
        "message": "(root) CMD (   /usr/libexec/atrun)",
        "priority": 78,
        "pid": "39172",
        "facility_label": "clock",
        "logsource": "ELI-PC",
        "severity_label": "Informational",
        "severity": 6,
        "facility": 9
    }
    

    时间字段清洗问题

    1. 问题描述:“2021-11-05T22:10:00.000Z” 时间数据用“YYYY-MM-DDTHH:mm:ss.SSSZ”格式清洗失败;
    2. 理想结果:时间字段和数据格式匹配应该清洗成功;
    3. 问题截图: image
    4. 日志原文:
    {
        "@timestamp": "2021-11-05T22:10:00.000Z",
        "@version": "1",
        "program": "/usr/sbin/cron",
        "host": "127.0.0.1",
        "timestamp": "Nov  6 06:10:00",
        "message": "(root) CMD (   /usr/libexec/atrun)",
        "priority": 78,
        "pid": "39172",
        "facility_label": "clock",
        "logsource": "ELI-PC",
        "severity_label": "Informational",
        "severity": 6,
        "facility": 9
    }
    
    kind/bug uat 
    opened by Eli-ZhangLu 1
  • Trace-Log-Metric关联方案

    Trace-Log-Metric关联方案

    指标关联Trace

    exemplar机制

    prometheus

    prometheus主要是采用 exemplars 的机制在 metrics 中带上额外的信息。通过metrics的接口可以同事暴露exemplar https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars-1

    # 后面的内容就是exemplar
    # lable 采样值 采样时间
    foo_bucket{le="0.1"} 8 # {} 0.054
    foo_bucket{le="1"} 11 # {trace_id="KOO5S4vxi0o"} 0.67
    foo_bucket{le="10"} 17 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
    

    注入方式

    c := GetPlayURLTotal.WithLabelValues(
                strconv.FormatInt(int64(callerType), 10),
                strconv.FormatInt(int64(device.GetOs()), 10),
                strconv.FormatInt(int64(device.GetNetwork()), 10),
                videoFormat,
    )
    sp := trace.SpanFromContext(ctx).SpanContext()
    if sp.IsSampled() { // 可以继续增加其他条件使得exemplar样本更加典型
        c.(prometheus.ExemplarAdder).AddWithExemplar(1, prometheus.Labels{
              "traceID": sp.TraceID().String(),
        }) // 如果是histogram类型的则类型断言为prometheus.ExemplarObserver
    } else {
        c.Inc()
    }
    

    otlp

    otlp在协议中有Exemplar字段 可以在指标上报时将被采样的span跟指标关联.otlp-SDK是自动进行注入的,因为trace-log-metric 三者共享同样的otlp-context,所以可以不必要进行手工关联

    // A representation of an exemplar, which is a sample input measurement.
    
    // Exemplars also hold information about the environment when the measurement
    
    // was recorded, for example the span and trace ID of the active span when the
    
    // exemplar was recorded.
    
    message Exemplar {
    
    // The set of key/value pairs that were filtered out by the aggregator, but
    
    // recorded alongside the original measurement. Only key/value pairs that were
    
    // filtered out by the aggregator should be included
    
    repeated opentelemetry.proto.common.v1.KeyValue filtered_attributes = 7;
    
    // Labels is deprecated and will be removed soon.
    
    // 1. Old senders and receivers that are not aware of this change will
    
    // continue using the `filtered_labels` field.
    
    // 2. New senders, which are aware of this change MUST send only
    
    // `filtered_attributes`.
    
    // 3. New receivers, which are aware of this change MUST convert this into
    
    // `filtered_labels` by simply converting all int64 values into float.
    
    //
    
    // This field will be removed in ~3 months, on July 1, 2021.
    
    repeated opentelemetry.proto.common.v1.StringKeyValue filtered_labels = 1 [deprecated = true];
    
    // time_unix_nano is the exact time when this exemplar was recorded
    
    //
    
    // Value is UNIX Epoch time in nanoseconds since 00:00:00 UTC on 1 January
    
    // 1970.
    
    fixed64 time_unix_nano = 2;
    
    // The value of the measurement that was recorded. An exemplar is
    
    // considered invalid when one of the recognized value fields is not present
    
    // inside this oneof.
    
    oneof value {
    
    double as_double = 3;
    
    sfixed64 as_int = 6;
    
    }
    
    // (Optional) Span ID of the exemplar trace.
    
    // span_id may be missing if the measurement is not recorded inside a trace
    
    // or if the trace is not sampled.
    
    bytes span_id = 4;
    
    // (Optional) Trace ID of the exemplar trace.
    
    // trace_id may be missing if the measurement is not recorded inside a trace
    
    // or if the trace is not sampled.
    
    bytes trace_id = 5;
    
    }
    

    prometheus存储方式(tjg使用该方式)

    https://github.com/prometheus/prometheus/pull/6635/files prometheus 实现了一种环形连续内存的结构来存储 exemplar,并实现了对应的查询接口

    $ curl -g 'http://localhost:9090/api/v1/query_exemplars?query=test_exemplar_metric_total&start=2020-09-14T15:22:25.479Z&end=020-09-14T15:23:25.479Z'
    {
        "status": "success",
        "data": [
            {
                "seriesLabels": {
                    "__name__": "test_exemplar_metric_total",
                    "instance": "localhost:8090",
                    "job": "prometheus",
                    "service": "bar"
                },
                "exemplars": [
                    {
                        "labels": {
                            "traceID": "EpTxMJ40fUus7aGY"
                        },
                        "value": "6",
                        "timestamp": 1600096945.479,
                    }
                ]
            },
            {
                "seriesLabels": {
                    "__name__": "test_exemplar_metric_total",
                    "instance": "localhost:8090",
                    "job": "prometheus",
                    "service": "foo"
                },
                "exemplars": [
                    {
                        "labels": {
                            "traceID": "Olp9XHlq763ccsfa"
                        },
                        "value": "19",
                        "timestamp": 1600096955.479,
                    },
                    {
                        "labels": {
                            "traceID": "hCtjygkIHwAN9vs4"
                        },
                        "value": "20",
                        "timestamp": 1600096965.489,
                    },
                ]
            }
        ]
    }
    

    image

    日志关联Trace

    日志关联Trace 比较简单 只要在打印日志的时候获取到链路的TraceId和spanId 就可以关联Trace和单条日志了

    Log
    timestamp= TraceId=xxxx SpanId=xxxxx
    Json
    {"trace_id": "xxx", "span_id": "xxx", "log": "xxxx"}
    

    最终清洗入库并标记trace_id和span_id即可实现联动 image

    otlp-SDK 最终可以实现默认关联因为共享Context

    监控存储exemplar

    由于influxdb目前不支持exemplar入库,所以基于现有存储结构监控可以使用ES进行exemplar存储,避免高基线问题 修改如下

    • 相关prometheus的数据解析需要支持exemplar类型的解析并上报
    • transfer需要支持exemplar数据入库到ES
    • saas支持exemplar数据的查询
    Technical solution 
    opened by zzhutianyu 0
Releases(V4.3.5)
  • V4.3.5(Oct 4, 2022)

    What's Changed

    新增的功能如下:

    • 新增支持蓝鲸BCS容器日志采集
    • 新增关联跳转蓝鲸监控metric、log、trace

    本更新还修复了如下的内容:

    • 修复若干问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.4-334(Aug 25, 2022)

    What's Changed

    新增的功能如下:

    • 变更日志采集下发流程
    • 集群可见范围多样化
    • 新增healthz自身健康检测
    • SLI指标上报

    本更新还修复了如下的内容:

    • 为了解决跨天分裂索引上下文无法定位到的问题
    • 修复grafana 添加variable值为主机时缺少innerip的bug
    • 第三方es中time_field获取失败
    • 修复若干问题

    Full Changelog: https://github.com/TencentBlueKing/bk-log/compare/V4.3.4-rc300...V4.3.4-334

    Source code(tar.gz)
    Source code(zip)
  • V4.3.4-rc300(Jul 1, 2022)

    -【新增】变更日志采集下发流程 -【新增】集群可见范围多样化 -【新增】新增healthz自身健康检测 -【新增】SLI指标上报 -【修复】若干问题

    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-299(Jun 14, 2022)

    What's Changed

    • minor: 第三方es中time_field获取失败 by @liuwenping in https://github.com/TencentBlueKing/bk-log/pull/1090
    • merge: merge from master 为了解决跨天分裂索引上下文无法定位到的问题 by @EvildoerXiaoyy in https://github.com/TencentBlueKing/bk-log/pull/1088
    • bugfix: 修复grafana 添加variable值为主机时缺少innerip的bug by @kiritoscs in https://github.com/TencentBlueKing/bk-log/pull/1098
    • minor: 添加版本文档 by @kiritoscs in https://github.com/TencentBlueKing/bk-log/pull/1104

    Full Changelog: https://github.com/TencentBlueKing/bk-log/compare/V4.3.1-275...V4.3.1-299

    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-275(Apr 29, 2022)

  • V4.3.3-272(Apr 19, 2022)

    修复

    • 修复title不符合2.0的规范问题
    • 修复问题反馈的跳转链接为空白页
    • 修复登录态续期小窗不消失的问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.1-273(Apr 19, 2022)

  • V4.3.1-267(Apr 15, 2022)

  • V4.3.1-257(Apr 12, 2022)

    修复

    • 修复日志提取成功后没有进行国际化转换的问题
    • 修复仪表盘首页国际化暂未适配的问题
    • 修复跳转为正确官网文档地址的问题
    Source code(tar.gz)
    Source code(zip)
  • V4.3.3-rc240(Mar 23, 2022)

  • V4.3.1-238(Mar 17, 2022)

  • V4.3.1-230(Mar 9, 2022)

  • V4.3.1-227(Mar 9, 2022)

  • V4.3.1.132(Nov 4, 2021)

Summarize LSF job properties by parsing log files.

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Kim 4 Jan 9, 2022
Splunk Add-On to collect audit log events from Github Enterprise Cloud

GitHub Enterprise Audit Log Monitoring Splunk modular input plugin to fetch the enterprise audit log from GitHub Enterprise Support for modular inputs

Splunk GitHub 12 Aug 18, 2022
Greppin' Logs: Leveling Up Log Analysis

This repo contains sample code and example datasets from Jon Stewart and Noah Rubin's presentation at the 2021 SANS DFIR Summit titled Greppin' Logs. The talk was centered around the idea that Forensics is Data Engineering and Data Science, and should be approached as such. Jon and Noah focused on the core (Unix) command line tools useful to anyone analyzing datasets from a terminal, purpose-built tools for handling structured tabular and JSON data, Stroz Friedberg's open source multipattern search tool Lightgrep, and scaling with AWS.

Stroz Friedberg 20 Sep 14, 2022
Monitor and log Network and Disks statistics in MegaBytes per second.

iometrics Monitor and log Network and Disks statistics in MegaBytes per second. Install pip install iometrics Usage Pytorch-lightning integration from

Leo Gallucci 17 May 3, 2022
Translating symbolicated Apple JSON format crash log into our old friends :)

CrashTranslation Translating symbolicated Apple JSON format crash log into our old friends :) Usage python3 translation.py -i {input_sybolicated_json_

Kam-To 11 May 16, 2022
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 4, 2021
Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

Log processor for nginx or apache that extracts user and user sessions and calculates other types of useful data for bot detection or traffic analysis

David Puerta Martín 1 Nov 11, 2021
A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

Alpwuf 1 Nov 16, 2021
This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

Rithin Lehan 2 Nov 25, 2021
Fuzzy-logger - Fuzzy project is here Log all your pc's actions Simple and free to use Security of datas !

Fuzzy-logger - ➡️⭐ Fuzzy ⭐ project is here ! ➡️ Log all your pc's actions ! ➡️ Simple and free to use ➡️ Security of datas !

natrix_dev 2 Oct 2, 2022