Extract data from ThousandEyes REST API and visualize it on your customized Grafana Dashboard.

Overview

ThousandEyes Grafana Dashboard

Extract data from the ThousandEyes REST API and visualize it on your customized Grafana Dashboard. Deploy Grafana, InfluxDBv2 and the Python connector script within a few minutes.

Supported Functions

  • Get historic data (custom time range) and pull new data (set your own interval) from your ThousandEyes tests.
  • Grafana, a sample Dashboard-template and InfluxDBv2 are already pre-configured
  • Collected data is persistent! It will remain untouched even after the containers will be restarted.
  • Currently Supported Tests:
    • (Web) Page load
    • (Web) HTTP server
    • (Network) End-to-End metrics
    • (Network) Path visualization

Visuals

Prepare: Create Label (optional), clone git-repo, change settings in config.py

Run: Start containers, copy and insert testId into Grafana

Dashboard-Template Screenshots

Installation

Prerequisites

  • git, Docker/Docker-compose installed

Installation Steps

  1. Clone repository
git clone https://github.com/flopach/thousandeyes-grafana-dashboard.git
  1. Change your desired settings in the docker/py_connector/config.py file.

  2. Start all containers from the docker/ directory. This may take some minutes.

docker-compose up

More useful commands:

  • Run as daemon mode: docker-compose up -d
  • Stops containers: docker-compose down
  • Build again the containers (when you changed the Python scripts): docker-compose build

Configuration

Configure: config.py

Simply open docker/py_connector/config.py in any text-editor to set your desired configuration.

  • Create OAuth Bearer Token: Account Settings > Users and Roles >> User API Tokens
  • 2 data selection options:
    • Default: py_connector will add ALL HTTP-Server and Page-Load tests
    • Option: py_connector will add ONLY tests with the defined label
  • time window
  • pull interval

Example: When you've created a ThousandEyes page-load test (see below), you will receive data from 4 views:

Configure: Dashboards

Login to Grafana dashboard and configure your dashboard modules by inserting the ThousandEyes testId. Additionally, use the query editor from the InfluxDB UI as a help to create and copy the Flux language syntax snippet.

Login Grafana - http://localhost:3000

  • username: admin
  • password: admin123

Login InfluxDB - http://localhost:8086

  • username: admin
  • password: admin123

Configure: docker-compose.yml (optional)

If you are not familiar with Docker compose, go ahead to check the documentation. Basically, 4 containers will be spun up:

  • InfluxDBv2: data will be stored outside of the container
  • InfluxDBv2 CLI setup instance: Will setup the other InfluxDBv2 container if it is not setup yet
  • Grafana Dashboard: configuration settings will be stored outside of the container
  • Py connector: Python connector scripts (stored in the py_connector folder)

Good to knows

- InfluxDB Setup Error

When you have already started docker-compose and are re-starting the containers, this error is normal:

influxdb_setup_1  | Error: instance at "http://influxdb:8086" has already been setup
influxdb_setup_1  | See 'influx setup -h' for help
influxdb_setup_1 exited with code 1

- Changing the settings in config.py

When you have already started docker-compose once and would like to change the settings, you need to re-build the py_connector container:

docker-compose build

Versioning

1.0 (Sep 2021) - Initial version

License

This project is licensed under the Cisco Samplel Code License 1.1 - see the LICENSE.md file for details

Further Links

You might also like...
Visualize data of Vietnam's regions with interactive maps.
Visualize data of Vietnam's regions with interactive maps.

Plotting Vietnam Development Map This is my personal project that I use plotly to analyse and visualize data of Vietnam's regions with interactive map

Eulera Dashboard is an easy and intuitive way to get a quick feel of what’s happening on the world’s market.
Eulera Dashboard is an easy and intuitive way to get a quick feel of what’s happening on the world’s market.

an easy and intuitive way to get a quick feel of what’s happening on the world’s market ! Eulera dashboard is a tool allows you to monitor historical

Visualize and compare datasets, target values and associations, with one line of code.
Visualize and compare datasets, target values and associations, with one line of code.

In-depth EDA (target analysis, comparison, feature analysis, correlation) in two lines of code! Sweetviz is an open-source Python library that generat

Visualize and compare datasets, target values and associations, with one line of code.
Visualize and compare datasets, target values and associations, with one line of code.

In-depth EDA (target analysis, comparison, feature analysis, correlation) in two lines of code! Sweetviz is an open-source Python library that generat

:bowtie: Create a dashboard with python!
:bowtie: Create a dashboard with python!

Installation | Documentation | Gitter Chat | Google Group Bowtie Introduction Bowtie is a library for writing dashboards in Python. No need to know we

Streamlit dashboard examples - Twitter cashtags, StockTwits, WSB, Charts, SQL Pattern Scanner

streamlit-dashboards Streamlit dashboard examples - Twitter cashtags, StockTwits, WSB, Charts, SQL Pattern Scanner Tutorial Video https://ww

An interactive dashboard built with python that enables you to visualise how rent prices differ across Sweden.
An interactive dashboard built with python that enables you to visualise how rent prices differ across Sweden.

sweden-rent-dashboard An interactive dashboard built with python that enables you to visualise how rent prices differ across Sweden. The dashboard/web

A dashboard built using Plotly-Dash for interactive visualization of Dex-connected individuals across the country.
A dashboard built using Plotly-Dash for interactive visualization of Dex-connected individuals across the country.

Dashboard For The DexConnect Platform of Dexterity Global Working prototype submission for internship at Dexterity Global Group. Dashboard for real ti

YOPO is an interactive dashboard which generates various standard plots.
YOPO is an interactive dashboard which generates various standard plots.

YOPO is an interactive dashboard which generates various standard plots.you can create various graphs and charts with a click of a button. This tool uses Dash and Flask in backend.

Comments
  • Unable to start docker containers

    Unable to start docker containers

    Hi. I have installed an Ubuntu 20.04 and installed git and docker-compose using apt. I have then configured the config.py file.

    But when I try to start the docker containers I get the following error:

    rael@grafana:~/thousandeyes-grafana-dashboard/docker$ sudo docker-compose up -d
    WARNING: The PWD variable is not set. Defaulting to a blank string.
    WARNING: The UID variable is not set. Defaulting to a blank string.
    WARNING: The GID variable is not set. Defaulting to a blank string.
    ERROR: build path /py_connector either does not exist, is not accessible, or is not a valid URL.
    rael@grafana:~/thousandeyes-grafana-dashboard/docker$ ls -lah
    total 20K
    drwxrwxr-x 4 rael rael 4.0K May 17 11:16 .
    drwxrwxr-x 5 rael rael 4.0K May 17 11:13 ..
    -rw-rw-r-- 1 rael rael 1.4K May 17 11:13 docker-compose.yml
    drwxrwxr-x 4 rael rael 4.0K May 17 11:13 provisioning
    drwxrwxr-x 2 rael rael 4.0K May 17 11:24 py_connector
    rael@grafana:~/thousandeyes-grafana-dashboard/docker$
    

    Any help would be appreciated.

    opened by fatman00 2
  • Add variables to Grafana Dashboard for TE Tests.

    Add variables to Grafana Dashboard for TE Tests.

    Updated the Grafana json file to use variables for each of the test ID's being pulled from TE via API. Dashboard now provides the test id's as variables on the dashboard so that each test can be selected and jumped between without the need for individual dashboards for each test ID.

    opened by martynrees 1
  • Adding agent to agent/server tests and Grafana dashboard with variable TestName

    Adding agent to agent/server tests and Grafana dashboard with variable TestName

    When running agent to agent/server, TE supports net metrics and path visibility metrics and the page load metrics do not apply. I have added these types of tests to config.py and main.py. I have also created a separate dashboard for these types of tests. I found using the test name easier to work with in a Grafana dashboard. The new dashboard defines a variable "testName" and is used in the Influx queries along with agent name to make the graphs easier to follow for these types of TE tests. The variable is also multi-value so you can compare several agent to agent/server in a single panel.

    opened by Opiekelly 0
  • python API call can't work with

    python API call can't work with "grafana"

    My customer is following your bellow published document and faces the bellow issue.

    https://urldefense.com/v3/https:/developer.cisco.com/codeexchange/github/repo/flopach/thousandeyes-grafana-dashboard/;!!LSAcJDlP!kkbZmbskK32do1AL7yhpZCLY7Z2O8_DW5zerXBr8meP9l59_IkFk_DGBRKzMvCVXGFo$

    If we use docker-compose or podman-compose to run docker-compose.yml, they will get error :

    [root@gbl25028897 1000eyes]# podman-compose up ['podman', '--version', ''] using podman version: 3.0.1 ** excluding: set() ['podman', 'network', 'exists', '1000eyes_default'] Error: unrecognized command podman network exists Try 'podman network --help' for more information. ['podman', 'network', 'create', '--label', 'io.podman.compose.project=1000eyes', '--label', 'com.docker.compose.project=1000eyes', '1000eyes_default'] ['podman', 'network', 'exists', '1000eyes_default'] Error: unrecognized command podman network exists Try 'podman network --help' for more information. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 563, in assert_cnt_nets try: compose.podman.output([], "network", ["exists", net_name]) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output return subprocess.check_output(cmd_ls) File "/usr/lib64/python3.9/subprocess.py", line 424, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib64/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['podman', 'network', 'exists', '1000eyes_default']' returned non-zero exit status 125.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/usr/local/bin/podman-compose", line 8, in sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1775, in main podman_compose.run() File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1024, in run cmd(self, args) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1248, in wrapped return func(*args, **kw) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1415, in compose_up podman_args = container_to_args(compose, cnt, detached=args.detach) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 645, in container_to_args assert_cnt_nets(compose, cnt) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 580, in assert_cnt_nets compose.podman.output([], "network", ["exists", net_name]) File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output return subprocess.check_output(cmd_ls) File "/usr/lib64/python3.9/subprocess.py", line 424, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib64/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['podman', 'network', 'exists', '1000eyes_default']' returned non-zero exit status 125. [root@gbl25028897 1000eyes]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@gbl25028897 1000eyes]# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@gbl25028897 1000eyes]# podman network rm 1000eyes_default 1000eyes_default

    [root@gbl25028897 1000eyes]# docker-compose up /usr/local/lib/python3.9/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, WARNING: The UID variable is not set. Defaulting to a blank string. WARNING: The GID variable is not set. Defaulting to a blank string. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib64/python3.9/http/client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1012, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 952, in send self.connect() File "/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py", line 30, in connect sock.connect(self.unix_socket) FileNotFoundError: [Errno 2] No such file or directory

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/lib64/python3.9/http/client.py", line 1257, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1303, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1252, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1012, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 952, in send self.connect() File "/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py", line 30, in connect sock.connect(self.unix_socket) urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 214, in _retrieve_server_version return self.version(api_version=False)["ApiVersion"] File "/usr/local/lib/python3.9/site-packages/docker/api/daemon.py", line 181, in version return self._result(self._get(url), json=True) File "/usr/local/lib/python3.9/site-packages/docker/utils/decorators.py", line 46, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 237, in _get return self.get(url, **self._set_request_timeout(kwargs)) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in get return self.request('GET', url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "/usr/local/bin/docker-compose", line 8, in sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/compose/cli/main.py", line 81, in main command_func() File "/usr/local/lib/python3.9/site-packages/compose/cli/main.py", line 200, in perform_command project = project_from_options('.', options) File "/usr/local/lib/python3.9/site-packages/compose/cli/command.py", line 60, in project_from_options return get_project( File "/usr/local/lib/python3.9/site-packages/compose/cli/command.py", line 152, in get_project client = get_client( File "/usr/local/lib/python3.9/site-packages/compose/cli/docker_client.py", line 41, in get_client client = docker_client( File "/usr/local/lib/python3.9/site-packages/compose/cli/docker_client.py", line 170, in docker_client client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs) File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 197, in init self._version = self._retrieve_server_version() File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 221, in _retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) [root@gbl25028897 1000eyes]# docker-compose --v /usr/local/lib/python3.9/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated "class": algorithms.Blowfish, Define and run multi-container applications with Docker.

    opened by mferdous-te 1
Owner
Flo Pachinger
Developer Advocate at Cisco DevNet
Flo Pachinger
Smarthome Dashboard with Grafana & InfluxDB

Smarthome Dashboard with Grafana & InfluxDB This is a complete overhaul of my Raspberry Dashboard done with Flask. I switched from sqlite to InfluxDB

null 6 Oct 20, 2022
Extract and visualize information from Gurobi log files

GRBlogtools Extract information from Gurobi log files and generate pandas DataFrames or Excel worksheets for further processing. Also includes a wrapp

Gurobi Optimization 56 Nov 17, 2022
A customized interface for single cell track visualisation based on pcnaDeep and napari.

pcnaDeep-napari A customized interface for single cell track visualisation based on pcnaDeep and napari. ?? Under construction You can get test image

ChanLab 2 Nov 7, 2021
Visualize your pandas data with one-line code

PandasEcharts 简介 基于pandas和pyecharts的可视化工具 安装 pip 安装 $ pip install pandasecharts 源码安装 $ git clone https://github.com/gamersover/pandasecharts $ cd pand

陈华杰 2 Apr 13, 2022
2021 grafana arbitrary file read

2021_grafana_arbitrary_file_read base on pocsuite3 try 40 default plugins of grafana alertlist annolist barchart cloudwatch dashlist elasticsearch gra

ATpiu 5 Nov 9, 2022
Backend app for visualizing CANedge log files in Grafana (directly from local disk or S3)

CANedge Grafana Backend - Visualize CAN/LIN Data in Dashboards This project enables easy dashboard visualization of log files from the CANedge CAN/LIN

null 13 Dec 15, 2022
An interactive dashboard for visualisation, integration and classification of data using Active Learning.

AstronomicAL An interactive dashboard for visualisation, integration and classification of data using Active Learning. AstronomicAL is a human-in-the-

null 45 Nov 28, 2022
Data aggregated from the reports found at the MCPS COVID Dashboard into a set of visualizations.

Montgomery County Public Schools COVID-19 Visualizer Contents About this project Data Support this project About this project Data All data we use can

James 3 Jan 19, 2022
Visualize the bitcoin blockchain from your local node

Project Overview A new feature in Bitcoin Core 0.20 allows users to dump the state of the blockchain (the UTXO set) using the command dumptxoutset. I'

null 18 Sep 11, 2022
Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database

SpiderFoot Neo4j Tools Import, visualize, and analyze SpiderFoot OSINT data in Neo4j, a graph database Step 1: Installation NOTE: This installs the sf

Black Lantern Security 42 Dec 26, 2022