This repository has automation content to test Arista devices.

Overview

GitHub test workflow

Network tests automation

About this repository

This repository has automation content to test Arista devices.

To run tests programatically on your EOS devices, once you are done with the requirements described below, you simply need to indicate your devices name in a text file and to indicate the tests you would like to run in a YAML file.

This repository has also content to collect commands output on devices, to clear counters on devices, and to test the devices reachability.

This repository uses Python scripts and eAPI (EOS API). You can find examples of EOS automation with eAPI in this repository.

Requirements

Requirements on your laptop

Python 3 (at least 3.3) and some packages that are not part of the standard Python library.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic
python -V
Python 3.9.6

The required packages are in the requirements.txt file. Clone the repository and install the requirements:

pip install -r requirements.txt

Verify

pip list | grep 'jsonrpclib-pelix\|PyYAML\|colorama\|prettytable'
colorama                   0.3.7
jsonrpclib-pelix           0.4.2
prettytable                2.5.0
PyYAML                     5.4.1

Requirements on the switches

  • Enable eAPI:
switch1#configure
switch1(config)#management api http-commands
switch1(config-mgmt-api-http-cmds)#protocol https port 443
switch1(config-mgmt-api-http-cmds)#no shutdown
switch1(config-mgmt-api-http-cmds)#vrf MGMT
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#no shutdown
switch1(config-mgmt-api-http-cmds-vrf-MGMT)#exit
switch1(config-mgmt-api-http-cmds)#exit
switch1(config)#
switch1#sh run int ma1
interface Management1
   description oob_management
   vrf MGMT
   ip address 10.73.1.105/24
switch1#

Now the swicth accepts HTTPS requests on port 443 in the MGMT VRF containing a list of CLI commands.

  • Verify:
switch1(config)#sh management http-server
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   VRF        Server Status      Enabled Services
---------- --------------------- ----------------
   MGMT       HTTPS: port 443    http-commands

switch1>show management api http-commands
Enabled:            Yes
HTTPS server:       running, set to use port 443
HTTP server:        shutdown, set to use port 80
Local HTTP server:  shutdown, no authentication, set to use port 8080
Unix Socket server: shutdown, no authentication
VRFs:               MGMT
Hits:               83
Last hit:           2631 seconds ago
Bytes in:           11348
Bytes out:          335951
Requests:           53
Commands:           64
Duration:           9.242 seconds
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   User          Requests       Bytes in       Bytes out    Last hit
------------- -------------- -------------- --------------- ------------------
   arista        2              305            1235         639908 seconds ago
   ansible       51             11043          334716       2631 seconds ago

URLs
-------------------------------------
Management1 : https://10.73.1.105:443

switch1>

Quick test

Run this python script to validate the requirements and the device reachability. Use your device credentials and IP address.

from jsonrpclib import Server
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
username = "arista"
password = "arista"
ip = "10.100.164.145"
url = "https://" + username + ":" + password + "@" + ip + "/command-api"
switch = Server(url)
result=switch.runCmds(1,['show version'], 'text')
print(result[0]['output'])

List of available tests

The tests are defined in the python module functions.py in the python package tests_eos. Each function returns True or False (or None when it can not run properly).

The documentation directory has the tests documentation:

We indicate the tests we would like to run in a YAML file. Some tests require an input. In that case we provide it using the same YAML file. Here's an example tests.yaml.

Repository usage

  • Clone this repository.
  • Install the requirements (see above)

To test devices reachability

vi devices.txt
./check-devices-reachability.py --help
./check-devices-reachability.py -i devices.txt -u username

To run tests on devices

  • Update the devices inventory devices.txt with the devices IP address or hostnames.
  • Update the file tests.yaml to indicate the tests you would like to run. Some tests require an input. In that case, provide it using the same YAML file.
  • Run the python script check-devices.py.
  • Check the result in the output file.
vi devices.txt
vi tests.yaml
./check-devices.py --help
./check-devices.py -i devices.txt -t tests.yaml -o output.txt -u username
cat output.txt

To collect commands output from EOS devices

  • Update the devices inventory devices.txt with your devices IP address or hostnames.
  • Update the EOS commands list eos-commands.yaml you would like to collect from the devices in text or JSON format.
  • Run the python script collect-eos-commands.py.
  • Check the output in the output directory.
vi devices-list.text
vi eos-commands.yaml
./collect-eos-commands.py --help
./collect-eos-commands.py -i devices.txt -c eos-commands.yaml -o outdir -u username
ls outdir

To clear counters on EOS devices

vi devices-list.text
./clear_counters.py --help
./clear_counters.py -i devices.txt -u username

Demo

$ ./check-devices.py -u ansible -i devices.txt -o output.txt -t tests.yaml
Device password:
Enable password (if any):
Can not connect to device 2.2.2.2
Running tests on device 10.73.1.101 ...
Running tests on device 10.73.1.102 ...
Running tests on device 10.73.1.106 ...
Test results are saved on output.txt
$ cat output.txt
Fri Jan 28 09:48:54 2022
devices inventory file was devices.txt
devices username was ansible
list of unreachable devices is
2.2.2.2
tests file was tests.yaml

***** Results *****

+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  01.01  |  01.02  |  01.03  |  01.04  |  02.01  |  02.02  |  02.03  |  02.04  |  02.05  |  02.06  |  02.07  |  02.08  |  03.01  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.102  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |
|  10.73.1.106  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |   Skip  |   Pass  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Skip  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  03.02  |  03.03  |  03.04  |  03.05  |  04.01  |  04.02  |  05.01  |  05.02  |  06.01  |  07.01  |  08.01  |  08.02  |  08.03  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Fail  |   Pass  |   Fail  |   Skip  |   Pass  |   Fail  |   Skip  |   Skip  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  08.04  |  08.05  |  09.01  |  09.02  |  09.03  |  09.04  |  10.01  |  11.01  |  11.02  |  12.01  |  13.01  |  14.01  |  14.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.102  |   Pass  |   Pass  |   Skip  |   Skip  |   Skip  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |   Pass  |   Pass  |   Pass  |
|  10.73.1.106  |   Pass  |   Pass  |   Fail  |   Pass  |   Fail  |   Fail  |   Fail  |   Pass  |   Fail  |   Pass  |   Pass  |   Pass  |   Pass  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|    devices    |  15.01  |  16.01  |  16.02  |  16.03  |  16.04  |  16.05  |  16.06  |  16.07  |  17.01  |  18.01  |  18.02  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|  10.73.1.101  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Pass  |   Fail  |
|  10.73.1.102  |   Pass  |   Fail  |   Pass  |   Fail  |   Skip  |   Fail  |   Skip  |   Fail  |   Fail  |   Skip  |   Pass  |
|  10.73.1.106  |   Pass  |   Fail  |   Pass  |   Pass  |   Skip  |   Pass  |   Skip  |   Fail  |   Fail  |   Skip  |   Fail  |
+---------------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+

***** Tests *****
01.01    {"name": "verify_eos_version", "versions": ["4.25.4M", "4.26.1F"]}
01.02    {"name": "verify_terminattr_version", "versions": ["v1.13.6", "v1.8.0"]}
01.03    "verify_eos_extensions"
01.04    "verify_field_notice_44_resolution"
02.01    {"name": "verify_uptime", "min": 86400}
02.02    "verify_reload_cause"
02.03    "verify_coredump"
02.04    "verify_agent_logs"
02.05    "verify_syslog"
02.06    "verify_cpu_utilization"
02.07    "verify_memory_utilization"
02.08    "verify_filesystem_utilization"
03.01    {"name": "verify_transceivers_manufacturers", "manufacturers": ["Not Present", "Arista Networks", "Arastra, Inc."]}
03.02    "verify_system_temperature"
03.03    "verify_transceiver_temperature"
03.04    "verify_environment_cooling"
03.05    "verify_environment_power"
04.01    "verify_zerotouch"
04.02    "verify_running_config_diffs"
05.01    {"name": "verify_unified_forwarding_table_mode", "mode": 3}
05.02    {"name": "verify_tcam_profile", "profile": "vxlan-routing"}
06.01    "verify_adverse_drops"
07.01    "verify_ntp"
08.01    "verify_interface_utilization"
08.02    "verify_interface_errors"
08.03    "verify_interface_discards"
08.04    "verify_interface_errdisabled"
08.05    {"name": "verify_interfaces_status", "minimum": 4}
09.01    "verify_portchannels"
09.02    "verify_illegal_lacp"
09.03    "verify_mlag_status"
09.04    "verify_mlag_interfaces"
10.01    {"name": "verify_loopback_count", "number": 3}
11.01    "verify_vxlan"
11.02    "verify_vxlan_config_sanity"
12.01    "verify_svi"
13.01    "verify_spanning_tree_blocked_ports"
14.01    {"name": "verify_routing_protocol_model", "model": "multi-agent"}
14.02    {"name": "verify_routing_table_size", "min": 2, "max": 20}
15.01    "verify_bfd"
16.01    "verify_bgp_ipv4_unicast_state"
16.02    "verify_bgp_ipv6_unicast_state"
16.03    "verify_bgp_evpn_state"
16.04    "verify_bgp_rtc_state"
16.05    {"name": "verify_bgp_evpn_count", "number": 2}
16.06    {"name": "verify_bgp_rtc_count", "number": 2}
16.07    {"name": "verify_bgp_ipv4_unicast_count", "number": 2}
17.01    {"name": "verify_ospf", "number": 3}
18.01    {"name": "verify_igmp_snooping_vlans", "configuration": "disabled", "vlans": [10, 12]}
18.02    {"name": "verify_igmp_snooping_global", "configuration": "enabled"}

Repository structure

devices.txt file

The file devices.txt is the devices inventory.

The devices inventory is a text file with the devices IP address or hostnames (if resolvable with DNS or your hosts file). This file has one device per ligne.

eos-commands.yaml file

The file eos-commands.yaml is a YAML file used to indicated the list of commands output we would like to collect from EOS devices in text or json format.

tests.yaml file

The file tests.yaml is a YAML file used to indicated the tests we would like to run. It is also used to indicated the parameters used by the tests. Each test has an identifier which is then used in the tests report. The tests are defined in the directory tests_eos.

tests_eos directory

The directory tests_eos is a python package.

The python functions to test EOS devices are defined the python module functions.py in the python package tests_eos.

generate_functions_documentation.py file

The script generate_functions_documentation.py is used to generate the functions documentation in markdown format. It requires the installation of the package lazydocs that is indicated in the file requirements-dev.txt

The functions to test EOS devices are coded in the python module functions.py in the python package tests_eos. These functions have docstrings. The docstrings are used by the script generate_functions_documentation.py to generate the functions documentation in markdown format in the directory documentation.

documentation directory

The documentation directory has the tests documentation in markdown format:

check-devices-reachability.py file

The python script check-devices-reachability.py is used to test devices reachability with eAPI.

The python script check-devices-reachability.py takes as input a text file with the devices IP address or hostnames (when resolvable), and tests devices reachability with eAPI and prints the unreachable devices on the console.

clear_counters.py file

The python script clear_counters.py is used to clear counters on EOS devices.

It takes as input a text file with the devices IP address or hostnames (when resolvable) and clears counters on these devices.

collect-eos-commands.py file

The python script collect-eos-commands.py is used to collect commands output from EOS devices.

The python script collect-eos-commands.py:

  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of EOS commands we would like to collect in text or JSON format.
  • Collects the EOS commands from the devices, and saves the result in files.

check-devices.py file

The python script check-devices.py is used to run tests on devices.

The python script check-devices.py:

  • Imports the python functions defined in the directory tests_eos.
    • These functions defined the tests.
  • Takes as input:
    • A text file with the devices IP address or hostnames (when resolvable).
    • A YAML file with the list of the tests we would like to use and their parameters.
  • Runs the tests, and prints the result on the console and saves the result in a file.

unit_test.py file

The python script unit_test.py is used to test the functions defined in the directory test_eos without using actual EOS devices. It requires the installation of the package pytest that is indicated in the file requirements-dev.txt

mock_data directory

The mock_data directory has data used by the python script unit_test.py to test the functions defined in the directory test_eos without using actual EOS devices.

Contribution guide

Contributions are welcome. The contribution guide is CONTRIBUTING.md

Credits

Thank you to Paul Lavelle, Colin MacGiollaEain, Matthieu Tache, Angélique Phillipps, Thomas Grimonet, Onur Gashi and Khelil Sator for their contributions and guidances.

Comments
  • feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    feat(anta.inventory): Add support for DNS names, eAPI port and device alias names

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. The PR is to support hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 3
  • fix(check-devices): Update --timeout management + add logging management

    fix(check-devices): Update --timeout management + add logging management

    • Fix an issue where timeout was not correctly set.
    • Fix timeout defined as an str where a float was expected
    • Add -log or --logging to set log level for inventory and result_manager

    Default mode

    check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    

    With logging enabled

    ❯ check-devices.py -i .personal/avd-lab.yml -c .personal/ceos-catalog.yml --tags arista --username admin --password admin123 --table --timeout 1 --loglevel info
    ╭──────────────────────── Settings ─────────────────────────╮
    │ Running check-devices with:                               │
    │               - Inventory: .personal/avd-lab.yml          │
    │               - Tests catalog: .personal/ceos-catalog.yml │
    ╰───────────────────────────────────────────────────────────╯
    [14:08:41] WARNING  Service not running on device 10.73.1.254 with:                                                                                                                                                                                                                                                                                    __init__.py:191
               WARNING  connection return: f('admin:[email protected]/command-api', 302, 'Found', <http.client.HTTPMessage object at 0x1062952b0>)                                                                                                                                                                                                      __init__.py:192
               WARNING  Service not running on device 10.73.1.254                                                                                                                                                                                                                                                                                          __init__.py:222
    [14:08:42] WARNING  Service not running on device 10.100.164.150 with:                                                                                                                                                                                                                                                                                 __init__.py:191
               WARNING  connection return: ftimed out                                                                                                                                                                                                                                                                                                      __init__.py:192
    [14:08:43] WARNING  Service not running on device 10.100.164.150                                                                                                                                                                                                                                                                                       __init__.py:222
               INFO     Inventory .personal/avd-lab.yml loaded                                                                                                                                                                                                                                                                                        check-devices.py:202
               INFO     starting running test on inventory ...                                                                                                                                                                                                                                                                                        check-devices.py:208
               INFO     testing done !                                                                                                                                                                                                                                                                                                                check-devices.py:226
               INFO     retrieve list of result using output_format list                                                                                                                                                                                                                                                                                   __init__.py:106
                     All tests results                  
    ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Device IP ┃ Test Name ┃ Test Status ┃ Message(s) ┃
    ┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    └───────────┴───────────┴─────────────┴────────────┘
    
    bug 
    opened by titom73 3
  • feat(reporter): Implement initial reporter using rich

    feat(reporter): Implement initial reporter using rich

    Reporting

    Provides easy reporting module

    • Get all tests results:
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Device IP    ┃ Test Name           ┃ Test Status ┃ Message(s)                                                                                   ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.10 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.11 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.12 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.13 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.14 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_eos_version  │ failure     │ device is running version 4.27.3F-26379303.4273F (engineering build) not in expected version │
    │ 192.168.0.15 │ verify_uptime       │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_reload_cause │ error       │ 'response'                                                                                   │
    │ 192.168.0.15 │ verify_coredump     │ success     │                                                                                              │
    │ 192.168.0.15 │ verify_agent_logs   │ success     │                                                                                              │
    └──────────────┴─────────────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get tests summarized
    ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Test Case          ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed or error nodes                                                                    ┃
    ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ verify_eos_version │ 0            │ 6            │ 0           │ ['192.168.0.10', '192.168.0.11', '192.168.0.12', '192.168.0.13', '192.168.0.14', '192.168.0.15'] │
    └────────────────────┴──────────────┴──────────────┴─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
    
    • Get hosts summarized
    ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃ Host IP      ┃ # of success ┃ # of failure ┃ # of errors ┃ List of failed ortest case                    ┃
    ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │ 192.168.0.15 │ 3            │ 1            │ 1           │ ['verify_eos_version', 'verify_reload_cause'] │
    └──────────────┴──────────────┴──────────────┴─────────────┴───────────────────────────────────────────────┘
    

    Testing

    python .github/scripts/anta-tester.py -h
    [17:03:13] INFO     ANTA program is loaded                                                                                                                                                                                                                                                                                                                 anta-tester.py:99
    usage: anta-tester.py [-h] [--inventory INVENTORY] [--catalog CATALOG] [--username USERNAME] [--password PASSWORD] [--search_ip SEARCH_IP] [--search_test SEARCH_TEST] [--verbose] [--list] [--full] [--devices] [--testcases] [--table] [--save] [--save_file SAVE_FILE]
    
    ANTA test & demo script
    
    optional arguments:
      -h, --help            show this help message and exit
      --inventory INVENTORY, -i INVENTORY
                            ANTA Inventory file
      --catalog CATALOG, -c CATALOG
                            ANTA Tests cagtalog
      --username USERNAME, -u USERNAME
                            EOS Username
      --password PASSWORD, -p PASSWORD
                            EOS Password
      --search_ip SEARCH_IP
                            search result for host
      --search_test SEARCH_TEST
                            search result for test
      --verbose, -v         Set script to verbose mode
      --list, -l            Display internal data
      --full                Display all test cases results
      --devices             Provides summary of test results per device
      --testcases           Provides summary of test results per test case
      --table               Result represented in tables
      --save                Save result to file
      --save_file SAVE_FILE
                            Generated report file
    
    framework-enhancement 
    opened by titom73 2
  • collect sheduled show tech

    collect sheduled show tech

    #!/usr/bin/env python3
    
    ## This script collects all the tech-support files stored on Arista switches flash and copies them locally
    
    import ssl
    import paramiko
    from jsonrpclib import Server
    from scp import SCPClient
    from time import strftime, gmtime
    from argparse import ArgumentParser
    from getpass import getpass
    import os
    from sys import exit
    from socket import setdefaulttimeout
    
    port = 22
    ssl._create_default_https_context = ssl._create_unverified_context
    date = strftime("%d %b %Y %H:%M:%S", gmtime())
    
    def device_directories (device, root_dir):
        cwd = os.getcwd()
        show_tech_directory = cwd + '/' + root_dir
        device_directory = show_tech_directory + '/' + device
        for directory in [show_tech_directory, device_directory]:
            if not os.path.exists(directory):
                os.makedirs(directory)
        result = show_tech_directory, device_directory
        return result
    
    def createSSHClient (device, port, username, password):
        client = paramiko.SSHClient()
        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        client.connect(device, port, username, password)
        return client
    
    def main():
        parser = ArgumentParser(
            description='Collect all the tech-support files'
            )
        parser.add_argument(
            '-i',
            help='Text file containing a list of switches',
            dest='file',
            required=True
            )
        parser.add_argument(
            '-u',
            help='Devices username',
            dest='username',
            required=True
            )
        parser.add_argument(
            '-o',
            help='Output directory',
            dest='output_directory',
            required=True
            )
        args = parser.parse_args()
        args.password = getpass(prompt='Device password: ')
    
        try:
            with open(args.file, 'r') as file:
                devices = file.readlines()
        except:
            print('Error opening ' + args.file)
            exit(1)
    
        for i,device in enumerate(devices):
            devices[i] = device.strip()
    
        # Deletes unreachable devices from devices list
    
        unreachable = []
        
        print('Connecting to devices .... please be patient ... ')
        
        for device in devices:
            try:
                setdefaulttimeout(5)
                url = 'https://%s:%s@%s/command-api' %(args.username, args.password, device)
                switch = Server(url)
                response = switch.runCmds(1, ['enable'])
            except:
                unreachable.append(device)
        
        for item in unreachable:
            devices.remove(item)
            print("Can not connect to device " + item)
    
        # Collects all the tech-support files stored on Arista switches flash and copies them locally
    
        for device in devices:
            url = "https://" + args.username + ":" + args.password + "@" + device + "/command-api"
            try:
                ## Create one zip file named all_files.zip on the device with the all the show tech-support files in it
                cmds=['enable','bash timeout 30 zip /mnt/flash/schedule/all_files.zip /mnt/flash/schedule/tech-support/*']
                switch = Server(url)
                switch.runCmds(1,cmds, 'text')
                ## Connect to the device using SSH
                ssh = createSSHClient(device, port, args.username, args.password)
                ## Create directories 
                output_dir = device_directories (device, args.output_directory)
                ## Get the zipped file all_files.zip using SCP and save it locally
                my_path = output_dir[1] + '/' + date + '.zip'
                ssh = createSSHClient(device, port, args.username, args.password)
                scp = SCPClient(ssh.get_transport())
                scp.get("/mnt/flash/schedule/all_files.zip",local_path = my_path)
                scp.close()
                ## Delete the created zip file on the device
                cmds=['enable','bash timeout 30 rm /mnt/flash/schedule/all_files.zip']
                switch.runCmds(1,cmds, 'text')
            except:
                print('You are unlucky today! ' + device + ' does not like this script')
    
    if __name__ == '__main__':
        main()
        ```
    opened by ksator 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    opened by gusmb 1
  • split the module `tests` of the package `anta` into multiples modules

    split the module `tests` of the package `anta` into multiples modules

    The module is too large (>1600 lines). we can split it into diff modules:

    • routing
    • hardware
    • interfaces
    • ... This will require an update of the check-devices.py
    opened by ksator 1
  • Scripts can't run due to 'jsonrpclib' incompatibility

    Scripts can't run due to 'jsonrpclib' incompatibility

    I am having an issue running the scripts, as jsonrpclib doesn't work with Python3:

    testvm:~/network_tests_automation# ./check-devices-reachability.py --help
    Traceback (most recent call last):
      File "./check-devices-reachability.py", line 5, in <module>
        from jsonrpclib import Server
    ModuleNotFoundError: No module named 'jsonrpclib'
    

    I've installed PIP dependencies as instructed, and even added 'pip install jsonrpclib' separately:

    test-vm:~/network_tests_automation# /usr/bin/env python3 -V
    Python 3.6.9
    
    testvm:~# pip list | grep jsonrpc
    jsonrpclib (0.1.7)
    jsonrpclib-pelix (0.4.3.2)
    
    

    There seems to be a known issue with this library and Python3: https://stackoverflow.com/questions/38664102/python-jsonrpclib-not-working-after-upgrade-to-python-3-5-2

    Any ideas how to get around this?

    opened by ctyrider 1
  • feat(anta.inventory): use aioeapi to handle a device session

    feat(anta.inventory): use aioeapi to handle a device session

    Using asyncio framework will help improving performances when running tests on a large network with a high number of tests. First step is to refactor the anta.inventory module to use an async library, this PR tries an attempt with aio-eapi. This PR must also refactor all the tests as coroutines and modify the scripts to run the asyncio event loop.


    Modifications to anta.inventory

    The session attribute of InventoryDevice is now a aioeapi.Device object. auto_connect is removed from the constructor: this would cause to run an asyncio event loop in an object constructor and prevent fine control from a ANTA user script on the asyncio scheduling.

    _is_device_online() uses aioeapi.Device.check_connection() to only tries to open a port on the device. _ refresh_device_fact() updates all the device flags and reduce the number of API calls compared to the previous implementation.

    connect_inventory() is a coroutine that needs to be called in a ANTA user script to establish connection towards the devices.

    async def main():
       await anta_inventory.connect_inventory()
    

    When using a containerlab topology on macOS, we need to use port forwarding to localhost to access eAPI endpoints. The current inventory only supports IP addresses. This PR supports hostnames (DNS names instead of IP), ports and alias names in the inventory.

    The following inventory works with this PR:

    anta_inventory:
      hosts:
      - name: spine1
        host: localhost
        port: 44301
      - name: spine2
        host: localhost
        port: 44302
      - name: leaf1
        host: localhost
        port: 44311
      - name: leaf1
        host: localhost
        port: 44312
      - name: leaf3
        host: localhost
        port: 44313
      - name: leaf4
        host: localhost
        port: 44314
    
    opened by mtache 2
  • Add support for custom defined tests

    Add support for custom defined tests

    Description

    In order to allow users to build their own test collection, it would be nice to add option to support catalog pointing to custom libraries.

    Today, we only look at anta.tests package but we could extend YAML format to support both anta.tests and a custom package with the following format:

    <package.name>:
          - <test function>:
               <function options>
    

    An example would be:

    anta.tests.software:
      - verify_eos_version: # Verifies the device is running one of the allowed EOS version.
          versions: # List of allowed EOS versions.
            - 4.25.4M
            - 4.26.1F
    inetsix.tests.mydesign:
      - check_topology:
    

    Potential impacts:

    • anta.loader.parse_catalog
    framework-enhancement 
    opened by titom73 0
  • Add attributes 'label' and 'description' with function decorator

    Add attributes 'label' and 'description' with function decorator

    Hello,

    Currently the only precision about the tests is the name of the function itself. It is therefore necessary to read the code of the function to understand how it works and to have more details.

    An idea would be to add to each test function a python decorator with two attributes: label and description.

    Example with this sample piece of code:

    def decorator(**kwargs):
        def wrapper(function):
            for k, v in kwargs.items():
                setattr(function, k, v)
            return function
        return wrapper
    
    @decorator(label='EOS version', description='Verifies the device is running one of the allowed EOS version')
    def verify_eos_version():
        pass
    
    print(f'label: {verify_eos_version.label}\ndescription: {verify_eos_version.description}')
    
    >>> label: EOS version
    >>> description: Verifies the device is running one of the allowed EOS version
    

    Now, we can access easily this information, without having to add two fields in a .yaml file, for example. Thank you very much.

    opened by noam-zeitoun 2
  • Implement

    Implement "AntaTest" and "AntaAsyncTest" as protocols

    Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

    framework-enhancement 
    opened by gusmb 2
  • Map tags to inventory devices and test catalog

    Map tags to inventory devices and test catalog

    Increase flexibility by using tags on inventory devices and the test catalog, so that catalog tests can run on relevant devices only. For instance, in s typical EVPN scenario there.might be some tests that should run only in spines and some other tests only on leaf switches, having also some tests that run on all devices. Using tags or filters on tests and inventory devices could achieve this easily and add flexibility to the way tests execute. For inventory networks/ranges, tags would apply to the whole range/network...

    framework-enhancement 
    opened by gusmb 1
Releases(v0.3.0)
  • v0.3.0(Sep 26, 2022)

    v0.3.0

    Features

    • 836f995 [anta.reporter] Add skipped column in aggregated results (#110)
    • 8c3404b [inventory] Add tag support to filter test execution (#112)
    • 40f6977 [result_manager] Support JSON output and save option (#113)

    Bug fixes

    • c010bae [check-devices] Update --timeout management + add logging management (#117)
    • cfd9d59 [fix] Missing docstring for tags (#114)

    Documentation

    Other changes

    • 39f80e4 [make] Build system: Prepare version management (#120)
    • 3262f09 [mkdocs] Build system: Fix incorrect requirement path
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0-alpha(Jul 22, 2022)

    Add the Inventory module in the anta package

    Migrated the following scripts to use the Inventory module :

    check-devices-reachability.py
    clear-counters.py
    collect-eos-commands.py
    collect-sheduled-show-tech.py
    evpn-blacklist-recovery.py
    

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jul 2, 2022)

    Includes the following package:

    • anta with 53 functions to test EOS

    Includes the following scripts:

    • check-devices-reachability.py
    • check-devices.py
    • clear-counters.py
    • collect-eos-commands.py
    • collect-sheduled-show-tech.py
    • evpn-blacklist-recovery.py

    This release is not in PyPi

    Source code(tar.gz)
    Source code(zip)
Owner
Netdevops Community
Arista Network Automation Community (Community projects only, no direct affiliation)
Netdevops Community
A complete test automation tool

Golem - Test Automation Golem is a test framework and a complete tool for browser automation. Tests can be written with code in Python, codeless using

null 486 Dec 30, 2022
Flexible test automation for Python

Nox - Flexible test automation for Python nox is a command-line tool that automates testing in multiple Python environments, similar to tox. Unlike to

Stargirl Flowers 941 Jan 3, 2023
API Test Automation with Requests and Pytest

api-testing-requests-pytest Install Make sure you have Python 3 installed on your machine. Then: 1.Install pipenv sudo apt-get install pipenv 2.Go to

Sulaiman Haque 2 Nov 21, 2021
Front End Test Automation with Pytest Framework

Front End Test Automation Framework with Pytest Installation and running instructions: 1. To install the framework on your local machine: clone the re

Sergey Kolokolov 2 Jun 17, 2022
a plugin for py.test that changes the default look and feel of py.test (e.g. progressbar, show tests that fail instantly)

pytest-sugar pytest-sugar is a plugin for pytest that shows failures and errors instantly and shows a progress bar. Requirements You will need the fol

Teemu 963 Dec 28, 2022
Pynguin, The PYthoN General UnIt Test geNerator is a test-generation tool for Python

Pynguin, the PYthoN General UnIt test geNerator, is a tool that allows developers to generate unit tests automatically.

Chair of Software Engineering II, Uni Passau 997 Jan 6, 2023
Ab testing - The using AB test to test of difference of conversion rate

Facebook recently introduced a new type of offer that is an alternative to the current type of bidding called maximum bidding he introduced average bidding.

null 5 Nov 21, 2022
A small automated test structure using python to test *.cpp codes

Get Started Insert C++ Codes Add Test Code Run Test Samples Check Coverages Insert C++ Codes you can easily add c++ files in /inputs directory there i

Alireza Zahiri 2 Aug 3, 2022
A simple serverless create api test repository. Please Ignore.

serverless-create-api-test A simple serverless create api test repository. Please Ignore. Things to remember: Setup workflow Change Name in workflow e

Sarvesh Bhatnagar 1 Jan 18, 2022
This repository contnains sample problems with test cases using Cormen-Lib

Cormen Lib Sample Problems Description This repository contnains sample problems with test cases using Cormen-Lib. These problems were made for the pu

Cormen Lib 3 Jun 30, 2022
ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries. With this f

Mustafa 1 Jul 11, 2022
WrightEagle AutoTest (Has been updated by Cyrus team members)

Autotest2d WrightEagle AutoTest (Has been updated by Cyrus team members) Thanks go to WrightEagle Members. Steps 1- prepare start_team file. In this s

Cyrus Soccer Simulation 2D Team 3 Sep 1, 2022
PacketPy is an open-source solution for stress testing network devices using different testing methods

PacketPy About PacketPy is an open-source solution for stress testing network devices using different testing methods. Currently, there are only two c

null 4 Sep 22, 2022
Generic automation framework for acceptance testing and RPA

Robot Framework Introduction Installation Example Usage Documentation Support and contact Contributing License Introduction Robot Framework is a gener

Robot Framework 7.7k Jan 7, 2023
A cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard.

PyAutoGUI PyAutoGUI is a cross-platform GUI automation Python module for human beings. Used to programmatically control the mouse & keyboard. pip inst

Al Sweigart 7.5k Dec 31, 2022
A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well

folder-automation A folder automation made using Watch-dog, it only works in linux for now but I assume, it will be adaptable to mac and PC as well Th

Parag Jyoti Paul 31 May 28, 2021
A browser automation framework and ecosystem.

Selenium Selenium is an umbrella project encapsulating a variety of tools and libraries enabling web browser automation. Selenium specifically provide

Selenium 25.5k Jan 1, 2023
Headless chrome/chromium automation library (unofficial port of puppeteer)

Pyppeteer Pyppeteer has moved to pyppeteer/pyppeteer Unofficial Python port of puppeteer JavaScript (headless) chrome/chromium browser automation libr

miyakogi 3.5k Dec 30, 2022
✅ Python web automation and testing. 🚀 Fast, easy, reliable. 💠

Build fast, reliable, end-to-end tests. SeleniumBase is a Python framework for web automation, end-to-end testing, and more. Tests are run with "pytes

SeleniumBase 3k Jan 4, 2023