AthenaCLI is a CLI tool for AWS Athena service that can do auto-completion and syntax highlighting.

Overview

Build Status PyPI Downloads image image Join the chat at https://gitter.im/dbcli/athenacli

Introduction

AthenaCLI is a command line interface (CLI) for the Athena service that can do auto-completion and syntax highlighting, and is a proud member of the dbcli community.

Quick Start

Install

Install via pip

If you already know how to install python packages, then you can simply do:

$ pip install athenacli

Install via brew

Homebrew users can install by:

$ brew install athenacli

If you don't know how to install python packages, please check the Install page for more options (e.g docker)

Config

A config file is automatically created at ~/.athenacli/athenaclirc at first launch (run athenacli). See the file itself for a description of all available options.

Below 4 variables are required. If you are a user of aws cli, you can refer to awsconfig file to see how to reuse credentials configuration of aws cli.

# AWS credentials
aws_access_key_id = ''
aws_secret_access_key = ''
region = '' # e.g us-west-2, us-east-1

# Amazon S3 staging directory where query results are stored.
# NOTE: S3 should in the same region as specified above.
# The format is 's3://
   
    '
s3_staging_dir = ''

# Name of athena workgroup that you want to use
work_group = '' # e.g. primary

   

or you can also use environment variables:

$ export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
$ export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
$ export AWS_DEFAULT_REGION=us-west-2
$ export AWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/
$ export AWS_ATHENA_WORK_GROUP=YOUR_ATHENA_WORK_GROUP

Create a table

$ athenacli -e examples/create_table.sql

You can find examples/create_table.sql here.

Run a query

$ athenacli -e 'select elb_name, request_ip from elb_logs LIMIT 10'

REPL

$ athenacli [<database_name>]

Features

  • Auto-completes as you type for SQL keywords as well as tables and columns in the database.
  • Syntax highlighting.
  • Smart-completion will suggest context-sensitive completion.
    • SELECT * FROM will only show table names.
    • SELECT * FROM users WHERE will only show column names.
  • Pretty prints tabular data and various table formats.
  • Some special commands. e.g. Favorite queries.
  • Alias support. Column completions will work even when table names are aliased.

Please refer to the Features page for the screenshots of above features.

Usages

$ athenacli --help
Usage: main.py [OPTIONS] [DATABASE]

A Athena terminal client with auto-completion and syntax highlighting.

Examples:
    - athenacli
    - athenacli my_database

Options:
-e, --execute TEXT            Execute a command (or a file) and quit.
-r, --region TEXT             AWS region.
--aws-access-key-id TEXT      AWS access key id.
--aws-secret-access-key TEXT  AWS secretaccess key.
--s3-staging-dir TEXT         Amazon S3 staging directory where query
                                results are stored.
--work-group TEXT             Amazon Athena workgroup in which query is run, default is primary
--athenaclirc PATH            Location of athenaclirc file.
--help                        Show this message and exit.

Please go to the Usages for detailed information on how to use AthenaCLI.

Contributions

If you're interested in contributing to this project, first of all I would like to extend my heartfelt gratitude. I've written a small doc to describe how to get this running in a development setup.

Please feel free to reach out to me if you need help. My email: zhuzhaolong0 AT gmail com

FAQs

Please refer to the FAQs for other information, e.g. "How can I get support for athenacli?".

Credits

A special thanks to Amjith Ramanujam for creating pgcli and mycli, which inspired me to create this AthenaCLI, and AthenaCLI is created based on a clone of mycli.

Thanks to Jonathan Slenders for creating the Python Prompt Toolkit, which leads me to pgcli and mycli. It's a lot of fun playing with this library.

Thanks to PyAthena for a pure python adapter to Athena database.

Last but not least, thanks my team and manager encourage me to work on this hobby project.

Similar projects

Comments
  • Fix bug when completing

    Fix bug when completing "ON parent." clauses

    Previously, the code passed the wrong arguments to identifiers.

    Fixes #40.

    This is my first time working in this project, so any and all feedback is welcome!

    opened by pgr0ss 8
  • Latest release (1.4.0) not working if athenaclirc file not in path

    Latest release (1.4.0) not working if athenaclirc file not in path

    athenacli works with version 1.3.3, but I get this message when trying to run athenacli 1.4.0:

    Required argument `s3_staging_dir` or `work_group` not found.
    
    There was an error while connecting to AWS Athena. It could be caused due to
    missing/incomplete configuration. Please verify the configuration in ~/.athenacli/athenaclirc
    and run athenacli again.
    
    opened by pdpark 7
  • Pin Pygments

    Pin Pygments

    Pygments made a backwards incompatible change resulting in error messages like:

    2022-01-10 20:00:55,648 (3535/MainThread) athenacli.main ERROR - traceback: 'Traceback (most recent call last):
      File "/home/admin/.local/lib/python3.7/site-packages/pygments/formatters/terminal256.py", line 261, in format_unencoded
        on, off = self.style_string[str(ttype)]
    KeyError: \'Token.Output.TableSeparator\'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/admin/.local/lib/python3.7/site-packages/athenacli/main.py", line 312, in one_iteration
    
    title, rows, headers, special.is_expanded_output(), None
      File "/home/admin/.local/lib/python3.7/site-packages/athenacli/main.py", line 453, in format_output
        **output_kwargs)
    

    This is a result of https://github.com/pygments/pygments/commit/8fdd6b9879a6c34b908fa03a6ff15f764192a54e

    See related error for pgcli - https://github.com/dbcli/pgcli/issues/1303

    See main pygments issue - https://github.com/pygments/pygments/issues/2027

    Description

    Checklist

    • [x] I've added this contribution to the changelog.md.
    • [x] I've added my name to the AUTHORS file (or it's already there).
    opened by allcentury 6
  • Can't install promt-tooklit >= 3.0 alongside athenacli

    Can't install promt-tooklit >= 3.0 alongside athenacli

    I regularly use both litecli and athenacli for local and remote work. Because this tool requires prompt-toolkit < 3.0, anything else that requires prompt toolkit >= 3.0 will cause pip errors (or just won't install at all, if using poetry). litecli is one such tool that requires prompt-toolkit >= 3.0 that I regularly use for quick local work, and I'd prefer to have both that and athenacli in the same virtual environment, if possible.

    Any chance the restriction on prompt-toolkit can be opened up to "<4.0"?

    opened by renzmann 5
  • Add

    Add "read" command to read and execute a SQL file

    Description

    Add command to allow executing a query file while inside the REPL.

    Heavily borrowed from execute_favorite_query and watch_query commands.

    Resolves https://github.com/dbcli/athenacli/issues/60

    Checklist

    • [x] I've added this contribution to the changelog.md.
    • [x] I've added my name to the AUTHORS file (or it's already there).
    opened by sco11morgan 5
  • Support for default credentials.

    Support for default credentials.

    It is common for the official awscli tools and boto to use credentials in ~/.aws/credentials, or in environment variables. We use this heavily to support logging in using short lived STS tokens, and avoid creating long lived secret keys, which are a security no-no.

    It would be great if this tool supported using those default credentials. (And STS tokens)

    enhancement P1 
    opened by EvilPuppetMaster 5
  • Table auto complete fails when a Glue table name has a

    Table auto complete fails when a Glue table name has a "." in its name

    Hi, I have a glue table named "example.example". When I'm using the Database and try to select * from, the auto completion fails, and the following error message is displayed:

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
        self.run()
      File "/opt/conda/lib/python3.7/threading.py", line 870, in run
        self._target(*self._args, **self._kwargs)
      File "/opt/conda/lib/python3.7/site-packages/athenacli/completion_refresher.py", line 71, in _bg_refresh
        refresher(completer, executor)
      File "/opt/conda/lib/python3.7/site-packages/athenacli/completion_refresher.py", line 113, in refresh_tables
        completer.extend_columns(executor.table_columns(), kind='tables')
      File "/opt/conda/lib/python3.7/site-packages/athenacli/completer.py", line 124, in extend_columns
        metadata[self.dbname][relname].append(column)
    KeyError: '"example.example"'
    

    Is this a known issue?

    opened by nadavaviv 4
  • update execution_time_in_millis as lib PyAthena has updated

    update execution_time_in_millis as lib PyAthena has updated

    Description

    https://github.com/laughingman7743/PyAthena/commit/dad812d4e2bf69683bfb73e56ca3c01f72b4f396

    Test Env, Centos 6.10, python 2.6 and python 3.6

    us-east-1:default> show TABLES; 'Cursor' object has no attribute 'execution_time_in_millis'

    After the change us-east-1:default> show TABLES; Query OK Execution time: 240 ms, Data scanned: 0 B, Approximate cost: $0.00

    Checklist

    • [x] I've added this contribution to the changelog.md.
    • [x] I've added my name to the AUTHORS file (or it's already there).
    opened by toskachin 4
  • Add support for workgroup

    Add support for workgroup

    Description

    Add support for configuring Athena Workgroup.

    https://docs.aws.amazon.com/athena/latest/ug/workgroups.html

    Checklist

    • [x] I've added this contribution to the changelog.md.
    • [x] I've added my name to the AUTHORS file (or it's already there).
    opened by WarFox 4
  • Improve performance of CSV formatter when handling NULLs

    Improve performance of CSV formatter when handling NULLs

    When using table_format = csv, the default value for missing values is the empty string which we don't override, but cli_helpers still tries to style this which is fairly costly.

    opened by tail 4
  • Add support for pipenv

    Add support for pipenv

    The old setup method is still valid. I haven't update develop.rst for pipenv because of my poor English. Please update that to pipenv. Contributors can setup dev environment by change directory into this repo and

    # install dependences for development   
    pipenv install --dev 
    # active virtual environment
    pipenv shell
    # deactive virtual environment
    deactivate
    
    opened by flappyBug 4
  • Feature request: Support converting zero bytes and high-bit-set bytes to octal sequences for VARBINARY

    Feature request: Support converting zero bytes and high-bit-set bytes to octal sequences for VARBINARY

    PostgreSQL has a function encode(data, 'escape') where data is a bytea. https://www.postgresql.org/docs/9.4/functions-binarystring.html

    Currently when using athencli w/ the Athena JDBC the default output for VARBINARY is to convert to UTF8 and if unable to convert it just outputs the hex representation of the VARBINARY. It would be extremely useful to have the ability to support this output mode directly in athenacli for any VARBINARY column when set.

    Here is an example select on a VARBINARY column from Athena when using the Python pyathenajdbc library which is a wrapper for the Athena JAR. Notice how the ASCII converts but other other bytes are left as-is:

    b'\x03\x00\x00+&\xe0\x00\x00\x00\x00\x00Cookie: mstshash=hello\r\n\x01\x00\x08\x00\x03\x00\x00\x00'

    opened by Obsecurus 4
  • Feature request: better auto-completion

    Feature request: better auto-completion

    Problems

    • For very large databases, we are experiencing a ~20 sec delay when switching using use
    • Autocompletion only works for 1 database at a time (e.g. doesn't work for cross-database queries)

    Possible solutions

    • Fetch auto completions on SELECT database_name.<tab>
    • Get all the autocompletions up-front?
    • Cache when switching databases?
    • Make it configurable?
    enhancement 
    opened by getaaron 2
  • Support role assumption & MFA prompts from profile without AWS_SESSION... env vars

    Support role assumption & MFA prompts from profile without AWS_SESSION... env vars

    In our environment, we often rely on profiles without the AWS_SESSION... environment variables, which is apparently the "new way" of doing temporary tokens in AWS-land.

    So boto3 tries to prompt for my MFA, at which point athenacli gets a bit confused: it appears that athenacli and boto3 are competing for control of the terminal. I get prompted several times, and get some really lovely stack traces:

    Enter MFA code for arn:aws:iam::xxxxxxxxxxxxxxx:mfa/rick.cobb: Failed to execute query.
    Traceback (most recent call last):
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/athenacli/sqlexecute.py", line 78, in run
        for result in special.execute(cur, sql):
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/athenacli/packages/special/main.py", line 58, in execute
        raise CommandNotFound
    athenacli.packages.special.main.CommandNotFound
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyathena/common.py", line 166, in _execute
        **request)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/pyathena/util.py", line 44, in retry_api_call
        return retry(func, *args, **kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call
        do = self.iter(retry_state=retry_state)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter
        return fut.result()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/concurrent/futures/_base.py", line 425, in result
        return self.__get_result()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
        raise self._exception
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call
        result = fn(*args, **kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
        return self._make_api_call(operation_name, kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/client.py", line 610, in _make_api_call
        operation_model, request_dict)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 102, in make_request
        return self._send_request(request_dict, operation_model)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 132, in _send_request
        request = self.create_request(request_dict, operation_model)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/endpoint.py", line 116, in create_request
        operation_name=operation_model.name)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 356, in emit
        return self._emitter.emit(aliased_event_name, **kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 228, in emit
        return self._emit(event_name, kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/hooks.py", line 211, in _emit
        response = handler(**kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 90, in handler
        return self.sign(operation_name, request)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 149, in sign
        auth = self.get_auth_instance(**kwargs)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/signers.py", line 229, in get_auth_instance
        frozen_credentials = self._credentials.get_frozen_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 518, in get_frozen_credentials
        self._refresh()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 413, in _refresh
        self._protected_refresh(is_mandatory=is_mandatory_refresh)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 429, in _protected_refresh
        metadata = self._refresh_using()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 566, in fetch_credentials
        return self._get_cached_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 576, in _get_cached_credentials
        response = self._get_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 697, in _get_credentials
        client = self._create_client()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 716, in _create_client
        frozen_credentials = self._source_credentials.get_frozen_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 518, in get_frozen_credentials
        self._refresh()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 413, in _refresh
        self._protected_refresh(is_mandatory=is_mandatory_refresh)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 429, in _protected_refresh
        metadata = self._refresh_using()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 177, in __call__
        return self._refresh()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 566, in fetch_credentials
        return self._get_cached_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 576, in _get_cached_credentials
        response = self._get_credentials()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 696, in _get_credentials
        kwargs = self._assume_role_kwargs()
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/site-packages/botocore/credentials.py", line 707, in _assume_role_kwargs
        token_code = self._mfa_prompter(prompt)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/getpass.py", line 77, in unix_getpass
        passwd = _raw_input(prompt, stream, input=input)
      File "/home/rcobb/.pyenv/versions/3.6.5/lib/python3.6/getpass.py", line 148, in _raw_input
        raise EOFError
    
    

    I'm currently working around this using a new feature of our development tools where we set the AWS_SESSION... env vars, which is fine, but it'd be nice to support the newer profile stuff.

    enhancement P3 
    opened by cobbr2 4
Owner
dbcli
Better CLIs for Databases
dbcli
Postgres CLI with autocompletion and syntax highlighting

A REPL for Postgres This is a postgres client that does auto-completion and syntax highlighting. Home Page: http://pgcli.com MySQL Equivalent: http://

dbcli 10.8k Jan 2, 2023
AWS Interactive CLI - Allows you to execute a complex AWS commands by chaining one or more other AWS CLI dependency

AWS Interactive CLI - Allows you to execute a complex AWS commands by chaining one or more other AWS CLI dependency

Rafael Torres 2 Dec 10, 2021
Interactive Redis: A Terminal Client for Redis with AutoCompletion and Syntax Highlighting.

Interactive Redis: A Cli for Redis with AutoCompletion and Syntax Highlighting. IRedis is a terminal client for redis with auto-completion and syntax

null 2.2k Dec 29, 2022
A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting.

mycli A command line client for MySQL that can do auto-completion and syntax highlighting. HomePage: http://mycli.net Documentation: http://mycli.net/

dbcli 10.7k Jan 7, 2023
A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool

Privateer A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool How

Shreyash Chavan 2 Apr 4, 2022
A CLI tool to disable and enable security standards controls in AWS Security Hub

Security Hub Controls CLI A CLI tool to disable and enable security standards controls in AWS Security Hub. It is designed to work together with AWS S

AWS Samples 4 Nov 14, 2022
🦎 A NeoVim plugin for highlighting visual selections like in a normal document editor!

?? HighStr.nvim A NeoVim plugin for highlighting visual selections like in a normal document editor! Demo TL;DR HighStr.nvim is a NeoVim plugin writte

Pocco81 222 Jan 3, 2023
Sink is a CLI tool that allows users to synchronize their local folders to their Google Drives. It is similar to the Git CLI and allows fast and reliable syncs with the drive.

Sink is a CLI synchronisation tool that enables a user to synchronise local system files and folders with their Google Drives. It follows a git C

Yash Thakre 16 May 29, 2022
[WIP]An ani-cli like cli tool for movies and webseries

mov-cli A cli to browse and watch movies. Installation This project is a work in progress. However, you can try it out python git clone https://github

null 166 Dec 30, 2022
spid-sp-test is a SAML2 SPID/CIE Service Provider validation tool that can be executed from the command line.

spid-sp-test spid-sp-test is a SAML2 SPID/CIE Service Provider validation tool that can be executed from the command line. This tool was born by separ

Developers Italia 30 Nov 8, 2022
triggercmd is a CLI client for the TRIGGERcmd cloud service.

TriggerCMD CLI client triggercmd is a CLI client for the TRIGGERcmd cloud service. installation the triggercmd package is available in PyPI. to instal

Gustavo Soares 7 Oct 18, 2022
A supercharged AWS command line interface (CLI).

SAWS Motivation AWS CLI Although the AWS CLI is a great resource to manage your AWS-powered services, it's tough to remember usage of: 70+ top-level c

Donne Martin 5.1k Jan 5, 2023
A cli tool , which shows you all the next possible words you can guess from in the game of Wordle.

wordle-helper A cli tool , which shows you all the next possible words you can guess from the Game Wordle. This repo has the code discussed in the You

null 1 Jan 17, 2022
flora-dev-cli (fd-cli) is command line interface software to interact with flora blockchain.

Install git clone https://github.com/Flora-Network/fd-cli.git cd fd-cli python3 -m venv venv source venv/bin/activate pip install -e . --extra-index-u

null 14 Sep 11, 2022
Python-Stock-Info-CLI: Get stock info through CLI by passing stock ticker.

Python-Stock-Info-CLI Get stock info through CLI by passing stock ticker. Installation Use the following command to install the required modules at on

Ayush Soni 1 Nov 5, 2021
Yts-cli-streamer - A CLI movie streaming client which works on yts.mx API written in python

YTSP It is a CLI movie streaming client which works on yts.mx API written in pyt

null 1 Feb 5, 2022
Fast as FUCK nvim completion. SQLite, concurrent scheduler, hundreds of hours of optimization.

Fast as FUCK nvim completion. SQLite, concurrent scheduler, hundreds of hours of optimization.

i love my dog 2.8k Jan 5, 2023
This is a CLI program which can help you generate your own QR Code.

Python-QR-code-generator This is a CLI program which can help you generate your own QR Code. Single.py This will allow you only to input a single mess

null 1 Dec 24, 2021