alien.py - Python interface to websocket endpoint of ALICE Grid Services

Overview

PyPI

alien.py - Python interface to websocket endpoint of ALICE Grid Services

Quick containerized testing:
singularity run library://adriansev/default/alienpy:latest [cmd]
or
singularity run oras://registry.cern.ch/asevcenc/alienpy:latest [cmd]

latest usually would point to master but not always.(if desired and needed, request by email a new latest tag)
see here what tags are available and their dates of creation.

The docker images can be found @DockerHub

docker run -it \
--user $(id -u):$(id -g) \
--workdir="/home/$USER" \
--env TMPDIR="${TMPDIR:-/tmp}" \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/home:/home" \
adriansevcenco/alienpy.dock:latest [cmd]

if no cmd is passed, the shell form will start

Basic usage

Can be used as command mode and interactive mode :

  1. Command mode :
    alien.py e.g :
    alien.py pwd
    N.B. command/arguments must be quoted to avoid being interpreted by the shell:
    alien.py 'rm my_alien_dir/*'

  2. Interactive/shell mode e.g :

alien.py
Welcome to the ALICE GRID
support mail: [email protected]

AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >pwd
/alice/cern.ch/user/a/asevcenc/
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >whoami
asevcenc
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >
  • For both command and shell mode multiple commands can be issued separated by ;
  • The interactive mode save the command history in ${HOME}/.alienpy_history and it can be navigated with Up/Down keys
  • ! is understood as running into shell whatever command follows
  • | pipe whatever output of AliEn command to a shell command (that follows after the first(only the first) |)

Environment steering

There are a few environment variables that influence the mechanics of the script :

  • JALIEN_TOKEN_CERT, JALIEN_TOKEN_KEY - will overwrite the defaults; they are either full path certificate,key token files OR their respective contents
  • If set the following will be honored: X509_USER_CERT, X509_USER_KEY, X509_CERT_DIR or X509_CERT_FILE
  • ALIENPY_TIMEOUT will change the interval for keep-alive mechanics.
  • ALIENPY_CONNECT_TRIES - default = 3 : number of connect trials
  • ALIENPY_CONNECT_TRIES_INTERVAL - default = 0.5 : seconds between connection trials

For debugging purposes there are a few environment toggles :

  • ALIENPY_JSON - print the unprocessed json message from the server
  • ALIENPY_JSONRAW - print the unprocessed byte stream message from the server

  • ALIENPY_JCENTRAL - it will connect to this server, ignoring any other options
  • ALIENPY_NO_STAGGER - disable staggered parallel host resolution and socket creation (see RFC8305)

  • ALIENPY_DEBUG - detailed debug meesages will be found in ALIENPY_DEBUG_FILE
  • ALIENPY_DEBUG_FILE - set the location of log file
  • ALIENPY_DEBUG_APPEND - is set the output will be appended to the present log file. if not the file will be overwritten.
  • ALIENPY_TIMECONNECT - if set will report time for websocket creation - e.g. ALIENPY_TIMECONNECT=1 alien.py pwd
  • ALIENPY_TIMING - report detailed operation timing in the log file.

DEBUG file copy operations:

  • ALIENPY_KEEP_META - keep the metafile generated for download operations. Can be directly used with xrdcp.
  • XRD_LOGLEVEL='Dump'
  • XRD_LOGFILE=xrdlog.txt

See also the native XRootD environment toggles: docs

Authentication

The authentication process needs the presence of a X509 certificate (enrolled into ALICE VO, see here) and of a CA certificates directory for verification. The default CA location that will be searched is within alice.cern.ch cvmfs repository If not found, the CApath will default to /etc/grid-security/certificates If these locations are not available, one must set X509_CERT_DIR to a valid location

Command usage and examples

The list of available commands can seen with: ? or help
Command help can be listed with: ? command, help command, command -h

Storage related operations

This section refer to any copy to/from grid or file interactions.

cat/more/less will download the target lfn to a temporary file and will act upon it while
vi/nano/mcedit/edit will, after the modification of downloaded temporary, backup the existing lfn, and upload the modified file
The target file upload can support grid specifiers like those described in cp command e.g. edit my_file@disk:2,SE1

cp option

cp can take as arguments both files and directories and have the following options:

: select only these files to be copied; N.B. this is a REGEX applied to full path!!! -name : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!! -name _string : where verb = begin|contain|ends|ext and string is the text selection criteria. verbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root N.B. the text to be filtered cannont have underline <_> within!!! -parent : in destination use this to add to destination ; defaults to 0 -a : copy also the hidden files .* (for recursive copy) -j : select only the files created by the job with (for recursive copy) -l : copy only nr of files (for recursive copy) -o : skip first files found in the src directory (for recursive copy) ">
alien.py cp -h
Command format is of the form of (with the strict order of arguments):
cp 
             
               src dst
or
cp 
              
                -input input_file

location prefixes are: file: | file:// | alien: | alien://
if one prefix is specified the other operator is considered of the other kind (no local -> local, or grid->grid operations allowed)
if no prefix is specified, the src will be _first_ checked if local and then if remote.

-input argument is a file with >src dst< pairs

after each src,dst can be added comma separated specifiers in the form of: @disk:N,SE1,SE2,!SE3
where disk selects the number of replicas and the following specifiers add (or remove) storage endpoints from the received list
options are the following :
-h : print help
-f : replace destination file (if destination is local it will be replaced only if integrity check fails)
-P : enable persist on successful close semantic
-cksum : check hash sum of the file; for downloads the central catalogue md5 will be verified;
for uploads (for xrootd client > 4.12.0) a hash type will be negociated with remote and transfer will be validated
-y 
               
                 : use up to the number of sources specified in parallel (N.B. Ignored as it breaks download of files stored in archives)
-S 
                
                  : uses num additional parallel streams to do the transfer. (max = 15)
-chunks 
                 
                   : number of chunks that should be requested in parallel -chunksz 
                  
                    : chunk size (bytes) -T 
                   
                     : number of parralel copy jobs from a set (for recursive copy); defaults to 8 for downloads -noxrdzip: circumvent the XRootD mechanism of zip member copy and download the archive and locally extract the intended member. N.B.!!! for recursive copy (all files) the same archive will be downloaded for each member. If there are problems with native XRootD zip mechanism, download only the zip archive and locally extract the contents for the recursive copy of directories the following options (of the find command) can be used: -glob 
                    
                      : this is the usual AliEn globbing format; N.B. this is NOT a REGEX!!! defaults to all "*" -select 
                     
                       : select only these files to be copied; N.B. this is a REGEX applied to full path!!! -name 
                      
                        : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!! -name 
                       
                        _string : where verb = begin|contain|ends|ext and string is the text selection criteria. verbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root N.B. the text to be filtered cannont have underline <_> within!!! -parent 
                        
                          : in destination use this 
                         
                           to add to destination ; defaults to 0 -a : copy also the hidden files .* (for recursive copy) -j 
                          
                            : select only the files created by the job with 
                           
                             (for recursive copy) -l 
                            
                              : copy only 
                             
                               nr of files (for recursive copy) -o 
                              
                                : skip first 
                               
                                 files found in the src directory (for recursive copy) 
                               
                              
                             
                            
                           
                          
                         
                        
                       
                      
                     
                    
                   
                  
                 
                
               
              
             

., .. are interpreted for all grid names (lfns)
%ALIEN is converted to user AliEn home directory
lfns that don't start with a / will have the current directory appended before being processed

Miscellaneous

The shell prompt

It can show date and/or local directory:

  • prompt date will toggle on/off the date
  • prompt pwd will toggle on/off the local current directory
    For permanent setting the following are env variables are available : ALIENPY_PROMPT_DATE, ALIENPY_PROMPT_CWD
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt date
2020-02-07T16:49:05 AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >

AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ >prompt pwd
AliEn[asevcenc]:/alice/cern.ch/user/a/asevcenc/ local:/home.hdd/adrian/work-GRID/jalien_py >

CWD persistence

Default behaviour is to save (and then restore) the last used CWD.
This bevahiour can be disabled with the env var ALIENPY_NO_CWD_RESTORE


ls aliases

ll, la, lla are aliases to ls -l, ls -a, ls -la


Custom aliases

A fixed file ${HOME}/.alienpy_aliases can be used to define alias=string pairs that will be used(translated) in the usage of alien.py. One can do myalias=cmd1;cmd2;cmd3 and the myalias string will be replaced by it's value when used.


Python shell

term command will open an Python shell within the context of alien.py and with a session object loaded


API usage

see examples directory

You might also like...
wssh ("wish") is a command-line utility/shell for WebSocket inpsired by netcat.

wssh ("wish") is a command-line utility/shell for WebSocket inspired by netcat

Whatsapp Clone using django, django-channels and websocket
Whatsapp Clone using django, django-channels and websocket

whatsapp-clone Whatsapp Clone using django, django-channels and websocket Features : Signup/Login One on One personal chat with other user Some screen

Minecraft WebSocket

Minecraft-WebSocket Pythonでマインクラフトと通信します。 紹介動画 推奨設定 Minecraft Windows Edition (Education Edition) 1.17 以上 Python 3系(3.8.2で動作確認済み) 必要なモジュール ・asyncio ・w

Tetri5 - Multiplayer Websocket Backend

Tetri5 - Multiplayer Websocket Backend This repository is the backend of the multiplayer portion of the Tetri5 game client. It uses the python websock

image stream publish server over websocket
image stream publish server over websocket

Image Stream Push Server 简介 通过浏览器网页实时查看图像处理结果。 环境 运行程序需要安装一下python依赖: tornado: 用于创建http及websocket服务; opencv-contrib-python: 用于图像数据源获取及图像处理。 使用 进入到src目

AWS API Gateway Websocket Asynchronous Notifications Pusher
AWS API Gateway Websocket Asynchronous Notifications Pusher

AWS API Gateway Websocket Asynchronous Pusher Fast AWS API Gateway websockets notifications' pusher using Python AsyncIO for managing asynchronous and

Discord.py Connect to Discord voice call with websocket

Discord.py Connect to Discord voice call with websocket

A websocket client for Source Filmmaker intended to trasmit scene and frame data to other applications.

SFM SOCK A websocket client for Source Filmmaker intended to trasmit scene and frame data to other applications. This software can be used to transmit

Synci - Learning project to create a websocket based client server messaging application

Synci Learning project to create a websocket based client server messaging appli

Comments
  • Console scripts

    Console scripts

    In this PR we create the using facing scripts (such as alien_cp) through the setuptools console-script mechanism. That is, the shell script present in bin/ of the repository are removed in favour of defining the entry-points in setup.py.

    So what happens: When installing the package (via pip, directly via setup.py - f.ex. in develop mode), setuptools writes small wrappers to the target installation bin directory (e.g., /usr/local/bin, ~/.local/bin) which imports the appropriate module or package and executes the named function from that module.

    As an example

    entry_points = {
        'console_scripts': [
            'alien_cp = xjalienfs.alien:cmd_cp',
         }
    }
    

    will generate the console-script alien_cp which essentially does

    import xjalienfs.alien 
    xjalienfs.alien.cmp_cp()
    

    To this end, we add a few (small) functions to alien.py - essentially doing, for example

    def do_cp():
        sys.argv = sys.argv[:1] + ['cp'] + sys.argv[1:]
        main()
    

    The benefits of using setuptools entry points like this is that it can correctly take care of platform idiosyncrasies, set-up proper load paths, and so on.

    Remember, one can always install a package in developer mode (i.e., no scripts are actually copied to destination - instead a "pointer" is created at the destination that points back to the working directory. In this way, any changes made to the source is immediately reflected in the the "installation") by

    python setup.py develop --user 
    

    The only script not rewritten like this is alice_pfn which remains a shell script that calls the entry point alien_which.

    opened by cholmcc 7
  • Debian

    Debian

    This PR adds the sub-directory debian for easy Debian (and hence Ubuntu) packaging. To package the project, simply do

    python3 setup.py sdist  -d ../ 
    n=`python3 setup.py --name`; v=`python setup.py --version` ; mv ../${n}-${v}.tar.gz ../${n}_${v}.orig.tar.gz
    dpkg-buildpackage -us -uc -tc -i"\.git|examples/alien.py"
    

    Note, the package isn't really distribution quality as some things are missing, but this PR provides a starting point. I'm not too familiar with Python packaging for RPM systems, so I do not have anything for that.

    opened by cholmcc 4
  • Fix setup

    Fix setup

    This PR removed the Python package gnureadline as a dependency. It does not disable gnureadline at run-time. Also, XRootD dependency renamed from pyxrootd to xrootd.

    The rational: gnureadline is deprecated on all platforms but MacOSX. Instead, one should use the default Python readline interface. Thus, the package must not depend on gnureadline but can still use it if present on the target system. Dependencies are supposed to stipulate minimal requirements - not nice to have features.

    The Python interface of XRootD provides xrootd not pyxrootd.

    opened by cholmcc 1
  • Fix security level for 80bit server

    Fix security level for 80bit server

    This PR fixes the problem when the client machine default SSL security level is set to 2 (Debian, Ubuntu, and similar). This is done by setting the Cipher string to DEFAULT@SECLEVEL=1 in the client directly. Similar fix should be applied to jalien-ROOT.

    opened by cholmcc 0
Owner
Adrian Sevcenco
Adrian Sevcenco
WebSocket and WAMP in Python for Twisted and asyncio

Autobahn|Python WebSocket & WAMP for Python on Twisted and asyncio. Quick Links: Source Code - Documentation - WebSocket Examples - WAMP Examples Comm

Crossbar.io 2.4k Jan 4, 2023
Library for building WebSocket servers and clients in Python

What is websockets? websockets is a library for building WebSocket servers and clients in Python with a focus on correctness and simplicity. Built on

Aymeric Augustin 4.3k Jan 4, 2023
WebSocket client for Python

websocket-client The websocket-client module is a WebSocket client for Python. It provides access to low level APIs for WebSockets. All APIs are for s

null 3.1k Jan 2, 2023
Websockify is a WebSocket to TCP proxy/bridge. This allows a browser to connect to any application/server/service. Implementations in Python, C, Node.js and Ruby.

websockify: WebSockets support for any application/server websockify was formerly named wsproxy and was part of the noVNC project. At the most basic l

noVNC 3.3k Jan 3, 2023
WebSocket emulation - Python server

SockJS-tornado server SockJS-tornado is a Python server side counterpart of SockJS-client browser library running on top of Tornado framework. Simplif

Serge S. Koval 854 Nov 19, 2022
一款为 go-cqhttp 的正向 WebSocket 设计的 Python SDK

Nakuru Project 一款为 go-cqhttp 的正向 WebSocket 设计的 Python SDK 在 kuriyama 的基础上改动 项目名来源于藍月なくる,图标由せら绘制 食用方法 将 nakuru 文件夹移至 Python 的 Lib/site-packages 目录下。

null 35 Dec 21, 2022
Using python-binance to provide websocket data to freqtrade

The goal of this project is to provide an alternative way to get realtime data from Binance and use it in freqtrade despite the exchange used. It also uses talipp for computing

null 58 Jan 1, 2023
Django Channels HTTP/WebSocket server

daphne Daphne is a HTTP, HTTP2 and WebSocket protocol server for ASGI and ASGI-HTTP, developed to power Django Channels. It supports automatic negotia

Django 1.9k Dec 31, 2022
This websocket program is for data transmission between server and client. Data transmission is for Federated Learning in Edge computing environment.

websocket-for-data-transmission This websocket program is for data transmission between server and client. Data transmission is for Federated Learning

null 9 Jul 19, 2022
Benchmark a WebSocket server's message throughput ⌛

?? WebSocket Benchmarker ⌚ Message throughput is how fast a WebSocket server can parse and respond to a message. Some people consider this to be a goo

Andrew Healey 24 Nov 17, 2022