Ralph is a command-line tool to fetch, extract, convert and push your tracking logs from various storage backends to your LRS or any other compatible storage or database backend.

Overview

Ralph

Ralph is a command-line tool to fetch, extract, convert and push your tracking logs (aka learning events) from various storage backends to your LRS or any other compatible storage or database backend.

Quick start guide

Ralph is distributed as a standard python package; it can be installed via pip or any other python package manager (e.g Poetry, Pipenv, etc.):

$ pip install ralph-malph

Once installed, the ralph command should be available in your PATH. Try to invoke the program usage thanks to the --help flag:

$ ralph --help

You should see a list of available commands and global flags for ralph. Note that each command has its own usage that can be invoked via:

$ ralph COMMAND --help

You should substitute COMMAND by the target command, e.g. list, to see its usage.

Documentation

We try our best to maintain an up-to-date reference documentation for this project. If you intend to install, test or contribute to ralph, we invite you to read this documentation and give us feedback if some parts are unclear or your use case is not (or poorly) covered.

Contributing

This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.

We try to raise our code quality standards and expect contributors to follow the recommandations from our handbook.

License

This work is released under the MIT License (see LICENSE).

Comments
  • ✨(models) add edx navigational event pydantic models

    ✨(models) add edx navigational event pydantic models

    Purpose

    With the validate and select command newly implemented, it is now possible to implement edx tracking logs events pydantic models to validate incoming events in ralph.

    Proposal

    The first events implemented are the easiest (navigation).

    • [x] event models
    • [x] event fixtures
    • [x] event tests
    needs review feature 
    opened by quitterie-lcs 9
  • Add WebSocket to backends documentation

    Add WebSocket to backends documentation

    Purpose

    Add WebSocket to Backends documentation

    Proposal

    WebSocket

    The webSocket backend is read-only and can be used to get real-time events.

    If you use OVH's Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

    Backend parameters

    • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)
    improvement WIP 
    opened by p-bizouard 6
  • ✨(converters) add edx server event xapi converter

    ✨(converters) add edx server event xapi converter

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to convert them into xAPI format.

    Proposal

    ServerEventSchema validates the edx server event.

    ServerEventToXapi uses the ServerEventSchema to validate the edx server event and then converts it to xAPI format.

    XapiConverterSelector deserializes the edx events and tries to match a known converter for the event conversion.

    feature 
    opened by SergioSim 6
  • Add helm chart

    Add helm chart

    Purpose

    Add Helm chart for Ralph

    Description

    Created the chart under src/helm I tried to migrate the logic from the .j2 tray files. The version label was replaced as the helm convention for app.kubernetes.io/version The checksum from the secret name was deleted. The annotation checksum/config: was used instead on the secret, the checksum is automatically generated from vault.yaml it help to update the secret object every time a value is updated. Note that the .Chart.AppVersion is better used to track the version deployed and it should reflect the image.tag There is only one vars files values.yaml the vault is under specific file vault.yaml used only for generating the secrets.

    Questions

    • Why do you have two service with the same config? the only difference is the label static
    • What is deployment_stamp ?
    • What is the purpose of prefix ?
    • How would you like to manage env.?

    ToDo

    • [ ] Add README.md

    You can use helm template . under ./src/helm/ralph to check the output of the chart, it will show you the final manifest that will be deployed.

    feature 
    opened by waammar 5
  • Add MongoDB backend

    Add MongoDB backend

    Purpose

    MongoDB is a classical database for LRSs such as LearningLocker or Apereo Learning Analytics Initiative. Supporting this backend seems a must-have for Ralph.

    Proposal

    • [x] add LearningLocker parser
    • [x] add MongoDB backend
    • [x] ~add learning locker parser~ edit: postponed to a new PR
    needs review feature 
    opened by jmaupetit 5
  • ✨(converter) add edx to xapi video converter

    ✨(converter) add edx to xapi video converter

    Purpose

    edX video events can be converted to xAPI for most of them. The interaction events can still not be converted.

    Proposal

    • [x] load_video to initialized
    • [x] play_video to played
    • [x] pause_video to paused
    • [x] stop_video to terminated
    • [x] seek_video to seeked
    improvement needs review FUN 
    opened by quitterie-lcs 4
  • ✨(models) add xAPI page_viewed and page_terminated models

    ✨(models) add xAPI page_viewed and page_terminated models

    Purpose

    These pydantic models can be used for xAPI format validation.

    Proposal

    The page_viewed pydantic model correspond to the edX simple server event. The page_terminated pydantic model correspond to the edX page_close browser event.

    • [X] page_viewed pydantic model
    • [X] page_terminated pydantic model
    • [x] tests
    feature WIP 
    opened by SergioSim 4
  • ✨(models) add edx server event model

    ✨(models) add edx server event model

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to validate them.

    Proposal

    • [x] ServerEventModel
    • [x] ServerEventFactory
    • [x] Tests
    feature WIP 
    opened by SergioSim 4
  • Add test fixtures

    Add test fixtures

    Purpose

    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    feature WIP 
    opened by SergioSim 4
  • Add

    Add "event" test fixture factory

    Purpose

    • Document the different events that are logged as "tracking logs" in the LMS to understand how, were and when things get logged
    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    • [ ] Events documentation
    feature WIP 
    opened by SergioSim 4
  • ✨(cli) add the new

    ✨(cli) add the new "auth" command

    Purpose

    Generating credentials to authenticate to our LRS should be smooth and painless for our users.

    Proposal

    • [x] add a new auth command to ease new user credentials generation
    • [x] add pydantic models for user credentials checking and credentials file validation
    • [x] track authentication issues to the LRS server
    needs review feature 
    opened by jmaupetit 3
  • `make bootstrap` is failing due to missing build dependences

    `make bootstrap` is failing due to missing build dependences

    Bug Report

    Hi all, I'm excited to get started developing with Ralph. Getting set up I ran into an issue, however. On the master branch make bootstrap errors out pip installing psutil due to gcc not being available.

    Expected behavior/code make bootstrap builds the docker images cleanly.

    Actual Behavior An error occurs:

    #0 22.22       creating build/temp.linux-aarch64-cpython-39/psutil
    #0 22.22       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=594 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-39/psutil/_psutil_common.o
    #0 22.22       C compiler or Python headers are not installed on this system. Try to run:
    #0 22.22       sudo apt-get install gcc python3-dev
    #0 22.22       error: command 'gcc' failed: No such file or directory
    

    Steps to Reproduce

    1. Sync the Ralph master branch
    2. run make bootstrap
    3. The above error occurs

    Environment

    • Ralph version: master branch
    • Platform: Docker Desktop on Mac OS 13.0.1

    Possible Solution I was able to get a successful build by adding the required development tools, see the PR here: https://github.com/bmtcril/ralph/pull/1

    bug 
    opened by bmtcril 1
  • 📝(project) add helm chart tutorial

    📝(project) add helm chart tutorial

    Purpose

    Ralph can be deployed to a Kubernetes cluster using its official Helm Chart. We need a step-by-steph guide to use it.

    Proposal

    This tutorial guides you to the deployment of a complete learning analytics stack to Kubernetes using Helm.

    FUN WIP 
    opened by jmaupetit 1
  • Implement xAPI forum models in `ralph`

    Implement xAPI forum models in `ralph`

    Purpose

    xAPI forum used in ashley models have to be described in ralph for validation purpose and usage of ralph as a library

    Proposal

    • [ ] Define pydantic models associated to each forum learning statement templates
    • [ ] Define selector for each model
    • [ ] Write model and selector tests
    feature 
    opened by quitterie-lcs 0
  • Integrate mypy for Python type checking

    Integrate mypy for Python type checking

    Purpose

    It has been decided that we use typing in all ralph project. We have to integrate a static type checker in our linting toolbox.

    Proposal

    • Integrate mypy in the project (local tooling for development + CI)
    improvement 
    opened by quitterie-lcs 0
  • 🏗️(backends) unify data backends fs

    🏗️(backends) unify data backends fs

    Purpose

    Storage and Database backends have similar interfaces and usage.

    Proposal

    Unify the interfaces of database and storage backends into a single DataBackend interface.

    • [x] add Data and LRS Backend interfaces
    • [x] add FileSystemDataBackend
    needs review feature 
    opened by SergioSim 1
  • Ora2 events

    Ora2 events

    Purpose

    ORA2 events describe the activity for assessments pair reviewing. Edx defined models and their selectors are described with Pydantic.

    Proposal

    Define Pydantic model for:

    • openassessmentblock.create_submission,
    • openassessmentblock.get_peer_submission,
    • openassessmentblock.get_submission_for_staff_grading,
    • openassessmentblock.peer_assess,
    • openassessmentblock.save_submission,
    • openassessmentblock.self_assess,
    • openassessmentblock.staff_assess,
    • openassessmentblock.student_training_assess_example,
    • openassessment.submit_feedback_on_assessments,
    • openassessment.upload_file,
    needs review feature 
    opened by quitterie-lcs 0
Releases(v3.1.0)
  • v3.1.0(Nov 17, 2022)

    Added

    • EdX to xAPI converters for video events

    Changed

    • Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
    • Upgrade fastapi to 0.87.0
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Oct 19, 2022)

    Added

    • Implement edx video browser events pydantic models
    • Create a post endpoint for statements implementing the LRS spec
    • Implement support for the MongoDB database backend
    • Implement support for custom queries when using database backends get method (used in the fetch command)
    • Add dotenv configuration file support and python-dotenv dependency
    • Add host and port options for the runserver cli command
    • Add support for database selection when running the Ralph LRS server
    • Implement support for xAPI statement forwarding
    • Add database backends status checking
    • Add health LRS router
    • Tray: add LRS server support

    Changed

    • Migrate to python-legacy handler for mkdocstrings package
    • Upgrade click to 8.1.3
    • Upgrade elasticsearch to 8.3.3
    • Upgrade fastapi to 0.79.1
    • Upgrade ovh to 1.0.0
    • Upgrade pydantic to 1.9.2
    • Upgrade pymongo to 4.2.0
    • Upgrade python-keystoneclient to 5.0.0
    • Upgrade python-swiftclient to 4.0.1
    • Upgrade requests to 2.28.1
    • Upgrade sentry_sdk to 1.9.5
    • Upgrade uvicorn to 0.18.2
    • Upgrade websockets to 10.3
    • Make backends yield results instead of writing to standard streams (BC)
    • Use pydantic settings management instead of global variables in defaults.py
    • Rename backend and parser parameter environment variables (BC)
    • Make project dependencies management more modular for library usage

    Removed

    • Remove YAML configuration file support and pyyaml dependency (BC)

    Fixed

    • Tray: do not create a cronjobs list when no cronjob has been defined
    • Restore history mixin logger
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Apr 25, 2022)

    Added

    • Implement edx problem interaction events pydantic models
    • Implement edx textbook interaction events pydantic models
    • ws websocket stream backend (compatible with the fetch command)
    • bundle jq, curl and wget in the fundocker/ralph Docker image
    • Tray: enable ralph app deployment command configuration
    • Add a runserver command with basic auth and a whoami route
    • Create a get endpoint for statements implementing the LRS spec

    Changed

    • Upgrade uvicorn to 0.17.4
    • Upgrade elasticsearch to 7.17.0
    • Upgrade sentry_sdk to 1.5.5
    • Upgrade fastapi to 0.73.0
    • Upgrade pyparsing to 3.0.7
    • Upgrade pydantic to 1.9.0
    • Upgrade python-keystoneclient to 4.4.0
    • Upgrade python-swiftclient to 3.13.0
    • Upgrade pyyaml to 6.0
    • Upgrade requests to 2.27.1
    • Upgrade websockets to 10.1
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Jul 15, 2021)

  • v2.0.0(Jul 12, 2021)

    Added

    • xAPI video interacted pydantic models
    • xAPI video terminated pydantic models
    • xAPI video completed pydantic models
    • xAPI video seeked pydantic models
    • xAPI video initialized pydantic models
    • xAPI video paused pydantic models
    • convert command to transform edx events to xAPI format
    • EdX to xAPI converters for page viewed andpage_close events
    • Implement core event format converter
    • xAPI video played pydantic models
    • xAPI page viewed and page terminated pydantic models
    • Implement edx navigational events pydantic models
    • Implement edx enrollment events pydantic models
    • Install security updates in project Docker images
    • Model selector to retrieve associated pydantic model of a given event
    • validate command to lint edx events using pydantic models
    • Support all available bulk operation types for the elasticsearch backend (create, index, update, delete) using the --es-op-type option

    Changed

    • Upgrade elasticsearch to 7.13.2
    • Upgrade python-swiftclient to 3.12.0
    • Upgrade click to 8.0.1
    • Upgrade click-option-group to 0.5.3
    • Upgrade pydantic to 1.8.2
    • Upgrade sentry_sdk to 1.1.0
    • Rename edx models
    • Migrate model tests from factories to hypothesis strategies
    • Tray: switch from openshift to k8s (BC)
    • Tray: remove useless deployment probes

    Fixed

    • Tray: remove version immutable field in DC selector
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 26, 2021)

    Added

    • EdX server event pydantic model and factory
    • EdX page_close browser event pydantic model and factory
    • Tray: allow to specify a self-generated elasticsearch cluster CA certificate

    Fixed

    • Tray: add missing Swift variables in the secret
    • Tray: fix pods anti-affinity selector

    Removed

    • pandas is no longer required
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 4, 2021)

    Added

    • Support for Swift storage backend
    • Use the push command --ignore-errors option to ignore ES bulk import errors
    • The elasticsearch backend now accepts passing all supported client options

    Changed

    • Remove click_log package dependency
    • Upgrade pyyaml to 5.4.1
    • Upgrade pandas to 1.2.1
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 13, 2021)

    Added

    • Implement base CLI commands (list, extract, fetch & push) for supported backends
    • Support for ElasticSearch database backend
    • Support for LDP storage backend
    • Support for FS storage backend
    • Parse (gzipped) tracking logs in GELF format
    • Support for application's configuration file
    • Add optional sentry integration
    • Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs
    Source code(tar.gz)
    Source code(zip)
Owner
France Université Numérique
Collaboratively Building Innovative Digital Learning Services
France Université Numérique
A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool

Privateer A simple CLI based any Download Tool, that find files and let you stream or download thorugh WebTorrent CLI or Aria or any command tool How

Shreyash Chavan 2 Apr 4, 2022
A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Stream your favorite shows straight from the command line.

A command-line based, minimal torrent streaming client made using Python and Webtorrent-cli. Installation pip install -r requirements.txt It use

Jonardon Hazarika 17 Dec 11, 2022
A command line application to analyse reports from TBC Warcraft Logs.

README A command line application to analyse reports from TBC Warcraft Logs. The application was written and tested with Python 3.9. Features Dumps an

null 2 Dec 17, 2021
Command line tool to keep track of your favorite playlists on YouTube and many other places.

Command line tool to keep track of your favorite playlists on YouTube and many other places.

Wolfgang Popp 144 Jan 5, 2023
A simple CLI tool for converting logs from Poker Now games to other formats

?? Poker Now Log Converter ?? A command line utility for converting logs from Poker Now games to other formats. Introduction Poker Now is a free onlin

null 6 Dec 23, 2022
A simple automation script that logs into your kra account and files your taxes with one command

EASY_TAX A simple automation script that logs into your kra account and files your taxes with one command Currently works for Chrome users. Will creat

leon koech 13 Sep 23, 2021
Unofficial Open Corporates CLI: OpenCorporates is a website that shares data on corporations under the copyleft Open Database License. This is an unofficial open corporates python command line tool.

Unofficial Open Corporates CLI OpenCorporates is a website that shares data on corporations under the copyleft Open Database License. This is an unoff

Richard Mwewa 30 Sep 8, 2022
A command line utility for tracking a stock market portfolio. Primarily featuring high resolution braille graphs.

A command line stock market / portfolio tracker originally insipred by Ericm's Stonks program, featuring unicode for incredibly high detailed graphs even in a terminal.

Conrad Selig 51 Nov 29, 2022
Tiny command-line utility for mapping broken keys to other positions.

brokenkey Tiny command-line utility for mapping broken keys to other positions. Installation Clone this repository using git: git clone https://github

null 0 Oct 4, 2021
Shortcut-Maker - It is a tool that can be set to run any tool with a single command

Shortcut-Maker It is a tool that can be set to run any tool with a single command Coded by Dave Smith(Owner of Sl Cyber Warriors) Command list ?? pkg

Dave Smith 10 Sep 14, 2022
A cd command that learns - easily navigate directories from the command line

NAME autojump - a faster way to navigate your filesystem DESCRIPTION autojump is a faster way to navigate your filesystem. It works by maintaining a d

William Ting 14.5k Jan 3, 2023
Ros command - Unifying the ROS command line tools

Unifying the ROS command line tools One impairment to ROS 2 adoption is that all

null 37 Dec 15, 2022
A command line tool to create a graph representing your Ansible playbook tasks and roles

Ansible Playbook Grapher ansible-playbook-grapher is a command line tool to create a graph representing your Ansible playbook plays, tasks and roles.

Mohamed El Mouctar Haidara 424 Dec 20, 2022
A command-line tool to upload local files and pastebin pastes to your mega account, without signing in anywhere

A command-line tool to upload local files and pastebin pastes to your mega account, without signing in anywhere

ADI 4 Nov 17, 2022
A command line tool to query source code from your current Python env

wxc wxc (pronounced "which") allows you to inspect source code in your Python environment from the command line. It is based on the inspect module fro

Clément Robert 13 Nov 8, 2022
Command-line tool to use LNURL with your LND instance

LND LNURL Commandline tool to use LNURL payRequest and withdrawRequest with LND. Usage: Customize config lndlnurl.conf Run python main.py <LNURL> Dock

Djuri Baars 6 Jan 14, 2022
Tidier - a simple command line tool that helps you make your files tidy up

Tidier - a simple command line tool that helps you make your files tidy up

AmirMohammad Hosseini Nasab 8 Aug 16, 2022
A lightweight Python module and command-line tool for generating NATO APP-6(D) compliant military symbols from both ID codes and natural language names

Python military symbols This is a lightweight Python module, including a command-line script, to generate NATO APP-6(D) compliant military symbol icon

Nick Royer 5 Dec 27, 2022