A github actions + python code to extract URLs to code repositories to put into standard form, starting with github

Overview

repo_link_extractor

A github actions + python code to extract URLs to code repositories to put into standard form, starting with github

---- NOTE: JUST STARTED ONLY AN IDEA TO COME BACK TO ----

Summary

first minimum viable product goal

The first minimum viable product goal will be to harvest from https://github.com/softwareunderground/awesome-open-geoscience/blob/main/README.md all the github repositories URLs such that the form returned is https://github.com/ + "username" + "repository name" and then add them to the "repos" key in an existing JSON in a form like this: https://github.com/softwareunderground/open_geosciene_code_projects_viz/blob/main/_explore/input_lists.json , which is summarized below:

{
    "memberOrgs": [
        "softwareunderground"
    ],
    "orgs": [
        "agile-geoscience",
        "softwareunderground"
    ],
    "repos": [
        "ahotovec/redpy",
        "whamlyn/auralib"
    ]
}

2nd intermediate product goal

  • Fires from a GitHubAction

3rd intermediate product goal

Eventual product goal

  • works for public, internal, and private GitHub URLs
  • Works for GitHub, GitLab, BitBucket, and other code repository URLS & APIs
  • Keeps track of harvest date, source file name, source file URL & code platform & domain in an intermediate file.

Related Projects

This is referenced on an issue here: https://github.com/softwareunderground/open_geosciene_code_projects_viz/issues/23

Potential Useful Bits

regular expression (https:\/\/github.com\/)\w+(\/)\w+ seems like a good starting point for the extraction of Github URLs.

GitHub Actions Structure Tentative:

  • download README file
  • replace old README file with new
  • extract all links matching a regular expression
  • sort & take out duplicates
  • make into JSON with domain, URL, org or username, repository name, source file name, source file link, and date of harvests
  • pull out org or username & repository name from above and put into appropriate key of the file JSON if not already there in either org or repo keys.

How to Integrate into https://github.com/softwareunderground/open_geosciene_code_projects_viz ??????

Options:

  1. Put all of the code here into the repository: https://github.com/softwareunderground/open_geosciene_code_projects_viz
  2. Call the code here from https://github.com/softwareunderground/open_geosciene_code_projects_viz
If calling the code....
  • (1) add the script to read the README to MASTER.sh as the first step
  • (2) set master.sh to be callled by GitHub actions
  • (3) when triggered the github actions does the entirity of the github actions in this repo, including calling the python scripts as its first step.
  • (4) latter steps include setting up the environnment and calling all the python scripts that the master.sh bash script calls. The code would need to be called by either a GitHub Action on (pull request, push, manual, or cron job) or by trigger after the call to refresh the
You might also like...
Automation that uses Github Actions, Google Drive API, YouTube Data API and youtube-dl together to feed BackJam app with new music

Automation that uses Github Actions, Google Drive API, YouTube Data API and youtube-dl together to feed BackJam app with new music

42-event-notifier - 42 Event notifier using 42API and Github Actions
42-event-notifier - 42 Event notifier using 42API and Github Actions

42 Event Notifier 42서울 Agenda에 새로운 이벤트가 등록되면 알려드립니다! 현재는 Github Issue로 등록되므로 상단

GitHub Actions Docker training

GitHub-Actions-Docker-training Training exercise repository for GitHub Actions using a docker base. This repository should be cloned and used for trai

A discord.py bot template with easy deployment through Github Actions
A discord.py bot template with easy deployment through Github Actions

discord.py bot template A discord.py bot template with easy deployment through Github Actions. You can use this template to just run a Python instance

A simple python script to send files into your telegram Bot form your PC, Server etc.

telegramSend A simple python script to send files into your telegram Bot form your PC, Server etc. How to Use Install requirements.txt pip3 install -r

An example of a chatbot with a number-based menu that can be used as a starting point for a project.
An example of a chatbot with a number-based menu that can be used as a starting point for a project.

NumMenu Bot NumMenu Bot is an example chatbot showing a way to design a number-based menu assistant with Rasa. This type of bot is very useful on plat

A Discord bot to scrape textfiles from messages and put them to Hastebin

A Discord bot to scrape textfiles from messages and put them to Hastebin. Intended to use on support servers to help users read textfiles on mobile.

GitPython is a python library used to interact with Git repositories.

Gitoxide: A peek into the future… I started working on GitPython in 2009, back in the days when Python was 'my thing' and I had great plans with it. O

A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel
A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel

mpgitleaks A Python script that wraps the gitleaks tool to enable scanning of multiple repositories in parallel. The motivation behind writing this sc

Owner
Justin Gosses
Machine-Learning | Data Visualization | Geoscience | NASA |
Justin Gosses
Use GitHub Actions to create a serverless service.

ActionServerless - Use GitHub Actions to create a serverless service ActionServerless is an action to do some computing and then generate a string/JSO

null 107 Oct 28, 2022
可基于【腾讯云函数】/【GitHub Actions】/【Docker】的每日签到脚本(支持多账号使用)签到列表: |爱奇艺|全民K歌|腾讯视频|有道云笔记|网易云音乐|一加手机社区官方论坛|百度贴吧|Bilibili|V2EX|咔叽网单|什么值得买|AcFun|天翼云盘|WPS|吾爱破解|芒果TV|联通营业厅|Fa米家|小米运动|百度搜索资源平台|每日天气预报|每日一句|哔咔漫画|和彩云|智友邦|微博|CSDN|王者营地|

每日签到集合 基于【腾讯云函数】/【GitHub Actions】/【Docker】的每日签到脚本 支持多账号使用 特别声明: 本仓库发布的脚本及其中涉及的任何解锁和解密分析脚本,仅用于测试和学习研究,禁止用于商业用途,不能保证其合法性,准确性,完整性和有效性,请根据情况自行判断。

null 87 Nov 12, 2022
Automatic SystemVerilog linting in github actions with the help of Verible

Verible Lint Action Usage See action.yml This is a GitHub Action used to lint Verilog and SystemVerilog source files and comment erroneous lines of co

CHIPS Alliance 10 Dec 26, 2022
Automate HoYoLAB Genshin Daily Check-In Using Github Actions

Genshin Daily Check-In ?? Automate HoYoLAB Daily Check-In Using Github Actions KOR, ENG Instructions Fork the repository Go to Settings -> Secrets Cli

Leo Kim 41 Jun 24, 2022
Mini Tool to lovers of debe from eksisozluk (one of the most famous website -reffered as collaborative dictionary like reddit- in Turkey) for pushing debe (Most Liked Entries of Yesterday) to kindle every day via Github Actions.

debe to kindle Mini Tool to lovers of debe from eksisozluk (one of the most famous website -refered as collaborative dictionary like reddit- in Turkey

null 11 Oct 11, 2022
POC de uma AWS lambda que executa a consulta de preços de criptomoedas, e é implantada na AWS usando Github actions.

Cryptocurrency Prices Overview Instalação Repositório Configuração CI/CD Roadmap Testes Overview A ideia deste projeto é aplicar o conteúdo estudado s

Gustavo Santos 3 Aug 31, 2022
Implement SAST + DAST through Github actions

Implement SAST + DAST through Github actions The repository is supposed to implement SAST+DAST checks using github actions against a vulnerable python

Syed Umar Arfeen 3 Nov 9, 2022
GitHub Actions Poll Mode AutoScaler (GAPMAS)

GitHub Actions Poll Mode AutoScaler, or GAPMAS, is a simple tool that helps you run ephemeral GitHub Actions self-hosted runners on your own infrastructure.

Frode Nordahl 4 Nov 4, 2022
Centralized whale instance using github actions, sourcing metadata from bigquery-public-data.

Whale Demo Instance: Bigquery Public Data This is a fully-functioning demo instance of the whale data catalog, actively scraping data from Bigquery's

Hyperquery 17 Dec 14, 2022
A GitHub Actions repo for tracking the dummies sending free money to Alex Jones + co.

A GitHub Actions repo for tracking the dummies sending free money to Alex Jones + co.

Egarok 2 Jul 20, 2022