Distributed, blockchain based hashtables middleware for deduplication of file uploads to the cloud

Overview

distributed-blockchain-based-secure-file-dedupe

  • Searching is Distributed, Block and Access List for each upload is unique and it is stored in a single VM to reduce overhead in the metadata (blockchain based).

  • Instructions for 4 nodes (including a storage node) docker swarm setup:

    1. get.docker.com .. install docker on all VMs (run both commands under "This script is meant for the quick & easy installation")

    2. ssh into all VMs with sudo access (or add docker usergroup)

    3. Run "docker swarm init" on the leader node.

    4. Run "docker swarm join-token manager", copy paste the output to all other VMs, so that all other nodes will be added to the swarm as manager.

    5. To confirm all the nodes present in swarm, run "docker node ls"

    6. Create a common network (attachable) among all nodes on the swarm using "docker network create --driver overlay --attachable dedupe"

    7. To confirm, run "docker network ls" on all the VMs and check the scope(==swarm) and driver(==overlay)

    8. Build and push the image "docker-compose build", "docker-compose push".

    9. Pull the image for dedupe nodes separately "docker image pull abhishekvtangod/blockdedupe"

    10. Pull the image for cloud node "docker image pull abhishekvtangod/blockdedupe_cloud"

    11. Run the containers on all VMs.

      • docker container run -d --name node1 --network dedupe --env PYTHONUNBUFFERED=1 --env node_id=1 -p 5001:5000 --hostname localhost abhishekvtangod/blockdedupe
      • docker container run -d --name node2 --network dedupe --env PYTHONUNBUFFERED=1 --env node_id=2 -p 5002:5000 --hostname localhost abhishekvtangod/blockdedupe
      • docker container run -d --name node3 --network dedupe --env PYTHONUNBUFFERED=1 --env node_id=3 -p 5003:5000 --hostname localhost abhishekvtangod/blockdedupe
      • docker container run -d --name cloud --network dedupe --env PYTHONUNBUFFERED=1 -p 5000:5000 --hostname localhost abhishekvtangod/blockdedupe_cloud
    12. Access all the urls from individual ips of VM with appropriate ports(change port mappings according to your ease and avoid confusions).

Pure Blockchain Implementation: https://github.com/lanxeon

You might also like...
A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

Smart Contacts Alogrand Smart Contracts using Choice Coin. Read Docs for how to implement Algogenous Smart Contracts for your own applications. Smart

A repository for Algogenous Smart Contracts created on the Algorand Blockchain.

Smart Contacts This Repository is dedicated to code for Alogrand Smart Contracts using Choice Coin. Read Docs for how to implement Algogenous Smart Co

High Performance Blockchain Deserializer

bitcoin_explorer is an efficient library for reading bitcoin-core binary blockchain file as a database (utilising multi-threading).

Python implementation of a blockchain.

The goal of this project is to explain and to make clearer how is a blockchain structured at the very core. It's not built with the intention to replicate an advanced blockchain like Bitcoin or Ethereum

A little side-project API for me to learn about Blockchain and Tokens

BlockChain API I built this little side project to learn more about Blockchain and Tokens. It might be maintained and implemented to other projects bu

A workshop to build an NFT smart contract on the polygon blockchain
A workshop to build an NFT smart contract on the polygon blockchain

Polygon NFT Workshop This is an interactive workshop that guides you through the steps to deploy an NFT smart contract on the Polygon blockchain. By t

Blockchain Python Implementation

Blockchain Python Implementation

Accounting Cycle Program with Blockchain Component

In the first iteration of my accounting cycle program, I talked about adding in a blockchain component that allows the user to verify the inegrity of

Best blockchain in the world

alphachain Best blockchain in the world!!! Can be used to implement Layer 2 cryptocurrency protocol just click alphachain.py and it will execute autom

Owner
Abhishek Tangod
Like what you do; Stay passionate, focused and motivated :)
Abhishek Tangod
J. Brandon Walker 1 May 13, 2022
Alpkunt 9 Sep 9, 2022
An BlockChain Based solution for storing the medical records

Blockchain-based Medical Records ?? Abstract Blockchain has the ability to keep an incorruptible, decentralized, and transparent log of all patient da

Yuvraj Singh Deora 3 Jan 14, 2022
Token drop template on Tezos blockchain, based on Merkle Tree Distribution mechanism.

?? Token Drop Template This is a template to perform token drops efficiently on Tezos blockchain. The drop is handled using Merkle Tree Distribution m

Anshu Jalan 5 Oct 11, 2022
Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions.

Blockchain ETL 2.3k Jan 1, 2023
Learn Blockchains by Building One, A simple Blockchain in Python using Flask as a micro web framework.

Blockchain ✨ Learn Blockchains by Building One Yourself Installation Make sure Python 3.6+ is installed. Install Flask Web Framework. Clone this repos

Vaibhaw 46 Jan 5, 2023
Tools for running airdrop and token distribution campaigns on the Solana blockchain.

Overview This repository contains some of the scripts we have used for running our airdrop campaigns and other distributions. Initially, all of these

null 147 Nov 17, 2022
seno-blockchain is just a fork of Chia, designed to be efficient, decentralized, and secure

seno-blockchain https://seno.uno Seno is just a fork of Chia, designed to be efficient, decentralized, and secure. Here are some of the features and b

Denis Erygin 27 Jul 2, 2022
Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) consensus algorithm.

Ceres Combine-Harvester Ceres is a combine harvester designed to harvest plots for Chia blockchain and its forks using proof-of-space-and-time(PoST) c

null 38 Nov 14, 2022
blockchain address database

Blockchain Address Ownership Database The database is in data/addresses.db This is a SQLite database of addresses from several blockchains. It's obtai

null 37 Nov 26, 2022