Microsoft Azure provides a wide number of services for managing and storing data

Overview

Azure SQL Database Pipeline Project

Table of Contents

Overview

Microsoft Azure provides a wide number of services for managing and storing data. One product is Microsoft Azure SQL. Which gives us the capability to create and manage instances of SQL Servers hosted in the cloud. This project, demonstrates how to use these services to manage data we collect from different sources.

Resources

To use this project you will need to install some dependencies to connect to the database. To download the drivers needed go to to Microsoft SQL Drivers for Python. Once you download it, run through the installation process.

Resources - PYODBC with Azure:

If you would like to read more on the topic of using PYODBC in conjunction with Microsoft Azure, then I would refer you to the documentation provided by Microsoft.

Setup

Right now, this project is not planned to be hosted on PyPi so you will need to do a local install on your system if you plan to use it in other scrips you use. First, clone this repo to your local system.

Setup - Local Install:

If you plan to use this project in other projects on your system, I would recommend you either install this project in editable mode or do a local install. For those of you, who want to make modifications to this project. I would recommend you install the library in editable mode.

If you want to install the library in editable mode, make sure to run the setup.py file, so you can install any dependencies you may need. To run the setup.py file, run the following command in your terminal.

pip install -e .

If you don't plan to make any modifications to the project but still want to use it across your different projects, then do a local install.

pip install .

This will install all the dependencies listed in the setup.py file. Once done you can use the library wherever you want.

Setup - Requirement Install:

If you don't plan to use this project in any of your other projects, I would recommend you just install the dependencies by using the requirement.txt file.

pip install --requirement requirements.txt

Usage

Here is a simple example of using the azure_sql_pipeline library to grab a specific database from our SQL Server.

from pprint import pprint
from configparser import ConfigParser
from azure_data_pipeline.client import AzureSQLClient

# Initialize the Parser.
config = ConfigParser()

# Read the file.
config.read('config/config.ini')

# Grab the Azure Management Credentials.
subscription_id = config.get('azure_credentials', 'azure_subscription_id')
tenant_id = config.get('azure_credentials', 'azure_tenant_id')
client_id = config.get('azure_credentials', 'azure_client_id')
client_secret = config.get('azure_credentials', 'azure_client_secret')

# Grab the Azure SQL Server Credentials.
server_username = config.get('server_info', 'administrator_login')
server_password = config.get('server_info', 'administrator_login_password')

# Initialize the client.
azure_pipeline_client = AzureSQLClient(
    client_id=client_id,
    client_secret=client_secret,
    subscription_id=subscription_id,
    tenant_id=tenant_id,
    username=server_username,
    password=server_password
)
You might also like...
 Object recognition using Azure Custom Vision AI and Azure Functions
Object recognition using Azure Custom Vision AI and Azure Functions

Step by Step on how to create an object recognition model using Custom Vision, export the model and run the model in an Azure Function

Azure DevOps Extension for Azure CLI

Azure DevOps Extension for Azure CLI The Azure DevOps Extension for Azure CLI adds Pipelines, Boards, Repos, Artifacts and DevOps commands to the Azur

Azure-function-proxy - Basic proxy as an azure function serverless app

azure function proxy (for phishing) here are config files for using *[.]azureweb

Automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azure

fwhr-calc-website This project is to automatically measure the facial Width-To-Height ratio and get facial analysis results provided by Microsoft Azur

Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about Machine Learning
Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about Machine Learning

Azure Cloud Advocates at Microsoft are pleased to offer a 12-week, 24-lesson curriculum all about Machine Learning

Project 2 for Microsoft Azure on WUT
Project 2 for Microsoft Azure on WUT

azure-proj2 Project 2 for Microsoft Azure on WUT Table of contents Team Tematyka projektu Architektura Opis rozwiązania Demo dzałania The Team Krzyszt

Bot made with Microsoft Azure' cloud service
Bot made with Microsoft Azure' cloud service

IttenWearBot Autori: Antonio Zizzari Simone Giglio IttenWearBot è un bot intelligente dotato di sofisticate tecniche di machile learning che aiuta gli

Two factor authentication system using azure services and python language and its api's
Two factor authentication system using azure services and python language and its api's

FUTURE READY TALENT VIRTUAL INTERSHIP PROJECT PROJECT NAME - TWO FACTOR AUTHENTICATION SYSTEM Resources used: * Azure functions(python)

Official s3cmd repo -- Command line tool for managing Amazon S3 and CloudFront services

S3cmd tool for Amazon Simple Storage Service (S3) Author: Michal Ludvig, [email protected] Project homepage (c) TGRMN Software and contributors S3tools

AML Command Transfer. A lightweight tool to transfer any command line to Azure Machine Learning Services

AML Command Transfer (ACT) ACT is a lightweight tool to transfer any command from the local machine to AML or ITP, both of which are Azure Machine Lea

This program uses trial auth token of Azure Cognitive Services to do speech synthesis for you.

🗣️ aspeak A simple text-to-speech client using azure TTS API(trial). 😆 TL;DR: This program uses trial auth token of Azure Cognitive Services to do s

A simple hosts picker for Microsoft Services

A simple Python scrip for you to select the fastest IP for Microsoft services.

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

Nasdaq Cloud Data Service (NCDS) Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and ot

Vehicle Identification Speed Detection (VISD) extracts vehicle information like License Plate number, Manufacturer and colour from a video and provides this data in the form of a CSV file
Vehicle Identification Speed Detection (VISD) extracts vehicle information like License Plate number, Manufacturer and colour from a video and provides this data in the form of a CSV file

Vehicle Identification Speed Detection (VISD) extracts vehicle information like License Plate number, Manufacturer and colour from a video and provides this data in the form of a CSV file. VISD can also perform vehicle speed detection on a video. All these features of VSID are provided to the user using a Web Application which is created using Flask

The Timescale NFT Starter Kit is a step-by-step guide to get up and running with collecting, storing, analyzing and visualizing NFT data from OpenSea, using PostgreSQL and TimescaleDB.

Timescale NFT Starter Kit The Timescale NFT Starter Kit is a step-by-step guide to get up and running with collecting, storing, analyzing and visualiz

Convert any-bit number to decimal number and vise versa.

2deci Convert any-bit number to decimal number and vise versa. --bit n to set bit to n --exp xxx to set expression to xxx --r to run reversely (from d

VG-Scraper is a python program using the module called BeautifulSoup which allows anyone to scrape something off an website. This program lets you put in a number trough an input and a number is 1 news article.

VG-Scraper VG-Scraper is a convinient program where you can find all the news articles instead of finding one yourself. Installing [Linux] Open a term

A Tool that provides automatic kerning for ligature based OpenType fonts in Microsoft Volt
A Tool that provides automatic kerning for ligature based OpenType fonts in Microsoft Volt

Kerning A Tool that provides automatic kerning for ligature based OpenType fonts in Microsoft Volt There are three stages of the algorithm. The first

Owner
Riya Vijay Vishwakarma
learner
Riya Vijay Vishwakarma
Build, test, deploy, iterate - Dev and prod tool for data science pipelines

Prodmodel is a build system for data science pipelines. Users, testers, contributors are welcome! Motivation · Concepts · Installation · Usage · Contr

Prodmodel 53 Nov 29, 2022
functional data manipulation for pandas

pandas-ply: functional data manipulation for pandas pandas-ply is a thin layer which makes it easier to manipulate data with pandas. In particular, it

Coursera 188 Nov 24, 2022
Clean APIs for data cleaning. Python implementation of R package Janitor

pyjanitor pyjanitor is a Python implementation of the R package janitor, and provides a clean API for cleaning data. Why janitor? Originally a port of

Eric Ma 1.1k Jan 1, 2023
A Python toolkit for processing tabular data

meza: A Python toolkit for processing tabular data Index Introduction | Requirements | Motivation | Hello World | Usage | Interoperability | Installat

Reuben Cummings 401 Dec 19, 2022
Microsoft Azure CLI - Azure Command-Line Interface

A great cloud needs great tools; we're excited to introduce Azure CLI, our next generation multi-platform command line experience for Azure.

Microsoft Azure 3.4k Dec 30, 2022
A Python SDK for connecting devices to Microsoft Azure IoT services

V2 - We are now GA! This repository contains code for the Azure IoT SDKs for Python. This enables python developers to easily create IoT device soluti

Microsoft Azure 381 Dec 30, 2022
A terminal application for managing images and artifacts in Azure Container Registry.

acr-browser acr-browser is a terminal-based user interface for managing container images and artifacts in Azure Container Registry. ?? This project ow

Sam Dobson 5 Jul 30, 2022
machine learning model deployment project of Iris classification model in a minimal UI using flask web framework and deployed it in Azure cloud using Azure app service

This is a machine learning model deployment project of Iris classification model in a minimal UI using flask web framework and deployed it in Azure cloud using Azure app service. We initially made this project as a requirement for an internship at Indian Servers. We are now making it open to contribution.

Krishna Priyatham Potluri 73 Dec 1, 2022