InfraGenie is allows you to split out your infrastructure project into separate independent pieces, each with its own terraform state.

Overview

🧞 InfraGenie

InfraGenie is allows you to split out your infrastructure project into separate independent pieces, each with its own terraform state. This is done using a pattern similar to dependency injection in programming languages, but under the hood it uses terraform data blocks.

infragenie drawio (3)

Why this pattern?

There are several reasons why you would want to adopt this pattern:

  1. Flexibility in customising your infrastructure. We know that for a single project the infrastructure might change from one environment to another. For example if you are using an Elasticsearch service in production you might use a self-hosted version in dev to save costs. InfraGenie makes this process very easy
  2. Split your terraform state. By splitting your state accross several modules you can run several applies in parallel. It makes the terraform refresh faster. It also makes your applies safer since if some apply goes haywire it will only affect the resources in the current module.

What about Terraform modules?

Terraform modules can allow you to acheive some flexibiliy buy you still share state across the entire project. It is more difficult to make part of a module optional. The count syntax and similar foreach declarative statements in terraform can be confusing if you are not used to the declarative style it uses.

How it works

To use infragenie you simply create a file called genie.hcl in the root of your project and use it to define your pipeline.

# genie.hcl


# unified variables for use in all pipelines
variables {
  project_name = "myproj123"
  environment = "dev"
  region = "us-east-1"
}

# resource injections definition
inject {
  main_vpc = {
    source = vpc.aws_vpc.vpc # source can come from any of the pipeline steps
  }
}

pipeline {
  steps = [
    {
      name = "vpc"
      description = "creates a vpc and 2 subnets"
      source = "./vpc"
    },
    {
      name = "ecs"
      description = "creates an ecs cluster and SG"
      source = "./ecs"
    },

  ]
}

Now with this genie file you can use the vpc in any of your modules as a data definition:

# ecs/main.tf

resource "aws_security_group" "ecs_service_sg" {
  # using global variables
  name_prefix = "${var.project_name}-${var.environment}"
  # using vpc from other module as data block
  vpc_id      = data.aws_vpc.main_vpc.id
}

Quickstart

You can install InfraGenie CLI via pip

pip install infragenie

Usage

You can use the examples to try out infragenie:

Clone the repository:

git clone https://github.com/diggerhq/infragenie
cd infragenie/examples/ecs_fargate

export your AWS keys:

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

Use igm to apply the example:

igm apply

take note of the generated .infragenie directory along with all the generated data. After exploration you can destroy the resources with:

igm destroy

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please start the project if you think other people will also find it useful.

License

MIT

Comments
  • Fix README content and its formatting

    Fix README content and its formatting

    Hello there, I noticed that there were some minor formatting issues in the README and also that the initial sentence didn't quite read well so I decided to modify it a little bit.

    I welcome feedback and amendment request if necessary.

    Cheers, Debo

    opened by debo 2
  • bug fixes and feature inclusions

    bug fixes and feature inclusions

    This should take care of issues 1-4

    • no longer failing if .infragenie dir not created before app init
    • now cataloging every resource, instead of just the first one
    • auto-approving apply/delete for better automation
    • writing ec2 ids as "instance_id"
    • destroy now removes stale state dirs
    • destroy no longer fails if pipeline source dirs aren't in .infragenie
    • destroy can now destroy individual modules
    • destroy now destroys all modules in reverse creation order
    opened by cringdahl 0
  • 'aws_instance' id doesn't populate genie_output or genie_input

    'aws_instance' id doesn't populate genie_output or genie_input

    The ec2 instance successfully creates; however, the code just doesn't seem to create the id 'output' or 'data'. There's no code dump or failure exhibited.

    Has aws_instance been tested? What do you need for troubleshooting?

    opened by cringdahl 0
  • 'igm destroy' example ecs_fargate fails:

    'igm destroy' example ecs_fargate fails: "The vpc has dependencies and cannot be deleted"

    Everything else in the 'vpc' module is deleted, but when the VPC itself wants to go, this happens:

    â•·
    │ Error: Error deleting VPC: DependencyViolation: The vpc 'vpc-0bd9d95f8960cf2af' has dependencies and cannot be deleted.
    │ 	status code: 400, request id: 5bc44443-a24d-41bd-b2bb-03b849598576
    │ 
    │ 
    ╵
    Traceback (most recent call last):
      File "/usr/local/bin/igm", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/infragenie/genie.py", line 178, in destroy
        destroyInfra(modulePath, pipelineName)
      File "/usr/local/lib/python3.9/site-packages/infragenie/genie.py", line 99, in destroyInfra
        subprocess.run(["terraform", "destroy", "-state", f"../../{pipelineName}.terraform.tfstate"], check=True)
      File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
        raise CalledProcessError(retcode, process.args,
    subprocess.CalledProcessError: Command '['terraform', 'destroy', '-state', '../../vpc.terraform.tfstate']' returned non-zero exit status 1.```
    
    This prevents the 'ecs' module assets from deleting. It's possible these need to be deleted first? Reverse the pipeline steps to delete?
    opened by cringdahl 0
  • initial use fails;

    initial use fails; "No such file or directory: '.infragenie'"

    If you run igm without first creating .infragenie, igm bails

    17:43 $ igm apply
    🧛 welcome to InfraGenie CLI! 😄
    parsing genie.hcl file ...
    found the following settings:
    {
        'variables': [
            {
                'app': 'apptimes',
                'environment': 'dev',
                'region': 'us-east-1',
                'appid_or_sso': 'funtimes',
                'amazon_linux_extras_install': 'epel',
                'yum_install': 'nano sysstat tinyfugue vim'
            }
        ],
        'inject': [{'biggles': {'source': '${ec2.aws_instance.this}'}}],
        'pipeline': [
            {
                'steps': [
                    {'name': 'ec2', 'description': 'creates an ecs cluster and SG', 'source': './ec2'},
                    {'name': 'ami', 'description': 'creates a vpc and 2 subnets', 'source': './ami'}
                ]
            }
        ]
    }
    Rendering data outputs
    biggles ec2.aws_instance.this
    rendering terraform outputs
    Traceback (most recent call last):
      File "/usr/local/bin/igm", line 8, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/infragenie/genie.py", line 141, in apply
        shutil.rmtree(".infragenie")
      File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 709, in rmtree
        onerror(os.lstat, path, sys.exc_info())
      File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 707, in rmtree
        orig_st = os.lstat(path)
    FileNotFoundError: [Errno 2] No such file or directory: '.infragenie'
    
    
    opened by cringdahl 0
Owner
Digger
Digger
IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

invasion 3 Feb 18, 2022
Passive TCP/IP Fingerprinting Tool. Run this on your server and find out what Operating Systems your clients are *really* using.

Passive TCP/IP Fingerprinting This is a passive TCP/IP fingerprinting tool. Run this on your server and find out what operating systems your clients a

Nikolai Tschacher 158 Dec 20, 2022
NSX-T infrastructure as code - SDDC deployment

Deploy NSX-T Infrastructure - Simple Topology by Nicolas MICHEL @vpackets / LinkedIn Introduction The purpose of this entire repository is to automate

null 21 Nov 28, 2022
IPE is a simple tool for analyzing IP addresses. With IPE you can find out the server region, city, country, longitude and latitude and much more in seconds.

IPE is a simple tool for analyzing IP addresses. With IPE you can find out the server region, city, country, longitude and latitude and much more in seconds.

Paul 0 Jun 11, 2022
PcapConverter - A project for generating 15min frames out of a .pcap file containing network traffic

CMB Assignment 02 code + notebooks This is a project for containing code for the

Yannik S 2 Jan 24, 2022
Aiotor - a pool of proxies, shifting on each request

Aiotor - a pool of proxies, shifting on each request

Leon 32 Dec 26, 2022
A TCP Chatroom built with python and TCP/IP sockets, consisting of a server and multiple clients which can connect with the server and chat with each other.

A TCP Chatroom built with python and TCP/IP sockets, consisting of a server and multiple clients which can connect with the server and chat with each other. It also provides an Admin role with features including kicking and baning of users.

null 3 May 22, 2022
Takes a file of hosts or domains and outputs the IP address of each host/domain in the file.

Takes a file of hosts or domains and outputs the IP address of each host/domain in the file. Installation $ git clone https://github.com/whoamisec75/i

whoami security 2 May 10, 2022
Socket programming is a way of connecting two nodes on a network to communicate with each other

Socket Programming in Python Socket programming is a way of connecting two nodes on a network to communicate with each other. One socket(node) listens

Janak raikhola 1 Jul 5, 2022
Way find out if DNS is down or your instance

DNS-PING Way to find out if DNS is down or your instance Problem: At times it happens that DNS provider services of a website URL is down and so to re

Giten Mitra 4 Nov 18, 2022
Python Scrcpy Client - allows you to view and control android device in realtime

Python Scrcpy Client This package allows you to view and control android device in realtime. Note: This gif is compressed and experience lower quality

LengYue 126 Jan 2, 2023
🎥 PYnema is a simple UDP server written in python, allows you to watch downloaded videos.

?? PYnema is a simple UDP server written in python, allows you to watch downloaded videos.

Jan Kupczyk 1 Jan 16, 2022
A Python module that allows you to create and use simple sockets.

EasySockets A Python module that allows you to create and use simple sockets. Installation The easysockets module can be installed using pip. pip inst

Matthias Wijnsma 2 Jan 16, 2022
A Python Packages to make own chat room

Chathon A Python packages for make own chat room Install PyPI pip install chathon

null 1 Dec 10, 2021
Find information about an IP address, such as its location, ISP, hostname, region, country, and city.

Find information about an IP address, such as its location, ISP, hostname, region, country, and city. An IP address can be traced, tracked, and located.

Sachit Yadav 2 Jul 9, 2022
Script and library to wait for a DNS authority server to get its configuration.

DNSWait dnswait is a small script to wait for the "propagation" of a namserver configuration. Installing It's as easy as: python -m pip install dnswai

Julien Palard 14 Jan 17, 2022
This application aims to read all wifi passwords and visualizes the complexity in graph formation by taking into account several criteria and help you generate new random passwords.

This application aims to read all wifi passwords and visualizes the complexity in graph formation by taking into account several criteria and help you generate new random passwords.

Njomza Rexhepi 0 May 29, 2022
Out-of-box Python RPC framework

typed-jsonrpc Out-of-box Python RPC framework. WIP. Make LSP easy for everyone. The conception of final usage: from typed_jsonrpc import * ls = Langu

Taine Zhao 4 Dec 28, 2021
This Tool can help enginners and biggener in network, the tool help you to find of any ip with subnet mask that can calucate them and show you ( Availble IP's , Subnet Mask, Network-ID, Broadcast-ID )

This Tool can help enginners and biggener in network, the tool help you to find of any ip with subnet mask that can calucate them and show you ( Availble IP's , Subnet Mask, Network-ID, Broadcast-ID )

null 12 Dec 13, 2022