StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html. Questions? https://forum.stackstorm.com/.

Overview

StackStorm

StackStorm is a platform for integration and automation across services and tools, taking actions in response to events. Learn more at www.stackstorm.com.

Build Status Travis Integration Tests Status Packages Build Status Codecov CII Best Practices Python 3.6 Apache Licensed Join our community Slack Forum


TL;DR

StackStorm Overview

StackStorm 5 min Intro Video

About

StackStorm is a platform for integration and automation across services and tools. It ties together your existing infrastructure and application environment so you can more easily automate that environment -- with a particular focus on taking actions in response to events.

StackStorm helps automate common operational patterns. Some examples are:

  • Facilitated Troubleshooting - triggering on system failures captured by Nagios, Sensu, New Relic and other monitoring, running a series of diagnostic checks on physical nodes, OpenStack or Amazon instances, and application components, and posting results to a shared communication context, like Slack or JIRA.
  • Automated remediation - identifying and verifying hardware failure on OpenStack compute node, properly evacuating instances and emailing VM about potential downtime, but if anything goes wrong - freezing the workflow and calling PagerDuty to wake up a human.
  • Continuous deployment - build and test with Jenkins, provision a new AWS cluster, turn on some traffic with the load balancer, and roll-forth or roll-back based on NewRelic app performance data.

StackStorm helps you compose these and other operational patterns as rules and workflows or actions; and these rules and workflows - the content within the StackStorm platform - are stored as code which means they support the same approach to collaboration that you use today for code development and can be shared with the broader open source community via StackStorm Exchange.

Who is using StackStorm?

See the list of known StackStorm ADOPTERS.md and Thought Leaders.

How it works

StackStorm architecture

StackStorm architecture diagram

StackStorm plugs into the environment via an extensible set of adapters: sensors and actions.

  • Sensors are Python plugins for inbound integration that watch for events from external systems and fire a StackStorm trigger when an event happens.

  • Triggers are StackStorm representations of external events. There are generic triggers (e.g., timers, webhooks) and integration triggers (e.g., Sensu alert, JIRA issue updated). A new trigger type can be defined by writing a sensor plugin.

  • Actions are StackStorm outbound integrations. There are generic actions (SSH, HTTP request), integrations (OpenStack, Docker, Puppet), or custom actions. Actions are either Python plugins, or any scripts, consumed into StackStorm by adding a few lines of metadata. Actions can be invoked directly by user via CLI, API, or the web UI, or used and called as part of automations - rules and workflows.

  • Rules map triggers to actions (or to workflows), applying matching criterias and map trigger payload data to action inputs.

  • Workflows stitch actions together into "uber-actions", defining the order, transition conditions, and passing context data from one action to the next. Most automations are multi-step (eg: more than one action). Workflows, just like "atomic" actions, are available in the action library, and can be invoked manually or triggered by rules.

  • Packs are the units of content deployment. They simplify the management and sharing of StackStorm pluggable content by grouping integrations (triggers and actions) and automations (rules and workflows). A growing number of packs is available on the StackStorm Exchange. Users can create their own packs, share them on GitHub, or submit them to the StackStorm Exchange organization.

  • Audit trail is the historical list of action executions, manual or automated, and is recorded and stored with full details of triggering context and execution results. It is is also captured in audit logs for integrating with external logging and analytical tools: LogStash, Splunk, statsd, or syslog.

StackStorm is a service with modular architecture. It is comprised of loosely coupled microservice components that communicate over a message bus, and scales horizontally to deliver automation at scale. StackStorm has a full REST API, CLI client, and web UI for admins and users to operate it locally or remotely, as well as Python client bindings for developer convenience.

StackStorm is an established project and remains actively developed by a broad community.

Documentation

Additional documentation, including installation proceduces, action/rule/workflow authoring, and how to setup and use triggers/sensors can be found at https://docs.stackstorm.com.

Hacking / Contributing

To set up a development environment and run StackStorm from sources, follow these instructions.

For information on how to contribute, our style guide, coding conventions and more, please visit the Development section in our documentation.

Security

If you believe you found a security issue or a vulnerability, please send a description of it to our private mailing list at info [at] stackstorm [dot] com.

Once you've submitted an issue, you should receive an acknowledgment from one our of team members in 48 hours or less. If further action is necessary, you may receive additional follow-up emails.

For more information, please refer to https://docs.stackstorm.com/latest/security.html

Copyright, License, and Contributor Agreement

Copyright 2020 The StackStorm Authors. Copyright 2019 Extreme Networks, Inc. Copyright 2014-2018 StackStorm, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

http://www.apache.org/licenses/LICENSE-2.0

By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the project, present and future, pursuant to the license of the project.

Comments
  • Optimize storage (serialization and de-serilization) of very large dictionaries inside MongoDB

    Optimize storage (serialization and de-serilization) of very large dictionaries inside MongoDB

    This pull request builds on top of various research I did in other PRs and issues in the past (#4837, #4838, etc).

    The goal is to speed up serialization and de-serialization of very large dictionaries (e.g.large results produced by various actions) inside MongoDB.

    Right now we store those dictionaries as native MongoDB dictionaries, but this means we need to escape the keys, since they can't contain $ and . which are special characters.

    This escaping is inefficient and wasteful with large (and nested) dictionaries.

    Proposed Implementation / Improvement

    This pull request tries to optimize that by serializing those dictionaries as JSON instead.

    Keep in mind that this is fine, since we always treat those values as opaque strings anyway.

    In fact, a future improvement would be to have a result database object per runner, this way we could utilize a simple "string" type for the actual result, but that's a much more involved change.

    Some actual numbers with synthetic dataset are available in other issues mentioned above. Actual numbers with production real-life data are to follow once we deploy this to CI/CD server.

    The change is fully backward compatible, but it's behind a feature flag (opt-in) until we are really sure after running it in production for a while with diverse datasets it doesn't make some corner / edge case worse or similar.

    Related optimization I talked about to @m4dcoder is getting rid of using EscapedDict DB field type in scenarios where keys can't contain special characters (e.g. potentially various workflow related models).

    TODO

    • [x] Update all the affected models to use this field type
    • [x] Tests
    • [x] Deploy it on CI/CD for a month, observe performance, memory usage, etc
    • [x] Add optional migration script for affected models - future
    • [x] Document change in upgrade notes and point users to the optional migration script - future
    • [x] Add some docs for developers on the new DB format
    • [x] Update execution list / get command to display execution run duration which includes database write time not just action run time
    • [x] Document the change
    performance workflows: orquesta mongodb service: workflow engine size/XXL service: action runner service: api 
    opened by Kami 50
  • Add migration script for migrating old execution field data to the new format

    Add migration script for migrating old execution field data to the new format

    This pull request adds a simple migration script which migrations data for executions which utilize old and slow + inefficient fielld type to the new type.

    I tested it locally and it works fine (it's also a bit of a PITA to manually test, easiest is to just create MongoDB snapshot with old version of st2 and re-use that with st2 master).

    Implementation Details

    #4846 introduced a new field types which is up to 10-30x faster when saving and retrieving objects with larger values.

    In that PR I wasn't a massive fan of a migration script - yes I could add it, but I already spent many, many days on those performance optimization changes and I think migration script adds little value yet the effort is not totally trivial - yes the code changes mostly is, but things are not just a code change - it includes test cases, documentation, upgrade notes, etc and that takes quite some effort.

    The migration script doesn't provide tons of value because it only works in old executions and most users only care about new and recent executions so if when viewing data for some old execution it takes longer to load it's not the end of the world.

    I would be all of migration if that would actually provide more value - e.g. reduce maintenance burden / amount of code we need to maintain, but that is not the case and it likely won't change any time soon. We still utilize those two field types in many other places and migration script step is optional and manual and we can't really force users to run them and assume by the next release everyone has done it so we can just remove that code.

    For now, I only implemented it for execution related objects. Technically we could also do it for workflow, trigger instance and rule related objects, but that would require even more work and some more "work around" to determine if specific object in database utilizes all type or now - it's possible to detect that, but it's not totally trivial and just re-writing all objects sames wasteful.

    Complete contributions (with tests) for other models are also welcome (especially in this case where the change itself will likely save many 10's of thousands and likely even much more $ per month in terms of CPU utilization across all the large StackStorm installations so contributing such a small change seems worth trade off for such a big improvement :)).

    TODO

    • [x] Unite tests (will be a bit of a PITA, but it's manageable)
    • [x] Upgrade notes entry - https://github.com/StackStorm/st2/pull/5255
    performance external dependency mongodb size/L migrations 
    opened by Kami 40
  • Modify API to remove related files on delete of actions/workflows

    Modify API to remove related files on delete of actions/workflows

    This change is to remove related files of actions/workflows from disk when API called from CLI and web UI (https://github.com/StackStorm/st2web/pull/898). Currently when this delete API is invoked then this only de-register the actions from database and when user reloads the st2 then these actions get registered into database again.

    size/L 
    opened by mahesh-orch 34
  • Python3.8 on Ubuntu 20.04 (requiring MongoDB 4.4)

    Python3.8 on Ubuntu 20.04 (requiring MongoDB 4.4)

    Work in Progress

    Add support for Python3.8 on Ubuntu 20.04. To support this configuration, it implicitly includes MongoDB 4.4 unless we consider standalone instances of StackStorm can no longer host the mongodb instance locally for Ubuntu 20.04.

    • Bump eventlet/greenlet to latest to benefit from threading fix for python3.7+
    • Bump pymongo/mongoengine to support Mongo4.4 which is the only version packaged for Ubuntu 20.04.
    • Added Python3.8 into tox tests.

    https://github.com/StackStorm/discussions/issues/68

    feature deployment mongodb size/XL OS support 
    opened by nzlosh 32
  • Add a new standalone st2-pack-install CLI command for installing a pack

    Add a new standalone st2-pack-install CLI command for installing a pack

    This pull request adds a new independent and standalone st2-pack-install CLI command for installing a pack and setting up pack virtual environment.

    End result is the same as running st2 pack install <foo> minus the content registration step.

    This command is fully standalone and only requires Python, pip, st2common PyPi package and git binary to be installed on the system where it's used. It doesn't need / require database (MongoDB) and message bus access (RabbitMQ).

    This command is designed to be used in environments where all the resources (packs) are baked into the base VM / container image which is then deployed.

    Keep in mind that the actual content still needs to be registered with StackStorm at some later point when a cluster is deployed and when MongoDB and RabbitMQ are up (st2ctl reload --register-all which is also an idempotent operation).

    NOTE: This change required some long needed refactoring - moved a bunch of code outside of the action which makes it re-usable elsewhere and also easier to test.

    Usage:

    ./st2common/bin/st2-pack-install libcloud
    ./st2common/bin/st2-pack-install doesnt-exist
    ./st2common/bin/st2-pack-install libcloud==9.9.9
    ./st2common/bin/st2-pack-install libcloud --debug
    ./st2common/bin/st2-pack-install libcloud --debug
    ./st2common/bin/st2-pack-install libcloud xml=9.9.9 csv
    ./st2common/bin/st2-pack-install libcloud doesntexist csv xml=9.9.9
    

    Resolves #3912.

    K8s Docker 
    opened by Kami 29
  • Execution stuck in

    Execution stuck in "Running" state

    I recently ran into a problem which caused the state of the execution to incorrectly report "Running". Looking at the logs, the actions clearly fails. There is no way to cancel it or remove it via the UI or the CLI. The action never seems to time out.

    It happened when the result of the action contained some non-utf8 characters. I posted the action runner log at here.

    Some background info: running stackstorm 0.11.6 on CentOS 7 with the latest python available on the OS.

    I was able to reproduce this multiple times. The action should fail with the error message seen in the logs.

    bug WIP 
    opened by felin-arch 27
  • st2client : Pack Install is hanging with HTTPS

    st2client : Pack Install is hanging with HTTPS

    Hello Stackstorm team,

    We're noticing an issue with "st2 pack install file://$PWD" command with HTTPS endpoints.

    For eg: if we set the below environment variables and try to install a pack , it works without any issues.

    export ST2_API_URL=http://127.0.0.1:9101 export ST2_STREAM_URL=http://127.0.0.1:9102 export ST2_AUTH_URL=http://127.0.0.1:9100 export ST2_BASE_URL=http://127.0.0.1

    But when we try to install a pack with the below environment variables wherein we route every request via Nginx reverse proxy/load balancer, the st2client fails to show the progress.

    export ST2_BASE_URL=https://127.0.0.1 export ST2_AUTH_URL=https://127.0.0.1/auth export ST2_API_URL=https://127.0.0.1/api export ST2_STREAM_URL=https://127.0.0.1/stream export ST2_CACERT=/etc/ssl/st2/st2.crt

    Basically the pack install will show the progress like but the client won't exit and it would simply hang forever. But the pack install action chain would have really executed and completed without any issues.

    image

    Appreciate if any pointers to resolved this issue.

    Regards Harish

    bug status:need more info component:st2client 
    opened by harishgopalan 26
  • [WIP] [RFC] Sensor and Python runner action sandboxing and isolation

    [WIP] [RFC] Sensor and Python runner action sandboxing and isolation

    This pull request introduces sensor sandboxing and isolation. Sensors are sandboxed by running them in a separate process. The sensor process uses Python binary from the virtual environment which is specific to the pack to which the sensor belongs.

    Related pull request which includes "packs" command changes is #778.

    (Note: Currently this is just a prototype and not everything is hooked up and working yet).

    Benefits

    In addition to the "common" sandboxing / isolation benefits, this approach also has some other benefits:

    1. It's easier to scale and run sensors on multiple servers.

    Sensor wrapper is mostly self-sustaining and doesn't depend on the whole reactor process running.

    This means only the pack content and sensor wrapper needs to be located on the remote host where we want to run the sensors.

    1. Looser coupling.

    Sensors are less coupled to the container process and other st2 components.

    They only communicate with st2 using the API endpoint which is used to dispatch the triggers. Previously, they were coupled to the container process.

    1. Easier to debug and test the sensors.

    User can run and test the sensors directly using the sensor wrapper script.

    Downsides

    One of the main downsides of this approach is increased resource consumption (memory and CPU).

    I personally view this more as a trade off than a downside (we trade security and isolation for some additional resource consumption).

    This should also be a less of an issue when we better support "scale-out" deployments (running components on multiple servers).

    TODO

    • [x] Use system python binary and virtual environment for actions and sensors which belong to system packs
    • [x] Implement bi-direction communication using message in the sensor wrapper (uses for create, update, delete handler methods)
    • [x] Implement trigger event dispatcher in the sensor wrapper via a message bus using a new queue - Kami
    • [x] Hook up TriggerDispatcher to listen for events on the new queue and handle them - Manas
    • [x] Add support for sensor meta data files (similar to action meta data file). Those files should define trigger_types so we avoid chicken and the egg problem. This also means updating sensor registration process, etc. - Kami
    • [ ] Update all the core sensors so they still work (notably web hook one needs to be updated to use new queue dispatch approach) - lakshmi
    • [x] Move scheduling and poll functionality from sensor classes to the sensor wrapper. This way we narrow sensor class responsibilities to fetching data and dispatching triggers. And we also need to do that since we need to use eventlet.sleep inside the schedule loop to prevent blocking and prevent other code from running. Poll interval should be defined in the sensor metadata file. - Kami
    • [ ] Documentation updates for the sensor metadata changes
    • [x] Hook up container manager to use the new MultiProcessSensorContainer
    • [x] Capture stdout and stderr of the child process in the process sensor container
    • [x] Sandboxing for Python runner actions
    • [x] Update affected unit tests - Kami
    • [x] Update code to use "packs_pase_path` configuration value once #780 has been merged - Kami
    • [ ] End to end testing - everyone
    • [x] Decide on the name for "Dispatcher" class and update affected code - Kami

    Open questions / To discuss

    Here are some things which still need to be discussed. Some of them are just things I have observed while working on this change and don't need to be addressed immediately.

    1. Sensor communication with the container process (aka dispatching triggers)

    Lakshmi and I have discussed this on Slack. We both agreed that having sensors dispatch trigger events by sending a request to the API is a good approach.

    Using this approach means we don't need a bi-directional communication between the container and sensor and the sensor wrapper process is more or less self-sustaining.

    For this to work, we need to add special API endpoint which takes a trigger and dispatches a payload. We can't use the existing webhook sensor anymore. The existing webhook sensor assumes it has direct access to the container service which is not true anymore.

    1. API authentication

    This is something we probably don't need and want to tackle right now since all the sensor processes will run on the same server as the other st2 components.

    Eventually though (when we scale out and support running components on multiple servers), we should figure out how sensors should authenticate to the API when they dispatch a trigger. One approach which I think is reasonable is generating a temporary and limited access API key for each sensor process.

    This key would have limited access (only allow to post triggers to a special endpoint) and be revoked when the sensor process exits.

    1. Running multiple instances of the same sensor

    Currently, you can only run one instance of a particular sensor class.

    This works fine for now, but I do see a use case where you would want to run multiple instances of the same sensor with different configuration options (e.g. you want to monitor multiple JIRA instances, etc.).

    This is a bigger task and would require quite a bit of refactoring so it's not something we probably want to do right now, but it's just something to keep in mind and think about.

    To support this use case, we would need to (among other things) allow user to use different config files and the sensor class would need to expose a method for returning it's ID (e.g. for JIRA sensor this would probably just be .).

    Other

    When we are all on the same page and agree that is the right approach, I fill finish this PR and work on a new PR which introduces the same changes for the Python runner actions.

    Comments and feedback welcome.

    opened by Kami 26
  • Immutable/default action parameters for aliases (ChatOps)

    Immutable/default action parameters for aliases (ChatOps)

    There is already mechanism to set optional parameter in alias:

    formats:
      - "google {{query=StackStorm}}"
    

    Problem

    But what if we want to add simple hardcoded chatops command and hide unneeded logic behind the default/immutable action parameters? I think that good ChatOps commands could be simple and easy to remember, without MANY optional parameters. Example case: deploy chatops

    To do that, we need to create additional action with default parameters, which will trigger another action. I do this way almost everywhere (bad way): https://github.com/armab/st2-chatops-aliases/tree/master/actions

    Proposal

    The proposal is to allow setting from alias default/immutable action parameters.

    Here is an example how it could look:

    
    ---
    name: chatops.deploy
    action_ref: packs.install
    description: Deploy ChatOps pack from `armab/st2-ansible-chatops` GitHub repo
    formats:
      - "deploy chatops"
    # these lines were added
    # naming/structure is doubtful, but you got the idea
    action_parameters:
      packs: "st2-ansible-chatops"
      repo_url: "armab/st2-ansible-chatops"
    

    Here you can see that behind simple and immutable/hardcoded ChatOps command:

    !deploy chatops
    

    stands:

    st2 run packs.install packs=st2-ansible-chatops repo_url=armab/st2-ansible-chatops
    

    This simplifies things a lot, making existing actions more reusable by aliases.

    feature status:under discussion proposal chatops 
    opened by armab 25
  • Mistral Deprecation Game Plan

    Mistral Deprecation Game Plan

    Mistral deprecation in favor of new workflow engine Orquesta was communicated for quite a long time. StackStorm v3.2 will be latest version to ship Mistral with full removal in st2 v3.3 per Roadmap.

    There is a helper Mistral -> Orquesta conversion tool available: https://github.com/EncoreTechnologies/orquestaconvert Special thanks @nmaludy an EncoreTechnologies for the tool

    The Plan: https://github.com/orgs/StackStorm/projects/16

    • [x] Identify & Migrate any missing mistral workflows in StackStorm Exchange
    • [x] Remove from st2ci testing and st2cd release automation
      • [x] st2ci https://github.com/StackStorm/st2ci/pull/189 (@amanda11)
      • [x] st2cd https://github.com/StackStorm/st2cd/pull/440 (@amanda11)
    • [x] st2 code cleanup to remove the mistral integration https://github.com/StackStorm/st2/pull/5011 (@amanda11)
    • [x] st2-packages
      • [x] Remove deb/rpm rules from st2-packages https://github.com/StackStorm/st2-packages/pull/656 (@amanda11)
      • [x] Remove mistral from scripted curl-bash installer https://github.com/StackStorm/st2-packages/pull/657 (@amanda11)
      • [ ] Later step: move st2-packages rpm/deb scriptlets into st2 https://github.com/StackStorm/st2-packages/issues/662 (@armab)
    • [x] Remove Mistral from st2tests https://github.com/StackStorm/st2tests/pull/190 (@amanda11)
    • [x] Close all Mistral issues
    • [x] Remove mistral from Deployments
      • [x] ~st2-docker~
      • [x] ~Chef~ (deprecated)
      • [x] Ansible https://github.com/StackStorm/ansible-st2/issues/270 (@amanda11)
      • [x] Puppet https://github.com/StackStorm/puppet-st2/pull/313 (@nmaludy)
    • [x] Remove mistral from the launch_dev.sh script (https://github.com/StackStorm/st2/issues/5012)
    • [x] Update docs
      • [x] https://github.com/StackStorm/st2docs/pull/1009 (@amanda11)
      • [x] https://github.com/StackStorm/st2docs/pull/1010 (@winem)
      • [x] Update any missing diagrams/pictures
      • [x] +Sync-up Roadmap
    • [x] Archive mistral forks/repos https://github.com/StackStorm/mistral_dev https://github.com/StackStorm/mistral https://github.com/StackStorm/st2mistral (@armab)
    • [ ] Blog post about Orquestaconvert project (@nmaludy)
    • [ ] Mention deprecation in v3.3.0 release announcement (@nmaludy as a release manager)

    @m4dcoder feel free to adjust the task list to make sure we don't forget any other places with mistral occurrences.

    help wanted 
    opened by armab 24
  • Implement runner, API, and client code for Inquiries

    Implement runner, API, and client code for Inquiries

    This PR introduces a new runner, inquirer, as well as API endpoints and st2client changes necessary for working with Inquiries.

    Usage

    Please review the corresponding st2docs PR for full usage documentation and examples.

    asciicast

    New "Inquiry" Runner

    This PR creates the inquirer runner, which forms the basis for "asking a question" in the middle of a Workflow.

    Based on previous and current (see below) design discussions, this runner is fairly simple:

    • Dispatch trigger indicating a new inquiry
    • Request that the parent execution (workflow) is paused (for nested workflows, this is done on the root workflow)
    • Return a pending status

    The entire act of handling a response (including validation) and resuming the workflow is handled by the API, and is outside the scope of this runner. This runner's sole purpose is to pause the workflow and provide enough context for a 3rd party to make a proper response.

    The inquirer runner supports a number of parameters with sensible defaults. These defaults can of course be overridden by passing them into the action upon invocation. The parameters used for an Inquiry will be placed in the action result.

    Currently the ttl parameter does nothing, and will be used in a future PR where st2garbagecollector will use it to clean up old Inquiries

    API

    For the time being, Inquiries are effectively treated as ActionExecutions, with a bit of additional logic, meaning we haven't built a full data model for Inquiries yet. This actually works pretty well, but there are a few "weird" things in this PR, especially around RBAC, because of this decision.

    For the time being, the API is (hopefully) designed under the assumption that Inquiries may be their own data model in the future, and acts like it is today.

    This API will provide:

    • GET /api/v1/inquiries: Retrieve all Inquiries
    • GET /api/v1/inquiries/{id}: Retrieve a specific inquiry by ID
    • PUT /api/v1/inquiries/{id}: Provide response data to an Inquiry

    st2client Updates

    There are three new commands, one for each new API function:

    • st2 inquiry list
    • st2 inquiry get <id>
    • st2 inquiry respond <id> <response json>

    I opted to keep command-line options to a minimum, as most of the options for the similar st2 executions list command are focused mainly on filtering, and Inquiries shouldn't be that long-lived. Inquiries (for now) are basically just executions with a certain status, and they shouldn't be in that status for long. So I added the limit option, but not much else. Let me know if you feel other filters, like datetime filters are required, but I'm thinking they shouldn't be necessary.

    Testing Instructions

    NOTE that this is not a complete picture of everything we want to do with Inquiries. This PR focused on an end-to-end implementation of the core functionality. There are a few other misc. things that need to be done to really round this feature out, like adding garbage collection and ensuring things work well with chatops. For this round of testing I want to focus on the core functionality, like the way the API and CLI work, and the invocation of the action

    This is a large and significant feature so manual testing before the merge is a good idea. Here are some things to test

    Please first review the corresponding st2docs PR containing formal descriptions of what Inquiries are and how to use them (and please leave comments on that PR as needed too)

    The following commands will check this branch out, and spin up a development instance of st2, with mistral, and install the client:

    git clone -branch api-ask-response [email protected]:StackStorm/st2.git
    cd st2
    make requirements
    source virtualenv/bin/activate
    tools/launchdev.sh stop && tools/launchdev.sh startclean -m -x && python st2client/setup.py install > /dev/null
    

    From there, you can create and execute workflows and rules that test Inquiries. Note the existing Mistral and ActionChain workflows in this branch that may serve as a good place to start.

    There are a number of things that I feel would be useful to test, but by all means, go beyond this list:

    • Basic testing. Run a simple workflow with an Inquiry (core.ask action), and respond to it. Confirm that the workflow initially pauses, then respond to the inquiry. Then confirm the workflow resumes. Also confirm that the response fails with invalid data (and workflow does not resume)
    • Test with various parameters. Each parameter has an assumed default, override these and confirm that the behavior is as expected. For instance, override the schema parameter with your own, and supply data that would only validate against that
    • Nested workflows. Confirm that pauses and resumes cascade up the chain to the root workflow
    • Standalone Inquiries (not in a workflow). Not a common use case, but in previous discussions, we decided to support this. Confirm you can run core.ask on its own, and that it supports the same response behavior.
    • Send slack notifications using a rule that consumes the core.st2.generic.inquiry trigger. Note that this is just a PoC - a future PR will introduce proper chatops integration once this core functionality is vetted
    • Test all client commands and flags (st2 inquiry get/list/respond). Ensure the output is as you would expect
    • RBAC (This is my first time doing anything with RBAC so I would appreciate some eyes on this) - Ensure you can lock down Inquiry resources the way you'd expect from other resources. I played around with this a bit myself, feel free to use my example as a starting point (slightly out of date - no longer necessary to use inquiry:ask; just use inquiry). This was one area where not having a dedicated model for Inquiries made things difficult (though not impossible).
    • Response permissions using users or roles runner parameters. This goes beyond RBAC and actually permits/denies a per-Inquiry response

    Again, please review the corresponding st2docs PR for full usage documentation and examples.

    TODOs

    • [x] Test RBAC manually to ensure things still work
    • [ ] (winson) Are there integration tests for the st2cd.st2_e2e_tests? Please include one for action chain and one for mistral. Also provide PR to st2cd (and st2ci?) to run this just like the mistral itests are being run
    • [ ] Finish docs
    • [ ] Check coverage
    • [ ] Ensure example chain and mistral workflow is solid
    opened by Mierdin 24
  • Tooz's redis does not appear to respect username setting

    Tooz's redis does not appear to respect username setting

    Hello. I wanted to check to see if I'm doing something wrong, or if this comment should just be updated to not suggest including a username for a redis connection:

    https://github.com/StackStorm/st2/blob/63f66b9595bf5b4e3974e7c5f327fa59f864ba3d/st2common/st2common/services/coordination.py#L206

    But, when trying to connect to a Redis instance using a username, I get this error: "tooz.ToozError: WRONGPASS invalid username-password pair"

    I set a debugger where the Tooz Redis driver creates the Redis client, and this is what I found:

    (Pdb) locals() {'cls': <class 'tooz.drivers.redis.RedisDriver'>, 'parsed_url': _ModifiedSplitResult(scheme='redis', netloc='stackstorm_appuser:[password removed]@redis-[hostname]:6379', path='', query='', fragment=''), 'options': {'lock_timeout': 60}, 'default_socket_timeout': 30, 'kwargs': {'host': 'redis-<>', 'port': 6379, 'password': '<>, 'socket_timeout': 30}, 'a': 'sentinel', 'pdb': <module 'pdb' from '/usr/lib/python3.8/pdb.py'>} (Pdb) c 2023-01-06 23:48:28,185 ERROR [-] (PID=1575) ST2 API quit due to exception. Traceback (most recent call last): File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 42, in _translate_failures yield File "/opt/stackstorm/st2/lib/python3.8/site-packages/tooz/drivers/redis.py", line 452, in _start self._server_info = self._client.info() File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/client.py", line 1304, in info return self.execute_command('INFO') File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/client.py", line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 1192, in get_connection connection.connect() File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 567, in connect self.on_connect() File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 643, in on_connect auth_response = self.read_response() File "/opt/stackstorm/st2/lib/python3.8/site-packages/redis/connection.py", line 756, in read_response raise response redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

    However, if I coerce the username into the kwargs (the Redis client does have the ability to accept and use the username), it works fine:

    (Pdb) kwargs["username"]="stackstorm_appuser" (Pdb) c 2023-01-06 23:47:17,499 INFO [-] Connected to amqp://guest:@rabbitmq:5672// 2023-01-06 23:47:17,612 INFO [-] Connected to amqp://guest:@rabbitmq:5672//

    The Tooz redis driver doesn't seem to actually ever try to set a username (its documentation doesn't actually include username in the pattern at all): https://github.com/openstack/tooz/blob/master/tooz/drivers/redis.py#L162

    Here is where the password is set, but there's no similar setting of the username: https://github.com/openstack/tooz/blob/master/tooz/drivers/redis.py#L424

    Has anybody used a Redis connection with a non-default username and gotten it to work? Or, am I doing something horribly wrong here?

    opened by dalarsen 0
  • Add `pants-plugins/sample_conf` to streamline regenerating `conf/st2.conf.sample`

    Add `pants-plugins/sample_conf` to streamline regenerating `conf/st2.conf.sample`

    Background

    This is another part of introducing pants, as discussed in various TSC meetings.

    Related PRs can be found in:

    • pantsbuild milestone (which includes PRs since #5713)
    • https://github.com/StackStorm/st2/labels/pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

    Overview of this PR

    This PR improves the DX (developer experience) by wiring up the fmt and lint goals to generate conf/st2.conf.sample if needed (or for lint, warn that it needs to be done).

    This includes creating a sample_conf plugin for pants that adds a sample_conf target.

    Developer Experience

    Today, if someone changes one of the files used to generate conf/st2.conf.sample, they will probably forget to run the Makefile target the regenerates them: make configgen.

    I've missed running that several times. CI complains as part of the ci-checks Makefile target, with instructions to Please run "make configgen". This also gets regenerated any time someone runs the lint or tests Makefile targets, which might be surprising (tests generally shouldn't handle codegen).

    With this PR, we add st2.conf.sample generation into the fmt goal. This also adds it to the lint goal which will tell people if fmt needs to run to update any code (or in this case, regenerate the sample conf file). Then, we only need simple instructions that say something like:

    Please run ./pants fmt lint :: before submitting your PR. This will reformat code, if needed, and warn you about any other errors.

    Fine grained results cache

    After someone runs ./pants fmt :: once, they will benefit from the pants' results cache. For the sample_conf plugin, here's what that means:

    First, the sample_conf target in conf/ depends on tools/config_gen.py:

    https://github.com/StackStorm/st2/blob/1faea8f17a9b6b55c21218df8f7d999700338826/conf/BUILD#L6-L9

    If the config_gen script changes, then pants now knows that the sample conf file need to be regenerated. Or, in other words, the config_gen.py file is part of the cache key for conf/st2.conf.sample. If that file changes, then pants will re-run the generator.

    For many of our scripts, we can depend on pants' python dependency inference to grab the other files. But, this script uses importlib instead of standard imports, so we need to add the dependencies manually (aside: there is a feature that tries to use strings for dep inference, but it is not enabled by default, and I have not tested to see how many false positives it will find across our repo). In the future, hopefully we can remove or simplify this.

    https://github.com/StackStorm/st2/blob/1faea8f17a9b6b55c21218df8f7d999700338826/tools/BUILD#L1-L24

    Please note the explicit dependency on //:auth_backends. Nothing in our code base directly imports from the auth backends. But, we ship with a default set of backends, so we need to make sure that their settings are included in the sample conf file's auth section. This is defined here:

    https://github.com/StackStorm/st2/blob/1faea8f17a9b6b55c21218df8f7d999700338826/BUILD#L43-L49

    Then, I ran into one more dependency issue. st2-auth-ldap does not declare a dependency on our st2auth module, in part because that is an implicit dep for auth modules. We should probably make that explicit at some point, but that might require publishing more wheels on pypi. That's messy, but pants meets us where we are in this mess. 😄 So, we can tell pants that any time we grab the st2-auth-ldap package, it requires something in the st2auth module, like this:

    https://github.com/StackStorm/st2/blob/1faea8f17a9b6b55c21218df8f7d999700338826/BUILD#L34-L39

    Then, once all of those deps that pants can't infer (for now) are recorded, pants is able to invalidate the st2.conf.sample cache if any of the dependency python source files (including all that get imported directly or transitively by tools/config_gen.py).

    This PR actually only wires up the fmt goal and took advantage of the fact that pants also runs formatters whenever it runs linters, which are side-effect free. And because pants runs the formatters in a sandbox, it doesn't have to materialize any changes during lint.

    Continuing the example, since pants cached the results when running the lint goal, running fmt will be very fast. Pants will just materialize the already-generated schemas from its cache.

    Developing pants plugins

    Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

    size/L pantsbuild 
    opened by cognifloyd 0
  • Issue/personal keys

    Issue/personal keys

    Code to add feature where Users(non-admin) can manage their own keys. Added a key in config file(personal_keys under rbac). When the key is enabled, Users(non-admin) can create/update/delete/view their own keys and cannot create/update/delete/view other user's keys.

    If the 'personal_keys' key is disabled everything behaves as before. Nothing is changed in regards to admins.

    My client requirement is such that every user(non-admin) in system should manage their own keys. Current implementation of StackStorm is such that if a user(non-admin) is given apikey 'create' permission that user can create apikeys for every other user. But we need it such that every user(non-admin) manages only their own keys(should not be able to create for others).

    Example: Assume 3 users: 'user1', 'user2','user3'. 'user1' is admin, 'user2' has apikey create permission, 'user3' has no permissions attached to him. We need a scenario:

    • 'user1' can create apikeys for everyone since he is admin
    • 'user2', 'user3' can only create apikeys for themselves.
    feature size/L 
    opened by bharath-orchestral 7
  • Refactor `DEFAULT_LOGGING_CONF_PATH` calculation to allow fine-grained pants dep inference

    Refactor `DEFAULT_LOGGING_CONF_PATH` calculation to allow fine-grained pants dep inference

    Background

    This is another part of introducing pants, as discussed in various TSC meetings.

    Related PRs can be found in:

    • pantsbuild milestone (which includes PRs since #5713)
    • https://github.com/StackStorm/st2/labels/pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

    Overview of this PR

    This is primarily a follow-up to:

    • #5846

    Unlike the simple BUILD config added in #5846, this PR refactors one constant so that pants can better use python imports to precisely detect who needs st2common/st2common/conf/base.logging.conf.

    Before this PR, anything that uses any of the constants had a hard-coded dependency on base.logging.conf:

    https://github.com/StackStorm/st2/blob/63f66b9595bf5b4e3974e7c5f327fa59f864ba3d/st2common/st2common/constants/BUILD#L1-L3

    After this PR, each file will only get an inferred dependency if it actually imports the DEFAULT_LOGGING_CONF_PATH constant that refers to that file.

    This also has the benefit of moving the constant calculation closer to the file it references. But, it still maintains backwards compatibility by also making the constant available in the old location.

    refactor size/M pantsbuild 
    opened by cognifloyd 0
  • Add `pants-plugins/api_spec` to streamline regenerating `openapi.yaml`

    Add `pants-plugins/api_spec` to streamline regenerating `openapi.yaml`

    Background

    This is another part of introducing pants, as discussed in various TSC meetings.

    Related PRs can be found in:

    • pantsbuild milestone (which includes PRs since #5713)
    • https://github.com/StackStorm/st2/labels/pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

    Overview of this PR

    This PR improves the DX (developer experience) by wiring up:

    • the fmt goal to generate st2common/st2common/openapi.yaml if needed (effectively runs st2-generate-api-spec), and
    • the lint goal to validate st2common/st2common/openapi.yaml to make sure the generated schema is valid (effectively runs st2-validate-api-spec)

    This includes creating a api_spec plugin for pants that adds a api_spec target.

    Developer Experience

    Today, if someone changes one of the files/scripts used to generate our openapi spec, then CI might complain as part of the ci-checks Makefile target, with instructions to Please run "make generate-api-spec". This also gets regenerated any time someone runs the lint or tests Makefile targets, which might be surprising (tests generally shouldn't handle codegen).

    In any case, this wires up pants to run st2-generate-api-spec and st2-validate-api-spec at times that are analogous to when the Makefile runs them.

    With this PR, we add api spec generation into the fmt goal and validation to the lint goal. Anything that runs under fmt also runs under the lint goal, so we will get two signals about the api spec: (1) whether or not it needs to be regenerated, and (2) whether or not there are validation errors in the spec.

    Our api spec is generated from a jinja template, which only does minimal interpolation (populating the values for some of the enums). There are actually quite a few inconsistencies between our models and the api spec, so we will have to refactor the spec, the spec generation, and possibly the models in the future. That refactoring work, however, is out-of-scope for this PR. After we get pants in, it would be excellent to revisit these issues.

    In working on this plugin, I discovered that our api spec validation has been broken for awhile, which I fixed in #5709. Part of our custom validation is still disabled because there are so many things in the spec that are out of date. This PR also does not fix that, but it does leave a comment in the pants plugin on how to enable that later.

    Fine grained results cache

    After someone runs ./pants fmt :: once, they will benefit from the pants' results cache. For the api_spec plugin, here's what that means:

    First, the api_spec target in st2common/st2common depends on the generate/validate binaries in st2common/bin:

    https://github.com/StackStorm/st2/blob/1856f6108219ebb1a6886afb798fd0f6c97b184c/st2common/st2common/BUILD#L12-L21

    If either script (st2-generate-api-spec or st2-validate-api-spec) changes, then pants now knows that openapi.yaml need to be regenerated. Or, in other words, the st2-generate-api-spec file is part of the cache key for st2common/st2common/openapi.yaml. If that file changes, then pants will re-run the generator.

    st2-generate-api-spec is a python script, and we use pants' python dependency inference. So pants (effectively) also adds the python files imported by st2-generate-api-spec to the cache key. Stepping through the code, we can see what pants would infer:

    https://github.com/StackStorm/st2/blob/1856f6108219ebb1a6886afb798fd0f6c97b184c/st2common/bin/st2-generate-api-spec#L18 https://github.com/StackStorm/st2/blob/1856f6108219ebb1a6886afb798fd0f6c97b184c/st2common/st2common/cmd/generate_api_spec.py#L22-L26 https://github.com/StackStorm/st2/blob/1856f6108219ebb1a6886afb798fd0f6c97b184c/st2common/st2common/util/spec_loader.py#L21-L26

    Here we reach the few things that get interpolated into the Jinja Template:

    https://github.com/StackStorm/st2/blob/1856f6108219ebb1a6886afb798fd0f6c97b184c/st2common/st2common/util/spec_loader.py#L30-L35

    Now, if anyone changes any of these files, or any of the python bits that get imported to help define things in them, then the next time someone runs the lint goal (or in CI) they will find out that they need to run the fmt goal.

    Developing pants plugins

    Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there are even mini tutorials about how to add a linter or a formatter (adding formatters/linters is a very common thing plugins to do).

    size/XL pantsbuild 
    opened by cognifloyd 1
  • Add `pants-plugins/schemas` to streamline regenerating `contrib/schemas`

    Add `pants-plugins/schemas` to streamline regenerating `contrib/schemas`

    Background

    This is another part of introducing pants, as discussed in various TSC meetings.

    Related PRs can be found in:

    • pantsbuild milestone (which includes PRs since #5713)
    • https://github.com/StackStorm/st2/labels/pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

    Overview of this PR

    This PR improves the DX (developer experience) by wiring up the fmt and lint goals to generate contrib/schemas/*.json if needed (or for lint, warn that it needs to be done).

    This includes creating a schemas plugin for pants that adds a schemas target.

    Developer Experience

    Today, if someone changes one of the files used to generate the schemas in contrib/schemas, they will probably forget to run the Makefile target the regenerates them: make schemasgen.

    I've missed running that several times. If I'm lucky, CI complains with an error telling me to run that. I've seen CI logs print an error without actually failing the build. I believe that particular Makefile bug is fixed, but that anecdote highlights a usability issue: friction around generated files.

    With this PR, we add schema generation into the fmt goal. This also adds it to the lint goal which will tell people if fmt needs to run to update any code (or in this case, regenerate the schema). Then, we only need simple instructions that say something like:

    Please run ./pants fmt lint :: before submitting your PR. This will reformat code, if needed, and warn you about any other errors.

    Fine grained results cache

    After someone runs ./pants fmt :: once, they will benefit from the pants' results cache. For the schemas plugin, here's what that means:

    First, the schemas target in contrib/schemas depends on st2common/bin/st2-generate-schemas:

    https://github.com/StackStorm/st2/blob/d4fba5621e50aa3ba57404f72b8e6748158d063c/contrib/schemas/BUILD#L1-L5

    If the st2-generate-schemas script changes, then pants now knows that the schemas need to be regenerated. Or, in other words, the st2-generate-schemas file is part of the cache key for contrib/schemas/*.json. If that file changes, then pants will re-run the generator.

    st2-generate-schemas is a python script, and we use pants' python dependency inference. So pants (effectively) also adds the python files imported by st2-generate-schemas to the cache key. Stepping through the code, we can see what pants would infer:

    https://github.com/StackStorm/st2/blob/d4fba5621e50aa3ba57404f72b8e6748158d063c/st2common/bin/st2-generate-schemas#L32 https://github.com/StackStorm/st2/blob/d4fba5621e50aa3ba57404f72b8e6748158d063c/st2common/st2common/cmd/generate_schemas.py#L26-L30

    Here we reach the primary definition for the schemas that end up in contrib/schemas: the definition comes from the st2common.model.api.* modules. Now, if anyone changes any of these files, or any of the python bits that get imported to help define things in them, then the next time someone runs the lint goal (or in CI) they will find out that they need to run the fmt goal.

    This PR actually only wires up the fmt goal and took advantage of the fact that pants also runs formatters whenever it runs linters, which are side-effect free. And because pants runs the formatters in a sandbox, it doesn't have to materialize any changes during lint.

    Continuing the example, since pants cached the results when running the lint goal, running fmt will be very fast. Pants will just materialize the already-generated schemas from its cache.

    Developing pants plugins

    Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

    Why not codegen?

    Pants has something called codegen. But it is for code that gets generated without getting checked into the repo. You can export those changes, but they are always exported in a directory under dist/, so we would have to add some custom logic somewhere that copies from dist/ to the contrib/schemas directory.

    These schemas are only available in this repo; if someone wants to use them, they have to clone the repo or grab the raw file from github. So, they have to be checked into the repo. We might be able to do something different in the future, but I'm trying not to change more than I have to to introduce pants.

    size/L infrastructure: ci/cd pantsbuild 
    opened by cognifloyd 0
Releases(v3.8.0)
  • v3.8.0(Nov 27, 2022)

    3.8.0 - November 18, 2022

    Fixed

    • Fix redis SSL problems with sentinel #5660

    • Fix a bug in the pack config loader so that objects covered by an patternProperties schema or arrays using additionalItems schema(s) can use encrypted datastore keys and have their default values applied correctly. #5321

      Contributed by @cognifloyd

    • Fixed st2client/st2client/base.py file to check for http_proxy and https_proxy environment variables for both lower and upper cases.

      Contributed by @S-T-A-R-L-O-R-D

    • Fixed a bug where calling 'get_by_name' on client for getting key details was not returning any results despite key being stored. #5677

      Contributed by @bharath-orchestral

    • Fixed st2client/st2client/base.py file to use https_proxy(not http_proxy) to check HTTPS_PROXY environment variables.

      Contributed by @wfgydbu

    • Fixed schema utils to more reliably handle schemas that define nested arrays (object-array-object-array-string) as discovered in some of the ansible installer RBAC tests (see #5684). This includes a test that reproduced the error so we don't hit this again. #5685

    • Fixed eventlet monkey patching so more of the unit tests work under pytest. #5689

    • Fix and reenable prance-based openapi spec validation, but make our custom x-api-model validation optional as the spec is out-of-date. #5709 Contributed by @cognifloyd

    • Fixed generation of st2.conf.sample to show correct syntax for [sensorcontainer].partition_provider (space separated key:value pairs). #5710 Contributed by @cognifloyd

    • Fix access to key-value pairs in workflow and action execution where RBAC rules did not get applied #5764

      Contributed by @m4dcoder

    • Add backward compatibility to secret masking introduced in #5319 to prevent security-relative issues. Migration to the new schema is required to take advantage of the full output schema validation. #5783

      Contributed by @m4dcoder

    Added

    • Added graceful shutdown for workflow engine. #5463 Contributed by @khushboobhatia01

    • Add ST2_USE_DEBUGGER env var as alternative to the --use-debugger cli flag. #5675 Contributed by @cognifloyd

    • Added purging of old tokens. #5679 Contributed by Amanda McGuinness (@amanda11 intive)

    • Begin introducing pants <https://www.pantsbuild.org/docs>_ to improve DX (Developer Experience) working on StackStorm, improve our security posture, and improve CI reliability thanks in part to pants' use of PEX lockfiles. This is not a user-facing addition. #5713 #5724 #5726 #5725 #5732 #5733 #5737 #5738 #5758 #5751 #5774 #5776 #5777 #5782 Contributed by @cognifloyd

    Changed

    • BREAKING CHANGE for anyone that uses output_schema, which is disabled by default. If you have [system].validate_output_schema = True in st2.conf AND you have added output_schema to any of your packs, then you must update your action metadata.

      output_schema must be a full jsonschema now. If a schema is not well-formed, we ignore it. Now, output can be types other than object such as list, bool, int, etc. This also means that all of an action's output can be masked as a secret.

      To get the same behavior, you'll need to update your output schema. For example, this schema:

    
        output_schema:
          property1:
            type: bool
          property2:
            type: str
    

    should be updated like this:

    
        output_schema:
          type: object
          properties:
            property1:
              type: bool
            property2:
              type: str
          additionalProperties: false
    

    #5319

    Contributed by @cognifloyd

    • Changed the X-XSS-Protection HTTP header from 1; mode=block to 0 in the conf/nginx/st2.conf to align with the OWASP security standards. #5298

      Contributed by @LiamRiddell

    • Use PEP 440 direct reference requirements instead of legacy PIP VCS requirements. Now, our *.requirements.txt files use package-name@ git+https://url@version ; markers instead of git+https://url@version#egg=package-name ; markers. #5673 Contributed by @cognifloyd

    • Move from udatetime to ciso8601 for date functionality ahead of supporting python3.9 #5692 Contributed by Amanda McGuinness (@amanda11 intive)

    • Refactor tests to use python imports to identify test fixtures. #5699 #5702 #5703 #5704 #5705 #5706 Contributed by @cognifloyd

    • Refactor st2-generate-schemas so that logic is in an importable module. #5708 Contributed by @cognifloyd

    Removed

    • Removed st2exporter service. It is unmaintained and does not get installed. It was originally meant to help with analytics by exporting executions as json files that could be imported into something like elasticsearch. Our code is now instrumented to make a wider variety of stats available to metrics drivers. #5676 Contributed by @cognifloyd
    Source code(tar.gz)
    Source code(zip)
  • v3.7.0(May 6, 2022)

    https://stackstorm.com/2022/05/10/stackstorm-3-7-0-released/

    3.7.0 - May 05, 2022

    Added

    • Added st2 API get action parameters by ref. #5509 API endpoint /api/v1/actions/views/parameters/{action_id} accepts ref_or_id. Contributed by @DavidMeu

    • Enable setting ttl for MockDatastoreService. #5468 Contributed by @ytjohn

    • Added st2 API and CLI command for actions clone operation.

      API endpoint /api/v1/actions/{ref_or_id}/clone takes ref_or_id of source action.Request method body takes destination pack and action name. Request method body also takes optional parameter overwrite. overwrite = true in case of destination action already exists and to be overwritten.

      CLI command st2 action clone <ref_or_id> <dest_pack> <dest_action> takes source ref_or_id, destination pack name and destination action name as mandatory arguments. In case destination already exists then command takes optional argument -f or --force to overwrite destination action. #5345

      Contributed by @mahesh-orch.

    • Implemented RBAC functionality for existing KEY_VALUE_VIEW, KEY_VALUE_SET, KEY_VALUE_DELETE and new permission types KEY_VALUE_LIST, KEY_VALUE_ALL. RBAC is enabled in the st2.conf file. Access to a key value pair is checked in the KeyValuePair API controller. #5354 Contributed by @m4dcoder and @ashwini-orchestral

    • Added service deregistration on shutdown of a service. #5396 Contributed by @khushboobhatia01

    • Added pysocks python package for SOCKS proxy support. #5460 Contributed by @kingsleyadam

    • Added support for multiple LDAP hosts to st2-auth-ldap. #5535, https://github.com/StackStorm/st2-auth-ldap/pull/100 Contributed by @ktyogurt

    • Implemented graceful shutdown for action runner. Enabled graceful_shutdown in st2.conf file. #5428 Contributed by @khushboobhatia01

    • Enhanced 'search' operator to allow complex criteria matching on payload items. #5482 Contributed by @erceth

    • Added cancel/pause/resume requester information to execution context. #5554 Contributed by @khushboobhatia01

    • Added trigger.headers_lower to webhook trigger payload. This allows rules to match webhook triggers without dealing with the case-sensitive nature of trigger.headers, as triggers.headers_lower providers the same headers, but with the header name lower cased. #5038 Contributed by @Rand01ph

    • Added support to override enabled parameter of resources. #5506 Contributed by Amanda McGuinness (@amanda11 Intive)

    • Add new api.auth_cookie_secure and api.auth_cookie_same_site config options which specify values which are set for secure and SameSite attribute for the auth cookie we set when authenticating via token / api key in query parameter value (e.g. via st2web).

      For security reasons, api.auth_cookie_secure defaults to True. This should only be changed to False if you have a valid reason to not run StackStorm behind HTTPs proxy.

      Default value for api.auth_cookie_same_site is lax. If you want to disable this functionality so it behaves the same as in the previous releases, you can set that option to None.

      #5248 Contributed by @Kami.

    • Add new st2 action-alias test <message string> CLI command which allows users to easily test action alias matching and result formatting.

      This command will first try to find a matching alias (same as st2 action-alias match command) and if a match is found, trigger an execution (same as st2 action-alias execute command) and format the execution result.

      This means it uses exactly the same flow as commands on chat, but the interaction avoids chat and hubot which should make testing and developing aliases easier and faster. #5143

      #5143 Contributed by @Kami.

    • Add new credentials.basic_auth = username:password CLI configuration option.

      This argument allows client to use additional set of basic auth credentials when talking to the StackStorm API endpoints (api, auth, stream) - that is, in addition to the token / api key native StackStorm auth.

      This allows for simple basic auth based multi factor authentication implementation for installations which don't utilize SSO.

      #5152 Contributed by @Kami.

    • Add new audit message when a user has decrypted a key whether manually in the container (st2 key get [] --decrypt) or through a workflow with a defined config. #5594 Contributed by @dmork123

    • Added garbage collection for rule_enforcement and trace models #5596/5602 Contributed by Amanda McGuinness (@amanda11 intive)

    • Added garbage collection for workflow execution and task execution objects #4924 Contributed by @srimandaleeka01 and @amanda11

    Changed

    • Minor updates for RockyLinux. #5552

      Contributed by Amanda McGuinness (@amanda11 intive)

    • Bump black to v22.3.0 - This is used internally to reformat our python code. #5606

    • Updated paramiko version to 2.10.3 to add support for more key verification algorithms. #5600

    Fixed

    • Fix deserialization bug in st2 API for url encoded payloads. #5536 Contributed by @sravs-dev

    • Fix issue of WinRM parameter passing fails for larger scripts.#5538 Contributed by @ashwini-orchestral

    • Fix Type error for time_diff critera comparison. convert the timediff value as float to match timedelta.total_seconds() return. #5462 Contributed by @blackstrip

    • Fix issue with pack option not working when running policy list cli #5534 Contributed by @momokuri-3

    • Fix exception thrown if action parameter contains {{ or {% and no closing jinja characters. #5556 contributed by @guzzijones12

    • Link shutdown routine and sigterm handler to main thread #5555 Contributed by @khushboobhatia01

    • Change compound index for ActionExecutionDB to improve query performance #5568 Contributed by @khushboobhatia01

    • Fix build issue due to MarkUpSafe 2.1.0 removing soft_unicode Contributed by Amanda McGuinness (@amanda11 intive) #5581

    • Fixed regression caused by #5358. Use string lock name instead of object ID. #5484 Contributed by @khushboobhatia01

    • Fix st2-self-check script reporting falsey success when the nested workflows runs failed. #5487

    • Fix actions from the contrib/linux pack that fail on CentOS-8 but work on other operating systems and distributions. (bug fix) #4999 #5004 Reported by @blag and @dove-young contributed by @winem.

    • Use byte type lock name which is supported by all tooz drivers. #5529 Contributed by @khushboobhatia01

    • Fixed issue where pack index searches are ignoring no_proxy #5497 Contributed by @minsis

    • Fixed trigger references emitted by linux.file_watch.line. #5467

      Prior to this patch multiple files could be watched but the rule reference of last registered file would be used for all trigger emissions causing rule enforcement to fail. References are now tracked on a per file basis and used in trigger emissions.

      Contributed by @nzlosh

    • Downgrade tenacity as tooz dependency on tenacity has always been < 7.0.0 #5607 Contributed by @khushboobhatia01

    • Pin typing-extensions<4.2 (used indirectly by st2client) to maintain python 3.6 support. #5638

    Source code(tar.gz)
    Source code(zip)
  • v3.6.0(Nov 22, 2021)

    https://stackstorm.com/2021/12/16/stackstorm-v3-6-0-released/

    3.6.0 - October 29, 2021

    Added

    • Added possibility to add new values to the KV store via CLI without leaking them to the shell history. #5164

    • st2.conf is now the only place to configure ports for st2api, st2auth, and st2stream.

      We replaced the static .socket sytemd units in deb and rpm packages with a python-based generator for the st2api, st2auth, and st2stream services. The generators will get <ip>:<port> from st2.conf to create the .socket files dynamically. #5286 and st2-packages#706

      Contributed by @nzlosh

    Changed

    • Modified action delete API to delete action files from disk along with backward compatibility.

      From CLI st2 action delete <pack>.<action> will delete only action database entry. From CLI st2 action delete --remove-files <pack>.<action> or st2 action delete -r <pack>.<action> will delete action database entry along with files from disk.

      API action DELETE method with {"remove_files": true} argument in json body will remove database entry of action along with files from disk. API action DELETE method with {"remove_files": false} or no additional argument in json body will remove only action database entry. #5304, #5351, #5360

      Contributed by @mahesh-orch.

    • Removed --python3 deprecated flag from st2client. #5305

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

      Contributed by @blag.

    • Fixed __init__.py files to use double quotes to better align with black linting #5299

      Contributed by @blag.

    • Reduced minimum TTL on garbage collection for action executions and trigger instances from 7 days to 1 day. #5287

      Contributed by @ericreeves.

    • update db connect mongo connection test - isMaster MongoDB command depreciated, switch to ping #5302, #5341

      Contributed by @lukepatrick

    • Actionrunner worker shutdown should stop Kombu consumer thread. #5338

      Contributed by @khushboobhatia01

    • Move to using Jinja sandboxed environment #5359

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Pinned python module networkx to versions between 2.5.1(included) and 2.6(excluded) because Python v3.6 support was dropped in v2.6. Also pinned decorator==4.4.2 (dependency of networkx<2.6) to work around missing python 3.8 classifiers on decorator's wheel. #5376

      Contributed by @nzlosh

    • Add new --enable-profiler flag to all the servies. This flag enables cProfiler based profiler for the service in question and dumps the profiling data to a file on process exit.

      This functionality should never be used in production, but only in development environments or similar when profiling code. #5199

      Contributed by @Kami.

    • Add new --enable-eventlet-blocking-detection flag to all the servies. This flag enables eventlet long operation / blocked main loop logic which throws an exception if a particular code blocks longer than a specific duration in seconds.

      This functionality should never be used in production, but only in development environments or similar when debugging code. #5199

    • Silence pylint about dev/debugging utility (tools/direct_queue_publisher.py) that uses pika because kombu doesn't support what it does. If anyone uses that utility, they have to install pika manually. #5380

    • Fixed version of cffi as changes in 1.15.0 meant that it attempted to load libffi.so.8. #5390

      Contributed by @amanda11, Ammeon Solutions

    • Updated Bash installer to install latest RabbitMQ version rather than out-dated version available in OS distributions.

      Contributed by @amanda11, Ammeon Solutions

    Fixed

    • Correct error reported when encrypted key value is reported, and another key value parameter that requires conversion is present. #5328 Contributed by @amanda11, Ammeon Solutions

    • Make update_executions() atomic by protecting the update with a coordination lock. Actions, like workflows, may have multiple concurrent updates to their execution state. This makes those updates safer, which should make the execution status more reliable. #5358

      Contributed by @khushboobhatia01

    • Fix "not iterable" error for output_schema handling. If a schema is not well-formed, we ignore it. Also, if action output is anything other than a JSON object, we do not try to process it any more. output_schema will change in a future release to support non-object output. #5309

      Contributed by @guzzijones

    • core.inject_trigger: resolve trigger payload shadowing by deprecating trigger param in favor of trigger_name. trigger param is still available for backwards compatibility, but will be removed in a future release. #5335 and #5383

      Contributed by @mjtice

    Source code(tar.gz)
    Source code(zip)
  • v3.5.0(Jun 28, 2021)

    Added

    • Added web header settings for additional security hardening to nginx.conf: X-Frame-Options, Strict-Transport-Security, X-XSS-Protection and server-tokens. #5183

      Contributed by @shital.

    • Added support for limit and offset argument to the list_values data store service method (#5097 and #5171).

      Contributed by @anirudhbagri.

    • Various additional metrics have been added to the action runner service to provide for better operational visibility. (improvement) #4846

      Contributed by @Kami.

    • Added sensor model to list of JSON schemas auto-generated by make schemasgen that can be used by development tools to validate pack contents. (improvement)

    • Added the command line utility st2-validate-pack that can be used by pack developers to validate pack contents. (improvement)

    • Fix a bug in the API and CLI code which would prevent users from being able to retrieve resources which contain non-ascii (utf-8) characters in the names / references. (bug fix) #5189

      Contributed by @Kami.

    • Fix a bug in the API router code and make sure we return correct and user-friendly error to the user in case we fail to parse the request URL / path because it contains invalid or incorrectly URL encoded data.

      Previously such errors weren't handled correctly which meant original exception with a stack trace got propagated to the user. (bug fix) #5189

      Contributed by @Kami.

    • Make redis the default coordinator backend.

    • Fix a bug in the pack config loader so that objects covered by an additionalProperties schema can use encrypted datastore keys and have their default values applied correctly. #5225

      Contributed by @cognifloyd.

    • Add new database.compressors and database.zlib_compression_level config option which specifies compression algorithms client supports for network / transport level compression when talking to MongoDB.

      Actual compression algorithm used will be then decided by the server and depends on the algorithms which are supported by the server + client.

      Possible / valid values include: zstd, zlib. Keep in mind that zstandard (zstd) is only supported by MongoDB >= 4.2.

      Our official Debian and RPM packages bundle zstandard dependency by default which means setting this value to zstd should work out of the box as long as the server runs MongoDB >= 4.2. #5177

      Contributed by @Kami.

    • Add support for compressing the payloads which are sent over the message bus. Compression is disabled by default and user can enable it by setting messaging.compression config option to one of the following values: zstd, lzma, bz2, gzip.

      In most cases we recommend using zstd (zstandard) since it offers best trade off between compression ratio and number of CPU cycles spent for compression and compression.

      How this will affect the deployment and throughput is very much user specific (workflow and resources available). It may make sense to enable it when generic action trigger is enabled and when working with executions with large textual results. #5241

      Contributed by @Kami.

    • Mask secrets in output of an action execution in the API if the action has an output schema defined and one or more output parameters are marked as secret. #5250

      Contributed by @mahesh-orch.

    Changed

    • All the code has been refactored using black and black style is automatically enforced and required for all the new code. (#5156)

      Contributed by @Kami.

    • Default nginx config (conf/nginx/st2.conf) which is used by the installer and Docker images has been updated to only support TLS v1.2 and TLS v1.3 (support for TLS v1.0 and v1.1 has been removed).

      Keep in mind that TLS v1.3 will only be used when nginx is running on more recent distros where nginx is compiled against OpenSSL v1.1.1 which supports TLS 1.3. #5183 #5216

      Contributed by @Kami and @shital.

    • Add new -x argument to the st2 execution get command which allows result field to be excluded from the output. (improvement) #4846

    • Update st2 execution get <id> command to also display execution log attribute which includes execution state transition information.

      By default end_timestamp attribute and duration attribute displayed in the command output only include the time it took action runner to finish running actual action, but it doesn't include the time it it takes action runner container to fully finish running the execution - this includes persisting execution result in the database.

      For actions which return large results, there could be a substantial discrepancy - e.g. action itself could finish in 0.5 seconds, but writing data to the database could take additional 5 seconds after the action code itself was executed.

      For all purposes until the execution result is persisted to the database, execution is not considered as finished.

      While writing result to the database action runner is also consuming CPU cycles since serialization of large results is a CPU intensive task.

      This means that "elapsed" attribute and start_timestamp + end_timestamp will make it look like actual action completed in 0.5 seconds, but in reality it took 5.5 seconds (0.5 + 5 seconds).

      Log attribute can be used to determine actual duration of the execution (from start to finish). (improvement) #4846

      Contributed by @Kami.

    • Various internal improvements (reducing number of DB queries, speeding up YAML parsing, using DB object cache, etc.) which should speed up pack action registration between 15-30%. This is especially pronounced with packs which have a lot of actions (e.g. aws one). (improvement) #4846

      Contributed by @Kami.

    • Underlying database field type and storage format for the Execution, LiveAction, WorkflowExecutionDB, TaskExecutionDB and TriggerInstanceDB database models has changed.

      This new format is much faster and efficient than the previous one. Users with larger executions (executions with larger results) should see the biggest improvements, but the change also scales down so there should also be improvements when reading and writing executions with small and medium sized results.

      Our micro and end to benchmarks have shown improvements up to 15-20x for write path (storing model in the database) and up to 10x for the read path.

      To put things into perspective - with previous version, running a Python runner action which returns 8 MB result would take around ~18 seconds total, but with this new storage format, it takes around 2 seconds (in this context, duration means the from the time the execution was scheduled to the time the execution model and result was written and available in the database).

      The difference is even larger when working with Orquesta workflows.

      Overall performance improvement doesn't just mean large decrease in those operation timings, but also large overall reduction of CPU usage - previously serializing large results was a CPU intensive time since it included tons of conversions and transformations back and forth.

      The new format is also around 10-20% more storage efficient which means that it should allows for larger model values (MongoDB document size limit is 16 MB).

      The actual change should be fully opaque and transparent to the end users - it's purely a field storage implementation detail and the code takes care of automatically handling both formats when working with those object.

      Same field data storage optimizations have also been applied to workflow related database models which should result in the same performance improvements for Orquesta workflows which pass larger data sets / execution results around.

      Trigger instance payload field has also been updated to use this new field type which should result in lower CPU utilization and better throughput of rules engine service when working with triggers with larger payloads.

      This should address a long standing issue where StackStorm was reported to be slow and CPU inefficient with handling large executions.

      If you want to migrate existing database objects to utilize the new type, you can use st2common/bin/migrations/v3.5/st2-migrate-db-dict-field-values migration script. (improvement) #4846

      Contributed by @Kami.

    • Add new result_size field to the ActionExecutionDB model. This field will only be populated for executions which utilize new field storage format.

      It holds the size of serialzed execution result field in bytes. This field will allow us to implement more efficient execution result retrieval and provide better UX since we will be able to avoid loading execution results in the WebUI for executions with very large results (which cause browser to freeze). (improvement) #4846

      Contributed by @Kami.

    • Add new /v1/executions/<id>/result[?download=1&compress=1&pretty_format=1] API endpoint which can be used used to retrieve or download raw execution result as (compressed) JSON file.

      This endpoint will primarily be used by st2web when executions produce very large results so we can avoid loading, parsing and formatting those very large results as JSON in the browser which freezes the browser window / tab. (improvement) #4846

      Contributed by @Kami.

    • Update jinja2 dependency to the latest stable version (2.11.3). #5195

    • Update pyyaml dependency to the latest stable version (5.4). #5207

    • Update various dependencies to latest stable versions (bcrypt, appscheduler, pytz, python-dateutil, psutil, passlib, gunicorn, flex, cryptography. eventlet, greenlet, webob , mongoengine, pymongo, requests, pyyaml, kombu, amqp, python-ldap).

      #5215, https://github.com/StackStorm/st2-auth-ldap/pull/94

      Contributed by @Kami.

    • Update code and dependencies so it supports Python 3.8 and Mongo DB 4.4 #5177

      Contributed by @nzloshm @winem @Kami.

    • StackStorm Web UI (st2web) has been updated to not render and display execution results larger than 200 KB directly in the history panel in the right side bar by default anymore. Instead a link to view or download the raw result is displayed.

      Execution result widget was never optimized to display very large results (especially for executions which return large nested dictionaries) so it would freeze and hang the whole browser tab / window when trying to render / display large results.

      If for some reason you want to revert to the old behavior (this is almost never a good idea since it will cause browser to freeze when trying to display large results), you can do that by setting max_execution_result_size_for_render option in the config to a very large value (e.g. max_execution_result_size_for_render: 16 * 1024 * 1024).

      https://github.com/StackStorm/st2web/pull/868

      Contributed by @Kami.

    • Some of the config option registration code has been refactored to ignore "option already registered" errors. That was done as a work around for an occasional race in the tests and also to make all of the config option registration code expose the same consistent API. #5234

      Contributed by @Kami.

    • Update pyywinrm dependency to the latest stable version (0.4.1). #5212

      Contributed by @chadpatt .

    • Monkey patch on st2stream earlier in flow #5240

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Support % in CLI arguments by reading the ConfigParser() arguments with raw=True.

      This removes support for '%' interpolations on the configuration arguments.

      See https://docs.python.org/3.8/library/configparser.html#configparser.ConfigParser.get for further details. #5253

      Contributed by @winem.

    • Remove duplicate host header in the nginx config for the auth endpoint.

    • Update orquesta to v1.4.0.

    Improvements

    • CLI has been updated to use or orjson when parsing API response and C version of the YAML safe dumper when formatting execution result for display. This should result in speed up when displaying execution result (st2 execution get, etc.) for executions with large results.

      When testing it locally, the difference for execution with 8 MB result was 18 seconds vs ~6 seconds. (improvement) #4846

      Contributed by @Kami.

    • Update various Jinja functiona to utilize C version of YAML safe_{load,dump} functions and orjson for better performance. (improvement) #4846

      Contributed by @Kami.

    • For performance reasons, use udatetime library for parsing ISO8601 / RFC3339 date strings where possible. (improvement) #4846

      Contributed by @Kami.

    • Speed up service start up time by speeding up runners registration on service start up by re-using existing stevedore ExtensionManager instance instead of instantiating new DriverManager instance per extension which is not necessary and it's slow since it requires disk / pkg resources scan for each extension. (improvement) #5198

      Contributed by @Kami.

    • Add new ?max_result_size query parameter filter to the GET /v1/executiond/<id> API endpoint.

      This query parameter allows clients to implement conditional execution result retrieval and only retrieve the result field if it's smaller than the provided value.

      This comes handy in the various client scenarios (such as st2web) where we don't display and render very large results directly since it allows to speed things up and decrease amount of data retrieved and parsed. (improvement) #5197

      Contributed by @Kami.

    • Update default nginx config which is used for proxying API requests and serving static content to only allow HTTP methods which are actually used by the services (get, post, put, delete, options, head).

      If a not-allowed method is used, nginx will abort the request early and return 405 status code. #5193

      Contributed by @ashwini-orchestral

    • Update default nginx config which is used for proxying API requests and serving static content to not allow range requests. #5193

      Contributed by @ashwini-orchestral

    • Drop unused python dependencies: prometheus_client, python-gnupg, more-itertools, zipp. #5228

      Contributed by @cognifloyd.

    • Update majority of the "resource get" CLI commands (e.g. st2 execution get, st2 action get, st2 rule get, st2 pack get, st2 apikey get, st2 trace get, st2 key get, st2 webhook get, st2 timer get, etc.) so they allow for retrieval and printing of information for multiple resources using the following notation: st2 <resource> get <id 1> <id 2> <id n>, e.g. st2 action.get pack.show packs.get packs.delete

      This change is fully backward compatible when retrieving only a single resource (aka single id is passed to the command).

      When retrieving a single source the command will throw and exit with non-zero if a resource is not found, but when retrieving multiple resources, command will just print an error and continue with printing the details of any other found resources. (new feature) #4912

      Contributed by @Kami.

    Fixed

    • Refactor spec_loader util to use yaml.load with SafeLoader. (security) Contributed by @ashwini-orchestral

    • Import ABC from collections.abc for Python 3.10 compatibility. (#5007) Contributed by @tirkarthi

    • Updated to use virtualenv 20.4.0/PIP20.3.3 and fixate-requirements to work with PIP 20.3.3 #512 Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Fix st2 execution get --with-schema flag. (bug fix) #4846

      Contributed by @Kami.

    • Fix SensorTypeAPI schema to use class_name instead of name since documentation for pack development uses class_name and registrar used to load sensor to database assign class_name to name in the database model. (bug fix)

    • Updated paramiko version to 2.7.2, to go with updated cryptography to prevent problems with ssh keys on remote actions. #5201

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Update rpm package metadata and fix Provides section for RHEL / CentOS 8 packages.

      In the previous versions, RPM metadata would incorrectly signal that the st2 package provides various Python libraries which it doesn't (those Python libraries are only used internally for the package local virtual environment).

      https://github.com/StackStorm/st2-packages/pull/697

      Contributed by @Kami.

    • Make sure st2common.util.green.shell.run_command() doesn't leave stray / zombie processes laying around in some command timeout scenarios. #5220

      Contributed by @r0m4n-z.

    • Fix support for skipping notifications for workflow actions. Previously if action metadata specified an empty list for notify parameter value, that would be ignored / not handled correctly for workflow (orquesta, action chain) actions. #5221 #5227

      Contributed by @khushboobhatia01.

    • Clean up to remove unused methods in the action execution concurrency policies. #5268

    Source code(tar.gz)
    Source code(zip)
  • v3.4.1(Mar 16, 2021)

    https://stackstorm.com/2021/03/10/stackstorm-v3-4-1-security-fix/

    Added

    • Update the service start up code to warn if the service locale encoding is not set to utf-8 #5184 Contributed by @Kami

    Changed

    • Use sudo -E to fix GitHub Actions tests #5187 Contributed by @cognifloyd
    • Properly handle unicode strings in logs #5184 Contributed by @Kami
    Source code(tar.gz)
    Source code(zip)
  • v3.4.0(Mar 4, 2021)

    https://stackstorm.com/2021/03/04/v3-4-0-released/

    Added

    • Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050 Contributed by @asthLucas

    • Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086 Contributed by @hnanchahal

    • Added notification support for err-stackstorm. (new feature) #5051

    • Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082 Contributed by @hnanchahal

    • Added --register-recreate-virtualenvs flag to st2ctl reload to recreate virtualenvs from scratch. (part of upgrade instructions) [#5167] Contributed by @winem and @blag

    Changed

    • Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099 Contributed by @amanda11

    • Improve the st2-self-check script to echo to stderr and exit if it isn't run with a ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068

    • Added timeout parameter for packs.install action to help with long running installs that exceed the default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084

      Contributed by @hnanchahal

    • Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095

    • Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).

      Contributed by @nmaludy, @winem, and @blag

    • Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151

    Fixed

    • Pin chardet version as newest version was incompatible with pinned requests version #5101 Contributed by @amanda11

    • Fixed issue were st2tests was not getting installed using pip because no version was specified. Contributed by @anirudhbagri

    • Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091

    • Fix nginx buffering long polling stream to client. Instead of waiting for closed connection wait for final event to be sent to client. (bug fix) #4842 #5042

      Contributed by @guzzijones

    • StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix) #5106, #5107

    • Fix incorrect array parameter value casting when executing action via chatops or using POST /aliasexecution/match_and_execute API endpoint. The code would incorrectly assume the value is always a string, but that may not be the cast - they value could already be a list and in this case we don't want any casting to be performed. (bug fix) #5141

      Contributed by @Kami.

    • Fix @parameter_name=/path/to/file/foo.json notation in the st2 run command which didn't work correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140

      Contributed by @Kami.

    • Fix broken st2 action-alias execute command and make sure it works correctly. (bug fix) #5138

      Contributed by @Kami.

    Removed

    • Removed --python3 pack install option #5100 Contributed by @amanda11

    • Removed submit-debug-info tool and the st2debug component #5103

    • Removed check-licence script (cleanup) #5092

      Contributed by @kroustou

    • Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090

      Contributed by @blag

    • Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108

      Contributed by @winem

    Source code(tar.gz)
    Source code(zip)
  • v3.3.0(Oct 22, 2020)

    https://stackstorm.com/2020/10/22/stackstorm-v3-3-0-released/

    Added

    • Add make command to autogen JSON schema from the models of action, rule, etc. Add check to ensure update to the models require schema to be regenerated. (new feature)

    • Improved st2sensor service logging message when a sensor will not be loaded when assigned to a different partition (@punkrokk) #4991

    • Add support for a configurable connect timeout for SSH connections as requested in #4715 by adding the new configuration parameter ssh_connect_timeout to the ssh_runner group in st2.conf. (new feature) #4914

      This option was requested by Harry Lee (@tclh123) and contributed by Marcel Weinberg (@winem).

    • Added a FAQ for the default user/pass for the tools/launch_dev.sh script and print out the default pass to screen when the script completes. (improvement) #5013

      Contributed by @punkrokk

    • Added deprecation warning if attempt to install or download a pack that only supports Python 2. (new feature) #5037

      Contributed by @amanda11

    • Added deprecation warning to each StackStorm service log, if service is running with Python 2. (new feature) #5043

      Contributed by @amanda11

    • Added deprecation warning to st2ctl, if st2 python version is Python 2. (new feature) #5044

      Contributed by @amanda11

    Changed

    • Switch to MongoDB 4.0 as the default version starting with all supported OS's in st2 v3.3.0 (improvement) #4972

      Contributed by @punkrokk

    • Added an enhancement where ST2api.log no longer reports the entire traceback when trying to get a datastore value that does not exist. It now reports a simplified log for cleaner reading. Addresses and Fixes #4979. (improvement) #4981

      Contributed by Justin Sostre (@saucetray)

    • The built-in st2.action.file_writen trigger has been renamed to st2.action.file_written to fix the typo (bug fix) #4992

    • Renamed reference to the RBAC backend/plugin from enterprise to default. Updated st2api validation to use the new value when checking RBAC configuration. Removed other references to enterprise for RBAC related contents. (improvement)

    • Remove authentication headers St2-Api-Key, X-Auth-Token and Cookie from webhook payloads to prevent them from being stored in the database. (security bug fix) #4983

      Contributed by @potato and @knagy

    • Updated orquesta to version v1.2.0.

    Fixed

    • Fixed a bug where type attribute was missing for netstat action in linux pack. Fixes #4946

      Reported by @scguoi and contributed by Sheshagiri (@sheshagiri)

    • Fixed a bug where persisting Orquesta to the MongoDB database returned an error message: key 'myvar.with.period' must not contain '.'. This happened anytime an input, output, publish or context var contained a key with a . within the name (such as with hostnames and IP addresses). This was a regression introduced by trying to improve performance. Fixing this bug means we are sacrificing performance of serialization/deserialization in favor of correctness for persisting workflows and their state to the MongoDB database. (bug fix) #4932

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    • Fix a bug where passing an empty list to a with items task in a subworkflow causes the parent workflow to be stuck in running status. (bug fix) #4954

    • Fixed a bug in the example nginx HA template declared headers twice (bug fix) #4966 Contributed by @punkrokk

    • Fixed a bug in the paramiko_ssh runner where SSH sockets were not getting cleaned up correctly, specifically when specifying a bastion host / jump box. (bug fix) #4973

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    • Fixed a bytes/string encoding bug in the linux.dig action so it should work on Python 3 (bug fix) #4993

    • Fixed a bug where a python3 sensor using ssl needs to be monkey patched earlier. See also #4832, #4975 and gevent/gevent#1016 (bug fix) #4976

      Contributed by @punkrokk

    • Fixed bug where action information in RuleDB object was not being parsed properly because mongoengine EmbeddedDocument objects were added to JSON_UNFRIENDLY_TYPES and skipped. Removed this and added if to use to_json method so that mongoengine EmbeddedDocument are parsed properly.

      Contributed by Bradley Bishop (@bishopbm1 Encore Technologies)

    • Fix a regression when updated dnspython pip dependency resulted in st2 services unable to connect to mongodb remote host (bug fix) #4997

    • Fixed a regression in the linux.dig action on Python 3. (bug fix) #4993

      Contributed by @blag

    • Fixed a bug in pack installation logging code where unicode strings were not being interpolated properly. (bug fix)

      Contributed by @misterpah

    • Fixed a compatibility issue with the latest version of the logging library API where the find_caller() function introduced some new variables. (bug fix) #4923

      Contributed by @Dahfizz9897

    • Fixed another logging compatibility issue with the logging API in Python 3. The return from the logging.findCaller() implementation now expects a 4-element tuple. Also, in Python 3 there are new arguments that are passed in and needs to be acted upon, specificall stack_info that determines the new 4th element in the returned tuple. (bug fix) #5057

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    Removed

    • Removed Mistral workflow engine (deprecation) #5011

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Removed CentOS 6/RHEL 6 support #4984

      Contributed by Amanda McGuinness (@amanda11 Ammeon Solutions)

    • Removed our fork of codecov-python for CI and have switched back to the upstream version (improvement) #5002

    Source code(tar.gz)
    Source code(zip)
  • v3.2.0(Apr 29, 2020)

    https://stackstorm.com/2020/04/30/stackstorm-v3-2-0-released/

    Added

    • Add support for blacklisting / whitelisting hosts to the HTTP runner by adding new url_hosts_blacklist and url_hosts_whitelist runner attribute. (new feature) stackstorm/st2#4757
    • Add user parameter to re_run method of st2client. stackstorm/st2#4785
    • Install pack dependencies automatically. stackstorm/st2#4769
    • Add support for immutable_parameters on Action Aliases. This feature allows default parameters to be supplied to the action on every execution of the alias. stackstorm/st2#4786
    • Add get_entrypoint() method to ActionResourceManager attribute of st2client. stackstorm/st2#4791
    • Add support for orquesta task retry. (new feature)
    • Add config option scheduler.execution_scheduling_timeout_threshold_min to better control the cleanup of scheduled actions that were orphaned. stackstorm/st2#4886

    Changed

    • Install pack with the latest tag version if it exists when branch is not specialized. (improvement) stackstorm/st2#4743

    • Implement "continue" engine command to orquesta workflow. (improvement) stackstorm/st2#4740

    • Update various internal dependencies to latest stable versions (apscheduler, eventlet, kombu, amqp, pyyaml, mongoengine, python-gnupg, paramiko, tooz, webob, bcrypt).

      Latest version of mongoengine should show some performance improvements (5-20%) when writing very large executions (executions with large results) to the database. stackstorm/st2#4767

    • Improved development instructions in requirements.txt and dist_utils.py comment headers (improvement) stackstorm/st2#4774

    • Add new actionrunner.stream_output_buffer_size config option and default it to -1 (previously default value was 0). This should result in a better performance and smaller CPU utilization for Python runner actions which produce a lot of output. (improvement)

      Reported and contributed by Joshua Meyer (@jdmeyer3) stackstorm/st2#4803

    • Add new action_runner.pip_opts st2.conf config option which allows user to specify a list of command line option which are passed to pip install command when installing pack dependencies into a pack specific virtual environment. stackstorm/st2#4792

    • Refactor how orquesta handles individual item result for with items task. Before the fix, when there are a lot of items and/or result size for each item is huge, there is a negative performance impact on write to the database when recording the conductor state. (improvement)

    • Remove automatic rendering of workflow output when updating task state for orquesta workflows. This caused workflow output to render incorrectly in certain use case. The render_workflow_output function must be called separately. (improvement)

    • Update various internal dependencies to latest stable versions (cryptography, jinja2, requests, apscheduler, eventlet, amqp, kombu, semver, six) stackstorm/st2#4819 (improvement)

    • Improve MongoDB connection timeout related code. Connection and server selection timeout is now set to 3 seconds. Previously a default value of 30 seconds was used which means that for many connection related errors, our code would first wait for this timeout to be reached (30 seconds) before returning error to the end user. stackstorm/st2#4834

    • Upgrade pymongo to the latest stable version (3.10.0.). stackstorm/st2#4835 (improvement)

    • Updated Paramiko to v2.7.1 to support new PEM ECDSA key formats stackstorm/st2#4901 (improvement)

    • Remove .scrutinizer.yml config file. No longer used.

    • Convert escaped dict and dynamic fields in workflow db models to normal dict and dynamic fields. (performnce improvement)

    • Add support for PEP 508 <https://www.python.org/dev/peps/pep-0508/stackstorm/st2#environment-markers>_ environment markers in generated requirements.txt files. (improvement) stackstorm/st2#4895

    • Use pip-compile from pip-tools instead of pip-conflict-checker (improvement) stackstorm/st2#4896

    • Refactor how inbound criteria for join task in orquesta workflow is evaluated to count by task completion instead of task transition. (improvement)

    • The workflow engine orquesta is updated to v1.1.0 for the st2 v3.2 release. The version upgrade contains various new features and bug fixes. Please review the release notes for the full list of changes at https://github.com/StackStorm/orquesta/releases/tag/v1.1.0 and the st2 upgrade notes for potential impact. (improvement)

    Fixed

    • Fix the action query when filtering tags. The old implementation returned actions which have the provided name as action name and not as tag name. (bug fix) stackstorm/st2#4828

      Reported by @AngryDeveloper and contributed by Marcel Weinberg (@winem)

    • Fix the passing of arrays to shell scripts where the arrays where not detected as such by the st2 action_db utility. This caused arrays to be passed as Python lists serialized into a string.

      Reported by @kingsleyadam stackstorm/st2#4804 and contributed by Marcel Weinberg (@winem) stackstorm/st2#4861

    • Fix ssh zombies when using ProxyCommand from ssh config stackstorm/st2#4881 [Eric Edgar]

    • Fix rbac with execution view where the rbac is unable to verify the pack or uid of the execution because it was not returned from the action execution db. This would result in an internal server error when trying to view the results of a single execution. Contributed by Joshua Meyer (@jdmeyer3) stackstorm/st2#4758

    • Fixed logging middleware to output a content_length of 0 instead of Infinity when the type of data being returned is not supported. Previously, when the value was set to Infinity this would result in invalid JSON being output into structured logs. (bug fix) stackstorm/st2#4722

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    • Fix the workflow execution cancelation to proceed even if the workflow execution is not found or completed. (bug fix) stackstorm/st2#4735

    • Added better error handling to contrib/linux/actions/dig.py to inform if dig is not installed. Contributed by JP Bourget (@punkrokk Syncurity) stackstorm/st2#4732

    • Update dist_utils module which is bundled with st2client and other Python packages so it doesn't depend on internal pip API and so it works with latest pip version. (bug fix) stackstorm/st2#4750

    • Fix dependency conflicts in pack CI runs: downgrade requests dependency back to 0.21.0, update internal dependencies and test expectations (amqp, pyyaml, prance, six) (bugfix) stackstorm/st2#4774

    • Fix secrets masking in action parameters section defined inside the rule when using GET /v1/rules and GET /v1/rules/<ref> API endpoint. (bug fix) stackstorm/st2#4788 stackstorm/st2#4807

      Contributed by @Nicodemos305 and @jeansfelix

    • Fix a bug with authentication API endpoint (POST /auth/v1/tokens) returning internal server error when running under gunicorn and whenauth.api_url config option was not set. (bug fix) stackstorm/st2#4809

      Reported by @guzzijones

    • Fixed st2 execution get and st2 run not printing the action.ref for non-workflow actions. (bug fix) stackstorm/st2#4739

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    • Update st2 execution get command to always include context.user, start_timestamp and end_timestamp attributes. (improvement) stackstorm/st2#4739

    • Fixed core.sendmail base64 encoding of longer subject lines (bug fix) stackstorm/st2#4795

      Contributed by @stevemuskiewicz and @guzzijones

    • Update all the various rule criteria comparison operators which also work with strings (equals, icontains, nequals, etc.) to work correctly on Python 3 deployments if one of the operators is of a type bytes and the other is of a type unicode / string. (bug fix) stackstorm/st2#4831

    • Fix SSL connection support for MongoDB and RabbitMQ which wouldn't work under Python 3 and would result in cryptic "maximum recursion depth exceeded while calling a Python object" error on connection failure.

      NOTE: This issue only affected installations using Python 3. (bug fix) stackstorm/st2#4832 stackstorm/st2#4834

      Reported by @alexku7.

    • Fix the amqp connection setup for WorkflowExecutionHandler to pass SSL params. (bug fix) stackstorm/st2#4845

      Contributed by Tatsuma Matsuki (@mtatsuma)

    • Fix dependency conflicts by updating requests (2.23.0) and gitpython (2.1.15). stackstorm/st2#4869

    • Fix orquesta syntax error for with items task where action is misindented or missing. (bug fix) PR StackStorm/orquesta#195.

    • Fix orquesta yaql/jinja vars extraction to ignore methods of base ctx() dict. (bug fix) PR StackStorm/orquesta#196. Fixes stackstorm/st2#4866.

    • Fix parsing of array of dicts in YAQL functions. Fix regression in YAQL/Jinja conversion functions as a result of the change. (bug fix) PR StackStorm/orquesta#191.

      Contributed by Hiroyasu Ohyama (@userlocalhost)

    Removed

    • Removed Ubuntu 14.04 from test matrix stackstorm/st2#4897
    Source code(tar.gz)
    Source code(zip)
  • v3.1.0(Jul 1, 2019)

    Changed

    • Allow the orquesta st2kv function to return default for nonexistent key. (improvement) #4678
    • Update requests library to latest version (2.22.0) in requirements. (improvement) #4680
    • Disallow "decrypt_kv" filter to be specified in the config for values that are marked as "secret: True" in the schema. (improvement) #4709
    • Upgrade tooz library to latest stable version (1.65.0) so it uses latest version of grpcio library. (improvement) #4713
    • Update st2-pack-install and st2-pack-download CLI command so it supports installing packs from local directories which are not git repositories. (improvement) #4713

    Fixed

    • Fix orquesta st2kv to return empty string and null values. (bug fix) #4678
    • Allow tasks defined in the same task transition with fail to run for orquesta. (bug fix)
    • Fix workflow service to handle unexpected coordinator and database errors. (bug fix) #4704 #4705
    • Fix filter to_yaml_string to handle mongoengine base types for dict and list. (bug fix) #4700
    • Fix timeout handling in the Python runner. In some scenarios where action would time out before producing any output (stdout, stder), timeout was not correctly propagated to the user. (bug fix) #4713
    • Update st2common/setup.py file so it correctly declares all the dependencies and script files it provides. This way st2-pack-* commands can be used in a standalone fashion just by installing st2common Python package and nothing else. (bug fix) #4713
    • Fix st2-pack-download command so it works in the environments where sudo binary is not available (e.g. Docker). (bug fix) #4713
    Source code(tar.gz)
    Source code(zip)
  • v3.0.1(May 29, 2019)

    Fixed

    • Fix a bug in the remote command and script runner so it correctly uses SSH port from a SSH config file if ssh_runner.use_ssh_config parameter is set to True and if a custom (non-default) value for SSH port is specified in the configured SSH config file (ssh_runner.ssh_config_file_path). (bug fix) #4660 #4661

    • Update pack install action so it works correctly when python_versions pack.yaml metadata attribute is used in combination with --python3 pack install flag. (bug fix) #4654 #4662

    • Add source_channel back to the context used by Mistral workflows for executions which are triggered via ChatOps (using action alias).

      In StackStorm v3.0.0, this variable was inadvertently removed from the context used by Mistral workflows. (bug fix) #4650 #4656

    • Fix a bug with timestamp attribute in the execution.log attribute being incorrect when server time where st2api is running was not set to UTC. (bug fix) #4668

      Contributed by Igor Cherkaev. (@emptywee)

    • Fix a bug with some packs which use --python3 flag (running Python 3 actions on installation where StackStorm components run under Python 2) which rely on modules from Python 3 standard library which are also available in Python 2 site-packages (e.g. concurrent) not working correctly.

      In such scenario, package / module was incorrectly loaded from Python 2 site-packages instead of Python 3 standard library which broke such packs. (bug fix) #4658 #4674

    • Remove policy-delayed status to avoid bouncing between delayed statuses. (bug fix) #4655

    • Fix a possible shell injection in the linux.service action. User who had access to run this action could cause a shell command injection by passing a compromised value for either the service or action parameter. (bug fix) #4675

      Reported by James Robinson (Netskope and Veracode).

    • Replace sseclient library on which CLI depends on with sseclient-py. sseclient has various issue which cause client to sometimes hang and keep the connection open which also causes st2 execution tail command to sometimes hang for a long time. (improvement)

    • Truncate some database index names so they are less than 65 characters long in total. This way it also works with AWS DocumentDB which doesn't support longer index name at the moment.

      NOTE: AWS DocumentDB is not officially supported. Use at your own risk. (improvement) #4688 #4690

      Reported by Guillaume Truchot (@GuiTeK)

    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Apr 26, 2019)

    Added

    • Allow access to user-scoped datastore items using {{ st2kv.user.<key name> }} Jinja template notation inside the action parameter default values. (improvement) #4463

      Contributed by Hiroyasu OHYAMA (@userlocalhost).

    • Add support for new python_versions (list of string) attribute to pack metadata file (pack.yaml). With this attribute pack declares which major Python versions it supports and works with (e.g. 2 and 3).

      For backward compatibility reasons, if pack metadata file doesn't contain that attribute, it's assumed it only works with Python 2. (new feature) #4474

    • Update service bootstrap code and make sure all the services register in a service registry once they come online and become available.

      This functionality is only used internally and will only work if configuration backend is correctly configured in st2.conf (new feature) #4548

    • Add new GET /v1/service_registry/groups and GET /v1/service_registry/groups/<group_id>/members API endpoint for listing available service registry groups and members.

      Also add corresponding CLI commands - st2 service-registry group list, st2 service registry member list [--group-id=<group id>]

      NOTE: This API endpoint is behind an RBAC wall and can only be viewed by the admins. (new feature) #4548

    • Add support for ?include_attributes and ?exclude_attributes query param filter to the GET /api/v1/executions/{id} API endpoint. Also update st2 execution get CLI command so it only retrieves attributes which are displayed. (new feature) #4497

      Contributed by Nick Maludy (@nmaludy Encore Technologies)

    • Add new --encrypted flag to st2 key set CLI command that allows users to pass in values which are already encrypted.

      This attribute signals the API that the value is already encrypted and should be used as-is.

      st2 key load CLI command has also been updated so it knows how to work with values which are already encrypted. This means that st2 key list -n 100 -j < data.json ; st2 key load data.json will now also work out of the box for encrypted datastore values (values which have encrypted: True and secret: True attribute will be treated as already encrypted and used as-is).

      The most common use case for this feature is migrating / restoring datastore values from one StackStorm instance to another which uses the same crypto key.

      Contributed by Nick Maludy (Encore Technologies) #4547

    • Add source_channel to Orquesta st2() context for workflows called via ChatOps. #4600

    Changed

    • Changed the inquiries API path from /exp to /api/v1. #4495

    • Refactored workflow state in orquesta workflow engine. Previously, state in the workflow engine is not status to be consistent with st2. Other terminologies used in the engine are also revised to make it easier for developers to understand. (improvement)

    • Update Python runner code so it prioritizes libraries from pack virtual environment over StackStorm system dependencies.

      For example, if pack depends on six==1.11.0 in pack requirements.txt, but StackStorm depends on six==1.10.0, six==1.11.0 will be used when running Python actions from that pack.

      Keep in mind that will not work correctly if pack depends on a library which brakes functionality used by Python action wrapper code.

      Contributed by Hiroyasu OHYAMA (@userlocalhost). #4571

    • Improved the way that the winrm-ps-script runner sends scripts to the target Windows host. Previously the script was read from the local filesystem and serialized as one long command executed on the command line. This failed when the script was longer than either 2047 or 8191 bytes (depending on Windows version) as the Windows command line uses this as its maximum length. To overcome this, the winrm-ps-script runner now uploads the script into a temporary directory on the target host, then executes the script. (improvement) #4514

      Contributed by Nick Maludy (Encore Technologies)

    • Update various internal dependencies to latest stable versions (apscheduler, pyyaml, kombu, mongoengine, pytz, stevedore, python-editor, jinja2). #4637

    • Update logging code so we exclude log messages with log level AUDIT from a default service log file (e.g. st2api.log). Log messages with level AUDIT are already logged in a dedicated service audit log file (e.g. st2api.audit.log) so there is no need for them to also be duplicated and included in regular service log file.

      NOTE: To aid with debugging, audit log messages are also included in a regular log file when log level is set to DEBUG or system.debug config option is set to True.

      Reported by Nick Maludy. (improvement) #4538 #4502 #4621

    • Add missing --user argument to st2 execution list CLI command. (improvement) #4632

      Contributed by Tristan Struthers (@trstruth).

    • Update decrypt_kv Jinja template filter so it to throws a more user-friendly error message when decryption fails because the variable references a datastore value which doesn't exist. (improvement) #4634

    • Updated orquesta to v0.5. (improvement)

    Fixed

    • Refactored orquesta execution graph to fix performance issue for workflows with many references to non-join tasks. st2workflowengine and DB models are refactored accordingly. (improvement) StackStorm/orquesta#122.

    • Fix orquesta workflow stuck in running status when one or more items failed execution for a with items task. (bug fix) #4523

    • Fix orquesta workflow bug where context variables are being overwritten on task join. (bug fix) StackStorm/orquesta#112

    • Fix orquesta with items task performance issue. Workflow runtime increase significantly when a with items task has many items and result in many retries on write conflicts. A distributed lock is acquired before write operations to avoid write conflicts. (bug fix) Stackstorm/orquesta#125

    • Fix a bug with some API endpoints returning 500 internal server error when an exception contained unicode data. (bug fix) #4598

    • Fix the st2 workflow inspect command so it correctly passes authentication token. (bug fix) #4615

    • Fix an issue with new line characters (\n) being converted to \r\n in remote shell command and script actions which use sudo. (bug fix) #4623

    • Update service bootstrap and st2-register-content script code so non-fatal errors are suppressed by default and only logged under DEBUG log level. (bug fix) #3933 #4626 #4630

    • Fix a bug with not being able to decrypt user-scoped datastore values inside Jinja expressions using decrypt_kv Jinja filter. (bug fix) #4634

      Contributed by Hiroyasu OHYAMA (@userlocalhost).

    • Fix a bug with user-scoped datastore values not working inside action-chain workflows. (bug fix) #4634

    • Added missing parameter types to linux.wait_for_ssh action metadata. (bug fix) #4611

    • Fix HTTP runner (http-request) so it works correctly with unicode (non-ascii) body payloads. (bug fix) #4601 #4599

      Reported by Carlos Santana (@kknyxkk) and Rafael Martins (@rsmartins78).

    • Fix st2-self-check so it sets correct permissions on pack directories which it copies over to /opt/stackstorm/packs. (bug fix) #4645

    • Fix POST /v1/actions API endpoint to throw a more user-friendly error when writing data file to disk fails because of incorrect permissions. (bug fix) #4645

    Source code(tar.gz)
    Source code(zip)
  • v2.10.4(Mar 15, 2019)

    Fixed

    • Fix inadvertent regression in notifier service which would cause generic action trigger to only be dispatched for completed states even if custom states were specified using action_sensor.emit_when config option. (bug fix) Reported by Shu Sugimoto (@shusugmt). #4591

    • Make sure we don't log auth token and api key inside st2api log file if those values are provided via query parameter and not header (?x-auth-token=foo, ?st2-api-key=bar). (bug fix) #4592 #4589

    • Fix rendering of {{ config_context. }} in orquesta task that references action from a different pack (bug fix) #4570 #4567

    • Add missing default config location (/etc/st2/st2.conf) to the following services: st2actionrunner, st2scheduler, st2workflowengine. (bug fix) #4596

    • Update statsd metrics driver so any exception thrown by statsd library is treated as non fatal.

      Previously there was an edge case if user used a hostname instead of an IP address for metrics backend server address. In such scenario, if hostname DNS resolution failed, statsd driver would throw the exception which would propagate all the way up and break the application. (bug fix) #4597

      Reported by Chris McKenzie.

    Source code(tar.gz)
    Source code(zip)
  • v2.10.3(Mar 7, 2019)

    Fixed

    • Fix improper CORS where request from an origin not listed in allowed_origins will be responded with null for the Access-Control-Allow-Origin header. The fix returns the first of our allowed origins if the requesting origin is not a supported origin. Reported by Barak Tawily. (bug fix)
    Source code(tar.gz)
    Source code(zip)
  • v2.9.3(Mar 7, 2019)

    Fixed

    • Fix improper CORS where request from an origin not listed in allowed_origins will be responded with null for the Access-Control-Allow-Origin header. The fix returns the first of our allowed origins if the requesting origin is not a supported origin. Reported by Barak Tawily. (bug fix)
    Source code(tar.gz)
    Source code(zip)
  • v2.10.2(Feb 28, 2019)

    Added

    • Add support for various new SSL / TLS related config options (ssl_keyfile, ssl_certfile, ssl_ca_certs, ssl_certfile, authentication_mechanism) to the messaging section in st2.conf config file.

      With those config options, user can configure things such as client based certificate authentication, client side verification of a server certificate against a specific CA bundle, etc.

      NOTE: Those options are only supported when using a default and officially supported AMQP backend with RabbitMQ server. (new feature) #4541

    • Add metrics instrumentation to the st2notifier service. For the available / exposed metrics, please refer to https://docs.stackstorm.com/reference/metrics.html. (improvement) #4536

    Changed

    • Update logging code so we exclude log messages with log level AUDIT from a default service log file (e.g. st2api.log). Log messages with level AUDIT are already logged in a dedicated service audit log file (e.g. st2api.audit.log) so there is no need for them to also be duplicated and included in regular service log file.

      NOTE: To aid with debugging, audit log messages are also included in a regular log file when log level is set to DEBUG or system.debug config option is set to True.

      Reported by Nick Maludy. (improvement) #4538 #4502

    • Update pyyaml dependency to the latest version. This latest version fixes an issue which could result in a code execution vulnerability if code uses yaml.load in an unsafe manner on untrusted input.

      NOTE: StackStorm platform itself is not affected, because we already used yaml.safe_load everywhere.

      Only custom packs which use yaml.load with non trusted user input could potentially be affected. (improvement) #4510 #4552 #4554

    • Update Orquesta to v0.4. #4551

    Fixed

    • Fixed the packs.pack_install / !pack install {{ packs }} action-alias to not have redundant patterns. Previously this prevented it from being executed via st2 action-alias execute 'pack install xxx'. #4511

      Contributed by Nick Maludy (Encore Technologies)

    • Fix datastore value encryption and make sure it also works correctly for unicode (non-ascii) values.

      Reported by @dswebbthg, @nickbaum. (bug fix) #4513 #4527 #4528

    • Fix a bug with action positional parameter serialization used in local and remote script runner not working correctly with non-ascii (unicode) values.

      This would prevent actions such as core.sendmail which utilize positional parameters from working correctly when a unicode value was provided.

      Reported by @johandahlberg (bug fix) #4533

    • Fix core.sendmail action so it specifies charset=UTF-8 in the Content-Type email header. This way it works correctly when an email subject and / or body contains unicode data.

      Reported by @johandahlberg (bug fix) #4533 4534

    • Fix CLI st2 apikey load not being idempotent and API endpoint /api/v1/apikeys not honoring desired ID for the new record creation. #4542

    • Moved the lock from concurrency policies into the scheduler to fix a race condition when there are multiple scheduler instances scheduling execution for action with concurrency policies. #4481 (bug fix)

    • Add retries to scheduler to handle temporary hiccup in DB connection. Refactor scheduler service to return proper exit code when there is a failure. #4539 (bug fix)

    • Update service setup code so we always ignore kombu library heartbeat_tick debug log messages.

      Previously if DEBUG log level was set in service logging config file, but --debug service CLI flag / system.debug = True config option was not used, those messages were still logged which caused a lot of noise which made actual useful log messages hard to find. (improvement) #4557

    Source code(tar.gz)
    Source code(zip)
  • v2.10.1(Dec 20, 2018)

    Fixed

    • Fix an issue with GET /v1/keys API endpoint not correctly handling ?scope=all and ?user=<username> query filter parameter inside the open-source edition. This would allow user A to retrieve datastore values from user B and similar.

      NOTE: Enterprise edition with RBAC was not affected, because in RBAC version, correct check is in place which only allows users with an admin role to use ?scope=all and retrieve / view datastore values for arbitrary system users. (security issue bug fix)

    Source code(tar.gz)
    Source code(zip)
  • v2.9.2(Dec 19, 2018)

    Fixed

    • Fix an issue with GET /v1/keys API endpoint not correctly handling ?scope=all and ?user=<username> query filter parameter inside the open-source edition. This would allow user A to retrieve datastore values from user B and similar.

      NOTE: Enterprise edition with RBAC was not affected, because in RBAC version, correct check is in place which only allows users with an admin role to use ?scope=all and retrieve / view datastore values for arbitrary system users. (security issue bug fix)

    Source code(tar.gz)
    Source code(zip)
  • v2.10.0(Dec 20, 2018)

    Added

    • Added notify runner parameter to Orquesta that allows user to specify which task(s) to get notified on completion.

    • Add support for task delay in Orquesta workflows. #4459 (new feature)

    • Add support for task with items in Orquesta workflows. #4400 (new feature)

    • Add support for workflow output on error in Orquesta workflows. #4436 (new feature)

    • Added -o and -m CLI options to st2-self-check script, to skip Orquesta and/or Mistral tests. #4347

    • Allow user to specify new database.authentication_mechanism config option in /etc/st2/st2.conf.

      By default, SCRAM-SHA-1 is used with MongoDB 3.0 and later and MONGODB-CR (MongoDB Challenge Response protocol) for older servers.

      Contributed by @aduca85 #4373

    • Add new metadata_file attribute to the following models: Action, Action Alias, Rule, Sensor, TriggerType. Value of this attribute points to a metadata file for a specific resource (YAML file which contains actual resource definition). Path is relative to the pack directory (e.g. actions/my_action1.meta.yaml, aliases/my_alias.yaml, sensors/my_sensor.yaml, rules/my_rule.yaml, triggers/my_trigger.yaml etc.).

      Keep in mind that triggers can be registered in two ways - either via sensor definition file in sensors/ directory or via trigger definition file in triggers/ directory. If metadata_file attribute on TriggerTypeDB model points to sensors/ directory it means that trigger is registered via sensor definition. (new feature) #4445

    • Add new st2client.executions.get_children method for returning children execution objects for a specific (parent) execution. (new feature) #4444

      Contributed by Tristan Struthers (@trstruth).

    • Allow user to run a subset of pack tests by utilizing the new -f command line option in the st2-run-pack-tests script.

      For example:

      1. Run all tests in a test file (module):

        st2-run-pack-tests -j -x -p contrib/packs/ -f test_action_download

      2. Run a single test class

        st2-run-pack-tests -j -x -p contrib/packs/ -f test_action_download:DownloadGitRepoActionTestCase

      3. Run a single test class method

        st2-run-pack-tests -j -x -p contrib/packs/ -f test_action_download:DownloadGitRepoActionTestCase.test_run_pack_download

      (new feature) #4464

    Changed

    • Redesigned and rewritten the action execution scheduler. Requested executions are put in a persistent queue for scheduler to process. Architecture is put into place for more complex execution scheduling. Action execution can be delayed on request. (improvement)

    • core.http action now supports additional HTTP methods: OPTIONS, TRACE, PATCH, PURGE.

      Contributed by @emptywee (improvement) #4379

    • Runner loading code has been updated so it utilizes new "runner as Python package" functionality which has been introduced in a previous release. This means that the runner loading is now fully automatic and dynamic.

      All the available / installed runners are automatically loaded and registering on each StackStorm service startup.

      This means that st2ctl reload --register-runners flag is now obsolete because runners are automatically registered on service start up. In addition to that, content.system_runners_base_path and content.runners_base_paths config options are now also deprecated and unused.

      For users who wish to develop and user custom action runners, they simply need to ensure they are packaged as Python packages and available / installed in StackStorm virtual environment (/opt/stackstorm/st2). (improvement) #4217

    • Old runner names which have been deprecated in StackStorm v0.9.0 have been removed (run-local, run-local-script, run-remote, run-remote-script, run-python, http-runner). If you are still using actions which reference runners using old names, you need to update them to keep it working. #4217

    • Update various CLI commands to only retrieve attributes which are displayed in the CLI from the API (st2 execution list, st2 execution get, st2 action list, st2 rule list, st2 sensor list). This speeds up run-time and means now those commands now finish faster.

      If user wants to retrieve and view all the attributes, they can use --attr all CLI command argument (same as before). (improvement) #4396

    • Update various internal dependencies to latest stable versions (greenlet, pymongo, pytz, stevedore, tooz). #4410

    • Improve st2.conf migration for the new services by using prod-friendly logging settings by default #4415

    • Refactor Orquesta workflow to output on error. Depends on PR https://github.com/StackStorm/orquesta/pull/101 and https://github.com/StackStorm/orquesta/pull/102 (improvement)

    • Rename st2client.liveactions to st2client.executions. st2client.liveactions already represented operations on execution objects, but it was incorrectly named.

      For backward compatibility reasons, st2client.liveactions will stay as an alias for st2client.executions and continue to work until it's fully removed in a future release.

    Fixed

    • st2 login CLI commands now exits with non zero exit code when login fails due to invalid credentials. (improvement) #4338

    • Fix st2 key load that errors when importing an empty file #43

    • Fixed warning in st2-run-pack-tests about invalid format for pip list. (bug fix)

      Contributed by Nick Maludy (Encore Technologies). #4380

    • Fix a bug with st2 execution get / st2 run CLI command throwing an exception if the result field contained a double backslash string which looked like an unicode escape sequence. CLI incorrectly tried to parse that string as unicode escape sequence.

      Reported by James E. King III @jeking3 (bug fix) #4407

    • Fix a bug so timersengine config section in st2.conf has precedence over timer section if explicitly specified in the config file.

      Also fix a bug with default config values for timer section being used if user only specified timersengine section in the config. Previously user options were incorrectly ignored in favor of the default values. (bug fix) #4424

    • st2 pack install -j now only spits JSON output. Similarly, st2 pack install -y only spits YAML output. This change would enable the output to be parsed by tools. The behavior of st2 pack install hasn't changed and is human friendly. If you want to get meta information about the pack as JSON (count of actions, sensors etc), you should rely on already existing st2 pack show -j.

      Reported by Nick Maludy (improvement) #4260

    • Fix string operations on unicode data in Orquesta workflows, associated with PR https://github.com/StackStorm/orquesta/pull/98. (bug fix)

    • Fix access to st2 and action context in Orquesta workflows, associated with PR https://github.com/StackStorm/orquesta/pull/104. (bug fix)

    • st2ctl reload --register-aliases and st2ctl reload --register-all now spits a warning when trying to register aliases with no corresponding action registered in the db.

      Reported by nzlosh (improvement) #4372.

    Source code(tar.gz)
    Source code(zip)
  • v2.9.1(Oct 15, 2018)

    Changed

    • Speed up pack registration through the /v1/packs/register API endpoint. (improvement) #4342
    • Triggertypes API now sorts by trigger ref by default. st2 trigger list will now show a sorted list. (#4348)
    • Update st2-self-check script to include per-test timing information. (improvement) #4359

    Fixed

    • Update st2sensorcontainer service to throw if user wants to run a sensor from a pack which is using Python 3 virtual environment. We only support running Python runner actions from packs which use mixed Python environments (StackStorm components are running under Python 2 and particular a pack virtual environment is using Python 3). #4354
    • Update st2-pack-install and st2 pack install command so it works with local git repos (file://) which are in a detached head state (e.g. specific revision is checked out). (improvement) #4366
    • Fix a race which occurs when there are multiple concurrent requests to resume a workflow. #4369
    Source code(tar.gz)
    Source code(zip)
  • v2.9.0(Sep 22, 2018)

    Added

    • Add new runners: winrm-cmd, winrm-ps-cmd and winrm-ps-script. The winrm-cmd runner executes Command Prompt commands remotely on Windows hosts using the WinRM protocol. The winrm-ps-cmd and winrm-ps-script runners execute PowerShell commands and scripts on remote Windows hosts using the WinRM protocol.

      To accompany these new runners, there are two new actions core.winrm_cmd that executes remote Command Prompt commands along with core.winrm_ps_cmd that executes remote PowerShell commands. (new feature) #1636

      Contributed by Nick Maludy (Encore Technologies).

    • Add new ?tags, query param filter to the /v1/actions API endpoint. This query parameter allows users to filter out actions based on the tag name . By default, when no filter values are provided, all actions are returned. (new feature) #4219

    • Add a new standalone standalone st2-pack-install CLI command. This command installs a pack (and sets up the pack virtual environment) on the server where it runs. It doesn't register the content. It only depends on the Python, git and pip binary and st2common Python package to be installed on the system where it runs. It doesn't depend on the database (MongoDB) and message bus (RabbitMQ).

      It's primary meant to be used in scenarios where the content (packs) are baked into the base container / VM image which is deployed to the cluster.

      Keep in mind that the content itself still needs to be registered with StackStorm at some later point when access to RabbitMQ and MongoDB is available by running st2ctl reload --register-all. (new feature) #3912 #4256

    • Add new /v1/stream/executions/<id>/output[?output_type=all|stdout|stderr] stream API endpoint.

      This API endpoint returns event source compatible response format.

      For running executions it returns any output produced so far and any new output as it's produced. Once the execution finishes, the connection is automatically closed.

      For completed executions it returns all the output produced by the execution. (new feature)

    • Add new core.inject_trigger action for injecting a trigger instance into the system.

      Keep in mind that the trigger which is to be injected must be registered and exist in the system. (new feature) #4231 #4259

    • Add support for ?include_attributes query param filter to all the content pack resource get all (list) API endpoints (actions, rules, trigger, executions, etc.). With this query parameter user can control which API model attributes (fields) to receive in the response. In situations where user is only interested in a subset of the model attributes, this allows for a significantly reduced response size and for a better performance. (new feature) (improvement) #4300

    • Add new action_sensor.emit_when config option which allows user to specify action status for which actiontrigger is emitted. For backward compatibility reasons it defaults to all the action completed states. (improvement) #4312 #4315

      Contributed by Shu Sugimoto.

    • Improve performance of schedule action execution (POST /v1/executions) API endpoint.

      Performance was improved by reducing the number of duplicated database queries, using atomic partial document updates instead of full document updates and by improving database document serialization and de-serialization performance. (improvement) #4030 #4331

    • Ported existing YAQL and Jinja functions from st2common to Orquesta. (new feature)

    • Add error entry in Orquesta workflow result on action execution failure. (improvement)

    Changed

    • st2 key list command now defaults to --scope=all aka displaying all the datastore values (system and current user scoped) . If you only want to display system scoped values (old behavior) you can do that by passing --scope=system argument to the st2 key list command (st2 key list --scope=system). (improvement) #4221

    • The orquesta conductor implemented event based state machines to manage state transition of workflow execution. Interfaces to set workflow state and update task on action execution completion have changed and calls to those interfaces are changed accordingly. (improvement)

    • Change GET /v1/executions/<id>/output API endpoint so it never blocks and returns data produced so far for running executions. Behavior for completed executions is the same and didn't change - all data produced by the execution is returned in the raw format.

      The streaming (block until execution has finished for running executions) behavior has been moved to the new /stream/v1/executions/<id>/output API endpoint.

      This way we are not mixing non-streaming (short lived) and streaming (long lived) connections inside a single service (st2api). (improvement)

    • Upgrade mongoengine (0.15.3) and pymongo (3.7.1) to the latest stable version. Those changes will allow us to support MongoDB 3.6 in the near future.

      New version of mongoengine should also offer better performance when inserting and updating larger database objects (e.g. executions). (improvement) #4292

    • Trigger parameters and payload schema validation is now enabled by default (system.validate_trigger_parameters and system.validate_trigger_payload config options now default to True).

      This means that trigger parameters are now validated against the parameters_schema defined on the trigger type when creating a rule and trigger payload is validated against payload_schema when dispatching a trigger via the sensor or via the webhooks API endpoint.

      This provides a much safer and user-friendly default value. Previously we didn't validate trigger payload for custom (non-system) triggers when dispatching a trigger via webhook which meant that webhooks API endpoint would silently accept an invalid trigger (e.g. referenced trigger doesn't exist in the database or the payload doesn't validate against the payload_schema), but TriggerInstanceDB object would never be created because creation failed inside the st2rulesengine service. This would make such issues very hard to troubleshoot because only way to find out about this failure would be to inspect the st2rulesengine service logs. (improvement) #4231

    • Improve code metric instrumentation and instrument code and various services with more metrics. Also document various exposed metrics. Documentation can be found at https://docs.stackstorm.com/latest/reference/metrics.html (improvement) #4310

    • Add new metrics.prefix config option. With this option user can specify an optional prefix which is prepended to each metric key (name). This comes handy in scenarios where user wants to submit metrics from multiple environments / deployments (e.g. testing, staging, dev) to the same backend instance. (improvement) #4310

    • Improve st2 execution tail CLI command so it also supports Orquesta workflows and arbitrarily nested workflows. Also fix the command so it doesn't include data from other unrelated running executions. (improvement) #4328

    • Change default NGINX configuration to use HTTP 308 redirect, rather than 301, for plaintext requests. #4335

    • Improve performance of the GET /v1/actions/views/overview API endpoint. (improvement) #4337

    Fixed

    • Fix an issue with AttributeError: module 'enum' has no attribute 'IntFlag' error which would appear when using Python 3 for a particular pack virtual environment and running on RHEL / CentOS. (bug fix) #4297

    • Fix a bug with action runner throwing an exception and failing to run an action if there was an empty pack config inside /opt/stackstorm/configs/. (bug fix) #4325

    • Fix action_sensor.enable config option so it works correctly if user sets this option to a non-default value of True. (bug fix) #4312 #4315

      Contributed by Shu Sugimoto.

    • Update GET /v1/actions/views/entry_point/<action ref> to return correct Content-Type response header based on the entry point type / file extension. Previously it would always incorrectly return application/json. (improvement) #4327

    Deprecated

    • The CloudSlang runner is now deprecated. In StackStorm 3.1 it will be removed from the core StackStorm codebase. The runner code will be moved to a separate repository, and no longer maintained by the core StackStorm team. Users will still be able to install and use this runner, but it will require additional steps to install.
    • The winexe-based Windows runners are now deprecated. They will be removed in StackStorm 3.1. They have been replaced by pywinrm-based Windows runners. See https://docs.stackstorm.com/latest/reference/runners.html#winrm-command-runner-winrm-cmd for more on using these new runners.
    Source code(tar.gz)
    Source code(zip)
  • v2.8.1(Jul 17, 2018)

    Added

    • Update st2 CLI to inspect COLUMNS environment variable first when determining the terminal size. Previously this environment variable was checked second last (after trying to retrieve terminal size using various OS specific methods and before falling back to the default value).

      This approach is more performant and allows user to easily overwrite the default value or value returned by the operating system checks - e.g. by running COLUMNS=200 st2 action list. (improvement) #4242

    Changed

    • Update st2client/setup.py file to dynamically load requirements from st2client/requirements.txt file. The code works with pip >= 6.0.0, although using pip 9.0.0 or higher is strongly recommended. (improvement) #4209

    • Update st2 CLI to use a more sensible default terminal size for table formatting purposes if we are unable to retrieve terminal size using various system-specific approaches.

      Previously we would fall back to a very unfriendly default of 20 columns for a total terminal width. This would cause every table column to wrap and make output impossible / hard to read. (improvement) #4242

    Fixed

    • Fixed a bug where secret: true was not applying to full object and array trees. (bugfix) #4234 Reported by @jjm

      Contributed by Nick Maludy (Encore Technologies).

    • Mark password http-runner parameter as a secret. (bug fix) #4245

      Reported by @daniel-mckenna

    Source code(tar.gz)
    Source code(zip)
  • v2.8.0(Jul 10, 2018)

    Added

    • Orchestra - new StackStorm-native workflow engine. This is currently in beta. (new feature)

    • Added metrics for collecting performance and health information about the various ST2 services and functions. (new feature) #4004 #2974

    • When running a dev (unstable) release include git revision hash in the output when using st2 --version CLI command. (new feature) #4117

    • Update rules engine to also create rule enforcement object when trigger instances fails to match a rule during the rule matching / filtering phase due to an exception in the rule criteria (e.g. invalid Jinja expression, etc.).

    • This change increases visibility into rules which didn't match due to an exception. Previously this was only visible / reflected in the rules engine log file. (improvement) #4134

    • Add new GET /v1/ruleenforcements/views[/] API endpoints which allow user to retrieve RuleEnforcement objects with the corresponding TriggerInstance and Execution objects. (new feature) #4134

    • Add new status field to the RuleEnforcement model. This field can contain the following values - succeeded (trigger instance matched a rule and action execution was triggered successfully), failed (trigger instance matched a rule, but it didn't result in an action execution due to Jinja rendering failure or other exception). (improvement) #4134 #4152

    • Add trigger type reference based filtering to the /v1/triggerinstances API endpoint - e.g. /v1/triggerinstances?trigger_type=core.st2.webhook. (new feature) #4151

    • Add new --python3 flag to st2 pack install CLI command and python3 parameter to packs.{install,setup_virtualenv} actions. When the value of this parameter is True, it uses python3 binary when creating virtual environment for that pack (based on the value of actionrunner.python3_binary config option).

      Note 1: For this feature to work, Python 3 needs to be installed on the system, virtualenv package installed on the system needs to support Python 3 (it needs to be a recent version) and pack in question needs to support Python 3.

      Note 2: This feature is experimental and opt-in. (new feature) #4016 #3922 #4149

    • Add two new Jinja filters - basename (os.path.basename) and dirname (os.path.dirname). #4184

      Contributed by Florian Reisinger (@reisingerf).

    Changed

    • Update st2 CLI to create the configuration directory and file, and authentication tokens with secure permissions (eg: readable only to owner) #4173

    • Refactor the callback module for the post run in runner to be more generic. (improvement)

    • Update various Python dependencies to the latest stable versions (gunicorn, gitpython, python-gnupg, tooz, flex). #4110

    • Update all the service and script entry points to use /etc/st2/st2.conf as a default value for the config file location.

    • This way users don't need to explicitly provide --config-file CLI argument when running various scripts (e.g. st2-track-result, st2-apply-rbac-definitions, etc.) and when they just want to use a default config file. (improvement) #4111

    • Update st2 CLI to print a warning if a non-unicode system locale which would prevent StackStorm to function correctly in some scenarios is used. (improvement) #4127 #4120

    • Upgrade various internal Python library dependencies to the latest stable versions (kombu, amqp, gitpython, pytz, semver, oslo.utils). (improvement) #4162

    • Move from keyczar library to cryptography library for handling symmetric encryption and decryption (secret datastore values).

      Note: This change is fully backward compatible since it just changes the underlying backend and implementation details. The same underlying encryption algorithm is used (AES256 in CBC mode with HMAC signature). (improvement) #4165

    Fixed

    • Fixed a bug where secrets in pack configs weren't being masked.

      Recently we introduced support for nested objects and arrays. Secret parameters within these nested objects and arrays were not being masked. The fix involves us fully traversing deeply nested objects and arrays and masking out any variables marked as secret. This means we now support pack config JSON schemas with type: object and its corresponding parameters: {} stanza, along with type: array and its corresponding items: {} stanza. We still do NOT support JSON schema combinations that includes the anyOf, allOf, oneOf, and not keywords. (bug fix) #4139

      Contributed by Nick Maludy (Encore Technologies).

    • Style clean up to transport queues module and various config modules. (improvement)

    • Fixed CLI help for st2 action-alias match and execute. (#4174).

    • Fix regression in ?include_attributes query param filter in the /v1/executions API endpoint. (bug fix) #4226

    Source code(tar.gz)
    Source code(zip)
  • v2.7.2(May 16, 2018)

    Changed

    • Reduce load on LDAP server and cache user groups response in an in-memory cache when RBAC remote LDAP group to local RBAC role synchronization feature is enabled.

      Previously on authentication the code would hit LDAP server multiple times to retrieve user groups. With this change, user LDAP groups are only retrieved once upon authentication and cached and re-used in-memory by default for 120 seconds.

      This reduces load on LDAP server and improves performance upon regular and concurrent user authentication.

      This functionality can be disabled by setting cache_user_groups_response LDAP authentication backend kwarg to false.

      Note: This change only affects users which utilize RBAC with remote LDAP groups to local RBAC roles synchronization feature enabled. (enterprise) (bug fix) #4103 #4105

    Fixed

    • Fix an issue (race condition) which would result in not all the remote LDAP groups being synchronized with local RBAC roles if a user tried to authenticate with the same auth token concurrently in a short time frame.

      Note: This issue only affects users which utilize RBAC with remote LDAP groups to local RBAC roles synchronization feature enabled. (enterprise) (bug fix) #4103 #4105

    • Fix an issue with some sensors which rely on select.poll() (FileWatch, GithubSensor, etc.) stopped working with StackStorm >= 2.7.0.

      StackStorm v2.7.0 inadvertently introduced a change which broke a small set of sensors which rely on select.poll() functionality. (bug fix) #4118

    • Throw if id CLI argument is not passed to the st2-track-result script. (bug fix) #4115

    • Fixed pack config's not properly rendering Jinja expressions within lists. (bugfix) #4121

      Contributed by Nick Maludy (Encore Technologies).

    • Fixed pack config rendering error throw meaningful message when a Jinja syntax error is encountered. (bugfix) #4123

      Contributed by Nick Maludy (Encore Technologies).

    Source code(tar.gz)
    Source code(zip)
  • v2.7.1(Apr 24, 2018)

    Changed

    • When creating a pack environment during the pack installation, we now pass --no-download flag to the virtualenv binary. This way version of pip, wheel and distutils which is enforced by virtualenv is used instead of downloading the latest stable versions from PyPi.

      This results in more reproducible pack virtual environments and we also ensure pip 9.0 is used ( there are some known issues with pip 10.0).

      If for some reason you want to revert to the old behavior, you can do that by passing no_download=False parameter to the packs.setup_virtualenv action. #4085

    Fixed

    • Fix st2 pack search and POST /api/v1/packs/index/search API endpoint so it doesn't return internal server error when a single pack search term is provided. (bug fix) #4083
    Source code(tar.gz)
    Source code(zip)
  • v2.7.0(Apr 12, 2018)

    Added

    • Update st2 execution tail command so it supports double nested workflows (workflow -> workflow -> execution). Previously, only top-level executions and single nested workflows (workflow -> execution) were supported. (improvement) #3962 #3960

    • Add support for utf-8 / unicode characters in the pack config files. (improvement) #3980 #3989

      Contributed by @sumkire.

    • Added the ability of st2ctl to utilize environment variables from /etc/default/st2ctl (for Ubuntu/Debian) and /etc/sysconfig/st2ctl (RHEL/CentOS). This allows deployments to override COMPONENTS and ST2_CONF in a global location so st2ctl can start/stop/restart selected components and utilize a non-default location for st2.conf. (new feature) #4027

      Contributed by Nick Maludy (Encore Technologies).

    • Add support for new optional content_version runner parameter to the Python and Local Shell Script runner. This parameter can contain a git commit hash / tag / branch from a pack git repository and runner will ensure this revision of the pack content (Python action / local shell script action) is used for a particular action execution.

      Keep in mind that providing this parameter only ensures a particular revision of the pack content is used. Python runner virtual environment and dependencies are outside of this scope.

      Note: To be able to utilize this functionality, git version >= 2.5.0 must be installed on the system. (new feature) #3997

    • Update windows runner to correctly handle and use timeout action execution status. (improvement) #4047

    • Add missing scope, decrypt and encrypt arguments to the datastore management related methods on the SensorService class. (improvement) #3895 #4057 #4058

      Reported by @djh2020, @mxmader.

    Changed

    • Modified RabbitMQ connection error message to make clear that it is an MQ connection issue. #3992

    • Additional refactor which makes action runners fully standalone and re-distributable Python packages. Also add support for multiple runners (runner modules) inside a single Python package and consolidate Python packages from two to one for the following runners: local runners, remote runners, windows runners. (improvement) #3999

    • Upgrade eventlet library to the latest stable version (0.22.1) (improvement) #4007 #3968

    • Increase maximum retry delay for action.retry policy from 5 seconds to 120 seconds. Because of the way retries are currently implemented (they are not st2notifier service restart safe), long retry delays are not recommended. For more information on this limitation please refer to the documentation - https://docs.stackstorm.com/reference/policies.html#retry. #3630 #3637

    • Update Python runner so it throws a more user-friendly exception in case Python script tries to access a key in self.config dictionary which doesn't exist. (improvement) #4014

    • Update various Python dependencies to the latest stable versions (apscheduler, gitpython, pymongo, stevedore, paramiko, tooz, flex, webob, prance).

    • Refactored mistral runner to support callback from mistral instead of relying on st2resultstracker. This reduces the unnecessary traffic and CPU time by querying the mistral API. Included a command to manually add a state entry for Mistral workflow execution to recover from any callback failures. (improvement)

    • Throw a more user-friendly error when writing pack data files to disk and when an invalid file path is provided (e.g. path is outside the pack directory, etc.). (improvement) #4039 #4046

    • Change the output object returned by Windows runners so it matches the format from the local and remote runner.

      Note: This change is backward incompatible - result attribute has been removed (same information is available in stdout attribute), exit_code renamed to return_code and two new attributes added - succeeded and failed.

      For more information, please refer to the upgrade notes. #4044 #4047

    Fixed

    • Fix Python runner actions and Argument list too long error when very large parameters are passed into the action. The fix utilizes stdin to pass parameters to the Python action wrapper process instead of CLI argument list. (bug fix) #1598 #3976

    • Fix a regression in POST /v1/webhooks/<webhook name> API endpoint introduced in v2.4.0 and add back support for arrays. In 2.4.0 support for arrays was inadvertently removed and only objects were supported. Keep in mind that this only applies to custom user-defined webhooks and system st2 webhook still requires input to be an object (dictionary). (bug fix) #3956 #3955

    • Fix a bug in the CLI causing st2 execution pause and st2 execution resume to not work. (bugfix) #4001

      Contributed by Nick Maludy (Encore Technologies).

    • Fixed missing "paused" status option from "st2 execution list" help output. (bugfix) #4037

      Contributed by Ben Hohnke (NTT Communications ICT Solutions)

    • Fix "st2 pack install" command so it doesn't require access to pack index (index.stackstorm.org) when installing a local pack (pack name starting with "file://"). (bug fix) #3771 #3772

    • Fix rules engine so it correctly handles and renders action parameters which contain Jinja expressions and default values. (bug fix) #4050 #4050

      Reported by @rakeshrm.

    • Make sure observer system role also grants pack_search permission. (bug fix) #4063 #4064

      Reported by @SURAJTHEGREAT.

    • Fix st2 webhook get -h which was asking for a name or id as opposed to the URL of the webhook. Also, fix st2 webhook list to explicitly add a webhook column. (bugfix) #4048

    Source code(tar.gz)
    Source code(zip)
  • v2.6.0(Jan 29, 2018)

    https://stackstorm.com/2018/01/25/new-year-new-stackstorm-v2-6-released/

    Added

    • Add new get_user_info method to action and sensor service. With this method, user can retrieve information about the user account which is used to perform datastore operations inside the action and sensor service. (new feature) #3831

    • Add new /api/v1/user API endpoint. This API endpoint is only available to the authenticated users and returns various metadata on the authenticated user (which method did the user use to authenticate, under which username the user is authenticated, which RBAC roles are assignment to this user in case RBAC is enabled, etc.) (new feature) #3831

    • The /api/v1/match_and_execute API endpoint matches a single alias and executes multiple times if the alias format has a match_multiple key set to true. Please refer to the documentation for usage. #3884

      Contributed by @ahubl-mz.

    • Add ability to share common code between python sensors and python actions. You can now place common code inside a lib directory inside a pack (with an __init__.py inside lib directory to declare it a python package). You can then import the common code in sensors and actions. Please refer to documentation for samples and guidelines. #3490

    • Add support for password protected sudo to the local and remote runner. Password can be provided via the new sudo_password runner parameter. (new feature) #3867

    • Add new --tail flag to the st2 run / st2 action execute and st2 execution re-run CLI command. When this flag is provided, new execution will automatically be followed and tailed after it has been scheduled. (new feature) #3867

    • Added flag --auto-dict to st2 run and st2 execution re-run commands. This flag must now be specified in order to automatically convert list items to dicts based on presence of colon (:) in all of the list items (new feature) #3909

    • Allow user to set default log level used by all the Python runner actions by setting actionrunner.pythonrunner```` option inst2.conf`` (new feature) #3929

    • Update st2client package which is also utilized by the CLI so it also works under Python 3.

      Note: Python 2.7 is only officially supported and tested Python version. Using Python 3 is at your own risk - they are likely still many bugs related to Python 3 compatibility. You have been warned. (new feature) #3929 #3932

      Contributed by Anthony Shaw.

    • Add ?limit=-1 support for the API to fetch full result set (CLI equivalent flag --last/-n). Post error message for limit=0 and fix corner case where negative values for limit query param were not handled correctly. #3761 #3708 #3735

    • Only allow RBAC admins to retrieve all the results at once using ?limit=-1 query param, upate the code so api.max_page_size config option only applies to non-admin users, meaning users with admin permission can specify arbitrary value for ?limit query param which can also be larger than api.max_page_size. (improvement) #3939

    • Add new ?include_attributes query param filter to /v1/executions/ API endpoint With this filter user can select which fields to include in the response (whitelist approach, opposite of the existing ?exclude_attributes filter).

      For example, if you only want to retrieve id and status field, the URL would look like this - /v1/executions?include_attributes=id,status. (new feature) #3953 #3858 #3856

    Changed

    • st2actions.runners.pythonrunner.Action class path for base Python runner actions has been deprecated since StackStorm v1.6.0 and will be fully removed in StackStorm v2.7.0. If you have any actions still using this path you are encouraged to update them to use st2common.runners.base_action.Action path. #3803
    • Refactor st2common Python package so it's fully self sustaining and can be used in a standalone manner. (improvement) #3803
    • Refactor Python action runner so it only depends on st2common Python package (previously it also depended on st2actions) and can be used in a standalone mode. Previously pack config and and some other parameters were retrieved inside the Python process wrapper, but now they are retrieved inside the runner container and passed to the runner. This also makes it easier to add support for pack configs to other runners in the future. (improvement) #3803
    • Update various Python dependencies to the latest stable versions (kombu, amqp, apscheduler, gitpython, pymongo, stevedore, paramiko, prompt-toolkit, flex). #3830
    • Mask values in an Inquiry response displayed to the user that were marked as "secret" in the inquiry's response schema. #3825
    • Real-time action output streaming is now enabled by default. For more information on this feature, please refer to the documentation - https://docs.stackstorm.com/latest/reference/action_output_streaming.html. You can disable this functionality by setting actionrunner.stream_output config option in st2.conf to False and restart the services (sudo st2ctl restart).

    Fixed

    • Fully fix performance regressions for short Python runner actions introduced in the past and partially fixed in #3809. (bug fix) #3803

    • Fix 'NameError: name 'cmd' is not defined' error when using linux.service with CentOS systems. #3843. Contributed by @shkadov

    • Fix bugs with newlines in execution formatter (client) (bug fix) #3872

    • Fixed st2ctl status to use better match when checking running process status. #3920

    • Removed invalid st2ctl option to re-open Mistral log files. #3920

    • Update garbage collection service and st2-purge-executions CLI tool and make deletion more efficient. Previously we incorrectly loaded all the execution fields in memory, but there was no need for that and now we only retrieve and load id which is the only field we need. #3936

      Reported by @kevin-vh.

    Source code(tar.gz)
    Source code(zip)
  • v2.5.1(Dec 15, 2017)

    https://stackstorm.com/2017/12/19/early-christmas-stackstorm-patch-release-2-5-1/

    Added

    • Add new log_level runner parameter to Python runner. With this parameter, user can control which log messages generated by Python runner actions are output to action stderr. For backward compatibility reasons it defaults to debug. This functionality comes handy in situations when an action depends on an external library which logs a lot of information under debug, but you only want to see messages with log level error or higher (or similar). (new feature) #3824

    • Add stevedore related metadata to Python package setup.py files for runner packages. This way runners can be installed using pip and dynamically enumerated and loaded using stevedore and corresponding helper functions.

      All runners are now also fully fledged Python packages (previously they were single module Python packages which caused various install and distribution related issues when installing them via pip) (new feature)

    • Add new search rule criteria comparison operator. Please refer to the documentation for usage. (new feature) #3833

      Contributed by @ahubl-mz.

    • Added flag --auto-dict to st2 run and st2 execution re-run commands. This flag must now be specified in order to automatically convert list items to dicts based on presence of colon (:) in all of the list items (new feature) #3909

    Changed

    • Update the output of st2 execution {run,get} CLI command to colorize the value of the status attribute (green for succeeded, red for failed, etc. aka the same as for the output of st2 execution list command). (improvement) #3810

      Contributed by Nick Maludy (Encore Technologies).

    • Update log messages in the datastore service to correctly use DEBUG log level instead of AUDIT. #3845

    Fixed

    • Fix log messages generated by Python runner actions to include the correct action class name. Previously they always incorrectly used "ABCMeta" instead of the actual action class name. (bug fix) #3824
    • Fix st2 execution tail [last] CLI command so it doesn't throw an exception if there are no executions in the database. (bug fix) #3760 #3802
    • Fix a bug with datastore service used inside the Python runner actions not correctly scoping the auth token to the user who triggered the action. Token was incorrectly scoped to api_service user without any permissions. (bug fix) #3823 #3535
    • Fix edge case for workflows stuck in running state. When Mistral receives a connection error from the st2 API on requesting action execution, there's a duplicate action execution stuck in requested state. This leads to the st2resultstracker assuming the workflow is still running.
    • Fix a regression and a bug with no API validation being performed and API returning 500 instead of 400 status code if user didn't include any request payload (body) when hitting POST and PUT API endpoints where body is mandatory. (bug fix) #3864
    • Fix a bug in Python runner which would cause action log messages to be duplicated in action stderr output when utilizing action service / datastore service inside actions. (bug fix) #3893
    Source code(tar.gz)
    Source code(zip)
  • v2.5.0(Oct 27, 2017)

    https://stackstorm.com/2017/10/26/stackstorm-2-5-hit-streets/

    Added

    • Add new feature which allows runner action output (stdout and stderr) to be streamed and consumed in real-time by using one of the following approaches:

      • /v1/executions/<execution id>/output[?type=stdout/stderr] API endpoint.
      • /v1/stream/ stream endpoint and listening for st2.execution.stdout__create and st2.execution.output__create /v1/stream stream API endpoint events.
      • st2 execution tail <execution id> [--type=stdout/stderr] CLI command (underneath it uses stream API endpoint).

      Right now this functionality is available for the following runners:

      • local command runner
      • local script runner
      • remote command runner
      • remote script runner
      • python runner

      Note: This feature is still experimental and it's disabled by default (opt-in). To enable it, set actionrunner.stream_output config option to True.

      (new feature) #2175 #3657 #3729

    • Update st2 role-assignment list RBAC CLI command to include information about where a particular assignment comes from (from which local assignment or mapping file). (improvement) #3763

    • Add support for overlapping RBAC role assignments for assignments via remote LDAP group to StackStorm role mappings. This means that the same role can now be granted via multiple RBAC mapping files. #3763

    • Add new Jinja filters from_json_string, from_yaml_string, and jsonpath_query. #3763

    • Add new "Inquiry" capability, which adds ability to "ask a question", usually in a workflow. Create a new runner type: "inquirer" to support this, as well as new API endpoints and client commands for interacting with Inquiries

      Contributed by mierdin. #3653

    • Added two new rule operators, inside and ninside which allow for the reverse intent of the contains and ncontains operators. #3781

      Contributed by @lampwins.

    • Allow user to use more expressive regular expressions inside action alias format string by allowing them to specify start (^) and end ($) anchors. Previously, those anchors were automatically added at the beginning and end of the alias format string. Now they are only added if a format string doesn't already contain them. #3789

      Contributed by @ahubl-mz.

    • Add new POST /v1/aliasexecution/match_and_execute API endpoint which allows user to schedule an execution based on a command string if a matching alias is found in the database.

      This API endpoint is meant to be used with chat bot plugins. It allows them to be simple thin wrappers around this API endpoint which send each chat line to this API endpoint and handle the response. #3773

    • Add several improvements to the installation scripts: They support using proxy servers. ~stanley no longer has to be /home/stanley. In addition to the on-screen display, the output from the installation script is now logged to a file beginning with st2-install under /var/log/st2/. Furthermore, the script handles re-runs better, although it's not fully idempotent yet. More improvements are expected in the near future. st2-packages: #505, #506, #507, #508, #509, #510, #512, #517.

    Fixed

    • Fix a bug where sensor watch queues were not deleted after sensor container process was shut down. This resulted in spurious queues left behind. This should not have caused performance impact but just messes with rabbitmqadmin output and maybe tedious for operators. (bug fix) #3628

      Reported by Igor.

    • Make sure all the temporary RabbitMQ queues used by the stream service are deleted once the connection to RabbitMQ is closed. Those queues are temporary and unique in nature and new ones are created on each service start-up so we need to make sure to correctly clean up old queues. #3746

    • Fix cancellation of subworkflow and subchain. Cancel of Mistral workflow or Action Chain is cascaded down to subworkflows appropriately. Cancel from tasks in the workflow or chain is cascaded up to the parent. (bug fix)

    • Fix delays in st2resultstracker on querying workflow status from Mistral. Make sleep time for empty queue and no workers configurable. Reduce the default sleep times to 5 seconds. StackStorm instances that handle more workflows should consider increasing the query interval for better CPU utilization.

    • Fix missing type for the parameters with enum in the core st2 packs.(bug fix) #3737

      Reported by Nick Maludy.

    • Add missing -h / --help CLI flag to the following execution CLI commands: cancel, pause, resume. (bug fix) #3750

    • Fix execution cancel and pause CLI commands and make id a required argument. (bug fix) #3750

    • Fix st2 role-assignment list CLI command and allow --user, --remote and --role arguments to be used together. Previously they were mutually exclusive so it wasn't possible to use them together. (bug fix) #3763

    • Update default event name whitelist for /v1/stream API endpoint and make sure st2.announcement__errbot and other event names starting with st2.announcement__* prefix are not filtered out. #3769 (bug fix)

      Reported by Carlos.

    • Fix action-alias execute response to show execution id and matching action-alias #3231 (bug fix) Reported by Carlos.

    • Fix st2 apikey load command to update an existing entry if items in input file contain id attribute and item already exists on the server. This way the behavior is consistent with st2 key load command and the command is idempotent if each item contains id attribute. #3748 #3786

      Reported by Christopher Baklid.

    • Don't log MongoDB database password if user specifies URI for database.db_host config parameter and that URI also includes a password. Default and a common scenario is specifying password as a separate database.password config parameter. #3797

      Reported by Igor Cherkaev.

    • Fix POST /v1/actionalias/match API endpoint to correctly return a dictionary instead of an array. We had a correct OpenAPI definition for the response, but the code incorrectly returned an array instead of a dictionary.

      Note: This is a breaking change so if your code utilizes this API endpoint you need to update to treat response as a dictionary and not as an array with a single item. #377

    • Partially fix performance overhead and regression for short and simple Python runner actions. Full / complete fix will be included in v2.6.0. #3809

    Changed

    • Minor language and style tidy up of help strings and error messages #3782
    Source code(tar.gz)
    Source code(zip)
  • v2.4.1(Sep 13, 2017)

    https://stackstorm.com/2017/09/18/quick-one-st2-2-4-1-released/

    Fixed

    • Fix a bug with /v1/packs/install and /v1/packs/uninstall API endpoints incorrectly using system user for scheduled pack install and pack uninstall executions instead of the user which performed the API operation.(bug fix) #3693 #3696

      Reported by theuiz.

    • Fix mistral callback failure when result contains unicode. (bug fix)

    • Fix cancellation of delayed action execution for tasks in workflow. (bug fix)

    • Fix timeout of mistral shutdown in systemd service. The fix is done upstream. https://review.openstack.org/#/c/499853/ (bug fix)

    Source code(tar.gz)
    Source code(zip)
  • v2.4.0(Aug 24, 2017)

    https://stackstorm.com/2017/08/24/whats-stackstorm-2-4-already/

    Added

    • Add sample passive sensor at contrib/examples/sensors/echo_flask_app. (improvement) #3667

    • Add pack config into action context. This is made available under the config_context key. #3183

    • Add limit/"-n" flag and pagination note(stderr) in the CLI for st2 key list. Default limit is 50. #3641

    • Implement pause and resume for Mistral workflow and Action Chain. Pause and resume will cascade down to subworkflows and/or subchains. Pause from a subworkflow or subchain will cascade up to the parent workflow. (new feature)

    • Add pack index endpoint. It will made a request for every index defined in st2.conf and return the combined list of available packs.

    • Added a new field timestamp_f to the GELF logging formatter that represents the time of the logging even in fractional time (resolution is dependent on your system). This allows adjacent logging events to be distinguished more accurately by the time they occurred. Contributed by Nick Maludy (Encore Technologies) #3362

    • Require new STREAM_VIEW RBAC permission type to be able to view /v1/stream stream API endpoint. (improvement) #3676

    • Add new ?events, ?action_refs and ?execution_ids query params to /v1/stream/ API endpoint. Those query parameters allow user to filter out which events to receive based on the event type, action ref and execution id. By default, when no filters are provided, all events are returned. (new feature) #3677

    • Show count of pack content (actions, sensors, triggers, rules and aliases) to be registered before the st2 pack install so that the delay in install is not mistaken as no response or hanging command. (improvement) #3586 #3675

    • Allow user to specify value for "array of objects" parameter type using a simple notation when using st2 run CLI command. (improvement) #3646 #3670

      Contributed by Hiroyasu OHYAMA.

    Changed

    • Rename ST2 action runner cancel queue from st2.actionrunner.canel to st2.actionrunner.cancel. (improvement) #3247
    • Install scripts and documentation has been updated to install MongoDB 3.4 by default (previously 3.2 was installed by default). If you want to upgrade an existing installation, please follow official instructions at https://docs.mongodb.com/v3.4/release-notes/3.4-upgrade-standalone/. (improvement)

    Removed

    • The feature to use local config.yaml in packs is removed.

    Fixed

    • Fix retrying in message bus exchange registration. (bug fix) #3635 #3638

      Reported by John Arnold.

    • Fix message bus related race condition which could, under some rare scenarios, cause first published message to be ignored because there were no consumers for that particular queue yet. This could happen in a scenario when API service came online and served a request before action runner service came online.

      This also fixes an issue with Redis kombu backend not working. (bug fix) #3635 #3639 #3648

    • Fix logrotate configuration to delete stale compressed st2actionrunner logs #3647

    • Fix trace list API endpoint sorting by start_timestamp, using ?sort_desc=True|False query parameters and by passing --sort=asc|desc parameter to the st2 trace list CLI command. Descending order by default.(bug fix) #3237 #3665

    • Fix pack index health endpoint. It now points to the right controller. #3672

    • Fix 'pack register content' failures appearing on some slower systems by lifting action timeout #3685

    Source code(tar.gz)
    Source code(zip)
Owner
StackStorm
Event-driven automation. Used by Netflix & many more....
StackStorm
a CLI that provides a generic automation layer for assessing the security of ML models

Counterfit About | Getting Started | Learn More | Acknowledgments | Contributing | Trademarks | Contact Us -------------------------------------------

Microsoft Azure 575 Jan 2, 2023
Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Linux, Jenkins, AWS, SRE, Prometheus, Docker, Python, Ansible, Git, Kubernetes, Terraform, OpenStack, SQL, NoSQL, Azure, GCP, DNS, Elastic, Network, Virtualization. DevOps Interview Questions

Arie Bregman 35.1k Jan 2, 2023
Repository tracking all OpenStack repositories as submodules. Mirror of code maintained at opendev.org.

OpenStack OpenStack is a collection of interoperable components that can be deployed to provide computing, networking and storage resources. Those inf

Mirrors of opendev.org/openstack 4.6k Dec 28, 2022
This projects provides the documentation and the automation(code) for the Oracle EMEA WLA COA Demo UseCase.

COA DevOps Training UseCase This projects provides the documentation and the automation(code) for the Oracle EMEA WLA COA Demo UseCase. Demo environme

Cosmin Tudor 1 Jan 28, 2022
CTF infrastructure deployment automation tool.

CTF infrastructure deployment automation tool. Focus on the challenges. Mirrored from

Fake News 1 Apr 12, 2022
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
GitGoat enables DevOps and Engineering teams to test security products intending to integrate with GitHub

GitGoat is an open source tool that was built to enable DevOps and Engineering teams to design and implement a sustainable misconfiguration prevention strategy. It can be used to test with products with access to GitHub repositories without a risk to your production environment.

Arnica 149 Dec 22, 2022
A Blazing fast Security Auditing tool for Kubernetes

A Blazing fast Security Auditing tool for kubernetes!! Basic Overview Kubestriker performs numerous in depth checks on kubernetes infra to identify th

Vasant Chinnipilli 934 Jan 4, 2023
Hubble - Network, Service & Security Observability for Kubernetes using eBPF

Network, Service & Security Observability for Kubernetes What is Hubble? Getting Started Features Service Dependency Graph Metrics & Monitoring Flow V

Cilium 2.4k Jan 4, 2023
Bugbane - Application security tools for CI/CD pipeline

BugBane Набор утилит для аудита безопасности приложений. Основные принципы и осо

GardaTech 20 Dec 9, 2022
Travis CI testing a Dockerfile based on Palantir's remix of Apache Cassandra, testing IaC, and testing integration health of Debian

Testing Palantir's remix of Apache Cassandra with Snyk & Travis CI This repository is to show Travis CI testing a Dockerfile based on Palantir's remix

Montana Mendy 1 Dec 20, 2021
Rundeck / Grafana / Prometheus / Rundeck Exporter integration demo

Rundeck / Prometheus / Grafana integration demo via Rundeck Exporter This is a demo environment that shows how to monitor a Rundeck instance using Run

Reiner 4 Oct 14, 2022
A Habitica Integration with Github Workflows.

Habitica-Workflow A Habitica Integration with Github Workflows. How To Use? Fork (and Star) this repository. Set environment variable in Settings -> S

Priate 2 Dec 20, 2021
Cado Response Integration with Amazon GuardDuty using AWS Lambda

Cado Response Integration with Amazon GuardDuty using AWS Lambda This repository contains a simple example where: An alert is triggered by GuardDuty T

Cado Security 4 Mar 2, 2022
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 6.1k Dec 31, 2022
Google Kubernetes Engine (GKE) with a Snyk Kubernetes controller installed/configured for Snyk App

Google Kubernetes Engine (GKE) with a Snyk Kubernetes controller installed/configured for Snyk App This example provisions a Google Kubernetes Engine

Pas Apicella 2 Feb 9, 2022
pyinfra automates infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployment, configuration management and more.

pyinfra automates/provisions/manages/deploys infrastructure super fast at massive scale. It can be used for ad-hoc command execution, service deployme

Nick Barrett 2.1k Dec 29, 2022
More than 130 check plugins for Icinga and other Nagios-compatible monitoring applications. Each plugin is a standalone command line tool (written in Python) that provides a specific type of check.

Python-based Monitoring Check Plugins Collection This Enterprise Class Check Plugin Collection offers a package of more than 130 Python-based, Nagios-

Linuxfabrik 119 Dec 27, 2022
A tool to convert AWS EC2 instances back and forth between On-Demand and Spot billing models.

ec2-spot-converter This tool converts existing AWS EC2 instances back and forth between On-Demand and 'persistent' Spot billing models while preservin

jcjorel 152 Dec 29, 2022