Steampipe dashboards and benchmarks on your information

In Steampipe unbundled we confirmed how its plugins, which initially labored solely with the overseas information wrapper loaded into Steampipe’s batteries-included Postgres, are actually additionally accessible as stand-alone distributions you can load into your individual cases of Postgres or SQLite. Now Steampipe itself is unbundled: its dashboard server and benchmark runner have migrated to a brand new open-source mission, Powerpipe.

While you begin Powerpipe it defaults to an area occasion of Steampipe so current dashboards and benchmarks work as all the time. The mods that drive them supply their information from the identical plugins, utilizing the identical SQL queries. However you need to use Powerpipe’s --database argument to specify a Postgres, SQLite, DuckDB, or MySQL connection string.

Why unbundle Steampipe on this manner? Unix philosophy holds {that a} software program instrument ought to do one factor effectively, and mix readily with different instruments that abide by the identical precept. Steampipe started that manner, however because the dashboard and benchmark layers appeared it turned clear that refactoring was so as. For devops engineers who use the instrument to visualise and assess cloud infrastructure, it is smart to decouple these layers into elements with well-defined interfaces. However Powerpipe is not only for them. Among the many set of latest mods launched with Powerpipe, there’s one which visualizes biomedical information from RNACentral by means of a public Postgres endpoint. To view these dashboards you put in the mod, then begin Powerpipe like so:

powerpipe server --database postgres://reader:NWDMCE5xdipIjRrp@hh-pgsql-public.ebi.ac.uk:5432/pfmegrnargs

The dashboards are actually browseable at http://localhost:9033.

You may in fact join any Postgres consumer to that endpoint. What Powerpipe brings to the social gathering is an as-code strategy to visualizing and validating information. The widgets that drive dashboards and benchmarks are written in a declarative language, HCL, that enhances the declarative SQL queries that fill these widgets with information. All of your HCL and SQL code lives in repos, beneath model management, open to the identical type of collaboration that you just count on and revel in for all different code artifacts.

Working with that code is a developer-friendly expertise in one other manner too. Powerpipe watches your recordsdata and reacts immediately whenever you make adjustments to both the HCL that configures dashboard widgets or the SQL that populates them.

IDG

SQL-powered relationship graphs

The suite of dashboard elements contains every thing you’d count on, together with tables, charts, infocards, and enter widgets. It additionally contains some belongings you may not count on, notably the nodes and edges that kind relationship graphs. Right here’s an instance that makes use of a choose widget to decide on an InfoWorld writer, then builds a graph that relates the chosen writer to articles written and to the (maybe a number of) classes related to every article.

IDG

Right here’s how the nodes and edges are outlined. Two of the nodes question for classes and authors, and the third makes use of the RSS plugin to question the chosen writer’s RSS feed. Then two edges join the nodes. One relates article hyperlinks to authors, the opposite relates the identical hyperlinks to classes. It’s all simply SQL, leveraged in an unconventional manner.

-- enumerate classes
node {
  class = class.class
  sql = <<EOQ
    choose
      class as id,
      class as title
    from
      infoworld_categories()
  EOQ
}

-- enumerate authors
node {
  class = class.writer
  args = [self.input.authors.value]
  sql = <<EOQ
    choose
      writer as id,
      writer as title
    from
      infoworld_authors()
    the place 
      writer = $1
  EOQ
}

-- checklist articles by writer
node  writer 

-- relate articles to authors
edge 

-- relate articles to classes
edge {
  sql = <<EOQ
    choose
      hyperlink as to_id,
      class as from_id
    from
      infoworld_category_urls()
  EOQ
}

Extra typical makes use of of those relationship graphs assist devops engineers perceive how the weather of their cloud infrastructures match collectively, as for instance on this graph which exhibits how the elements of a Kubernetes deployment—deployments, replicasets, pods, containers, servers, and nodes—relate to 1 one other. Every entity within the graph is hyperlinked to a different graph that drills into the entity and particulars its surrounding infrastructure.

IDG

These visualizations, that are additionally accessible for AWS, Azure, and GCP, are a superb method to discover and perceive your cloud architectures. And the queries you write to try this are reusable. You may circulation the identical information into dashboard charts and tables.

SQL-powered benchmarks and controls

Powerpipe can also be the engine that powers suites of compliance benchmarks, additionally for AWS, Azure, GCP, Kubernetes, and others. Right here’s a benchmark that validates the picture URLs embedded within the extensions subject of InfoWorld RSS feeds.

IDG

And here is the way it’s outlined. The benchmark contains one management that runs a multistep question to pick out feed URLs for authors, drill into every merchandise’s media tag, then examine the HTTP response code for every URL.

management "image_urls" 

benchmark "feeds" {
  title = "Verify InfoWorld feeds"
  kids = [
    control.image_urls
  ]
}

Notable factors right here:

• The feed_link column returned from the primary CTE (frequent desk expression) joins with its counterpart within the RSS plugin to fetch the feed for every writer.
• As a result of the again finish is Steampipe, which is Postgres-based, the Postgres JSONB operators can be found to drill into the media tag and extract the URL.
• Every url then joins with the corresponding column of the Web plugin—an HTTP consumer wrapped as a database desk!—to examine the response code.

A management is only a SQL question that returns the required columns standing, cause, and useful resource. You may consider a management as a unit check for information, with Powerpipe because the testrunner.

These controls sometimes help normal compliance suites: CIS, FedRamp, GDPR, HIPAA, NIST, PCI, SOC 2, and extra. Mods like AWS Compliance supply broad and deep help for these, constructing on the equally broad and deep API protection offered by Steampipe plugins just like the one for AWS.

However you possibly can construct benchmark mods to validate any type of information, wherever it lives: in ephemeral tables populated by cloud companies by way of plugins, or in your individual databases as native tables.

Dashboards and benchmarks as code

Whether or not you’re visualizing information with interactive tables, charts, and graphs, or validating information utilizing controls, the mannequin is identical. You employ SQL queries to accumulate the info, and HCL widgets to show it, with reside enhancing in each instances. The code lives in packages known as mods you can set up, create, and remix.

Just like the Steampipe product from which it was decoupled, Powerpipe is a single binary you can run regionally, or in a cloud VM, or in a CI/CD pipeline. And like Steampipe, it’s accessible in hosted kind at Turbot Pipes the place you possibly can collaborate together with your staff and share snapshots of dashboards and benchmarks.

The present suites of Powerpipe dashboards and benchmarks concentrate on what devops folks want most: a regular question language, with reside entry to cloud APIs, embedded in HCL wrappers that reside in repositories together with the remainder of your managed code. That’s the candy spot, however with the unbundling of Steampipe now you can use the identical applied sciences extra broadly.