Welcome
we won't be talking about...
containers
scheduling
orchestration
DevOps
no I love talking about those
(let's meet in the hallway)
Some truths:
- I can be old fashioned
- your container scheduler will run on something
- yes, some of us will keep running DCs
- the big clouds are not our future
- actually, forget about the "old fashioned" bit:
- we will never stop tweaking our base tool chain
So let's talk about two of my faves
Running Puppet from mgmt
...on overdrive
CfgMgmtCamp 2020, Ghent
Felix Frank, Systems Architect
The unbelievable Machine Company GmbH
Felix Frank
@felis_rex on Twitter
ffrank everywhere else
...go figure
- recovering infra tooling hacker
- Scrum skeptic
- professional Ansibilist
- hobbyist boxer/kick-boxer
mgmt in a nutshell
- explores another angle of config management
- operates on graphs (DAGs)
- built in golang
- can run without pre-installed agents
- no central server ever
- events, parallelism, node merging
Demo I
A quick look at mgmt basics
$ cat file-demo.mcl
# simple plain file
file "/tmp/demo1-file" {
state => "exists",
content => "basic mgmt operation\n",
}
$ mgmt run --tmp-prefix lang file-demo.mcl
23:55:48.991112 I | cli: lang: lexing/parsing...
...
23:55:50 gapi: generating new graph...
23:55:50 engine: Worker(file[/tmp/demo1-file])
23:55:50 engine: Watch(file[/tmp/demo1-file])
23:55:50 engine: file[/tmp/demo1-file]: CheckApply(true)
...
23:55:50 main: graph: Vertices(1), Edges(0)
23:55:50 main: waiting...
$ cat /tmp/demo1-file
basic mgmt operation
$ echo broken >/tmp/demo1-file
$ cat /tmp/demo1-file
basic mgmt operation
$ echo broken >/tmp/demo1-file ; cat /tmp/demo1-file
basic mgmt operation
$ echo broken >/tmp/demo1-file ; cat /tmp/demo1-file
basic mgmt operation
class apt_repo($name,$url,$key_url) {
exec "get key for repo " + $name {
cmd => "curl -s " + $key_url + " | apt-key add -",
shell => "/bin/bash",
}
Pkg["curl"] -> Exec["get key for repo " + $name]
file "/etc/apt/sources.list.d/" + $name + ".list" {
state => "exists",
content => "deb " + $url + " stable main\n",
}
}
pkg "curl" { state => "installed", }
include apt_repo("elasticsearch",
"https://artifacts.elastic.co/packages/7.x/apt",
"https://artifacts.elastic.co/GPG-KEY-elasticsearch")
There is an overlap
with Puppet
and that's fine
Demo II
Running mgmt from Puppet code
node "dev-app01-ffrank.localdomain" {
file { "/tmp/puppet-managed-file":
ensure => "file",
content => "please don't change too often\n",
}
}
$ puppet master --no-daemonize
...
Notice: Starting Puppet master version 5.5.8
$ puppet agent --server dev-app01-ffrank.localdomain --no-daemonize -v
Notice: Starting Puppet client version 5.5.8
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Caching catalog for dev-app01-ffrank.localdomain
Info: Applying configuration version '1580085107'
Notice: /Stage[main]/Main/Node[dev-app01-ffrank.localdomain]/File[/tmp/puppet-managed-file]/ensure: defined content as '{md5}e4b750c2c778368e279096aac5318e4d'
Notice: Applied catalog in 0.02 seconds
$ cat /tmp/puppet-managed-file
please don't change too often
$ echo broken >/tmp/puppet-managed-file
$ cat /tmp/puppet-managed-file
broken
$ puppet agent --server dev-app01-ffrank.localdomain --test
...
Notice: /Stage[main]/Main/Node[dev-app01-ffrank.localdomain]/File[/tmp/puppet-managed-file]/content:
--- /tmp/puppet-managed-file 2020-01-27 01:34:42.715148625 +0100
+++ /tmp/puppet-file20200127-10652-j5hfiu 2020-01-27 01:36:04.023449520 +0100
@@ -1 +1 @@
-broken
+please don't change too often
Info: Computing checksum on file /tmp/puppet-managed-file
...
Notice: Applied catalog in 0.29 seconds
$ cat /tmp/puppet-managed-file
please don't change too often
$ cat puppet.conf
server=dev-app01-ffrank.localdomain
$ mgmt run --tmp-prefix puppet \
--puppet-conf puppet.conf \
agent
$ echo broken >/tmp/puppet-managed-file ; cat /tmp/puppet-managed-file
please don't change too often
Essentially
{
file { "/tmp/a": }
->
file { "/tmp/b": }
}
resources:
file:
- name: /tmp/a
path: /tmp/a
- name: /tmp/b
path: /tmp/b
edges:
- name: File[/tmp/a] -> File[/tmp/b]
from:
kind: file
name: /tmp/a
to:
kind: file
name: /tmp/b
Demo III
Syncing unsupported Puppet resources
package { "haproxy":
ensure => "installed",
}
->
file { "/etc/haproxy/haproxy.conf":
mode => "0640",
}
->
service { "haproxy":
enable => true,
}
ec2_instance { 'name-of-instance':
ensure => running,
region => 'us-east-1',
availability_zone => 'us-east-1a',
image_id => 'ami-123456',
instance_type => 't2.micro',
key_name => 'name-of-existing-key',
subnet => 'name-of-subnet',
security_groups => ['name-of-security-group'],
tags => {
tag_name => 'value',
},
}
gcompute_instance { 'instance-test':
ensure => present,
machine_type => 'n1-standard-1',
disks => [
{ auto_delete => true, boot => true,
source => 'instance-test-os-1'
}
],
metadata => $metadata_test,
network_interfaces => $interfaces_test,
zone => 'us-central1-a',
project => $project,
credential => 'mycred',
}
cron { "check in":
command => "curl https://my.apisrv.org/i/heartbeat#$hostname",
minute => 10,
}
We currently don't translate this, because mgmt's cron support is systemd based and thus has different semantics
$ mgmt run --tmp-prefix puppet demo3/cron-example.pp
...
00:49:21 gapi: launching translator
00:49:28 gapi: (output) Error: Cron[check in] cannot be translated natively, falling back to 'exec puppet resource'
...
00:49:30 engine: exec[Cron:check in]: resource: Apply
00:49:32 engine: exec[Cron:check in]: resource: command output is:
00:49:32 engine: exec[Cron:check in]: resource: Notice: /Cron[check in]/ensure: created
--- !ruby/object:Puppet::Resource
type: Cron
title: check in
...
parameters:
ensure: present
command: curl https://my.apisrv.org/i/heartbeat#dev-app01-ffrank
...
We currently don't translate this, because mgmt's cron support is systemd based and thus has different semantics
[0,5,10,15,20,25,30,35,40,45,50,55].each |$minute| {
cron { "check in ($minute)":
command => "curl https://my.apisrv.org/i/heartbeat#$hostname",
minute => $minute,
}
}
$ time mgmt run --tmp-prefix puppet demo3/cron-example-multi.pp
...
real 1m8.787s
user 0m41.484s
sys 0m18.324s
Demo IV
Ludicrous speed
the latest mgmt and
the latest translator module
$ puppet module upgrade ffrank-mgmtgraph
...
$ time mgmt run --tmp-prefix puppet demo3/cron-example-multi.pp
...
real 0m20.697s
user 0m5.404s
sys 0m3.572s
Faster Puppet(?)
REMOTE CONTROL When you start RRDtool with the command line option '-' it waits for input via standard input ( STDIN ). With this feature you can improve performance by attaching RRDtool to another process ( MRTG is one example) through a set of pipes.
Speaking of pipes
Demo V
Nuts and bolts
$ puppet module upgrade ffrank-mgmtgraph --version 0.5.0
$ puppet mgmtgraph print --code 'cron { "demo":
> command => "echo demo", minute => 1 }'
...
exec:
- name: Cron:demo [36/1815]
cmd: |-
puppet yamlresource cron 'demo' '{"provider": "crontab", "command": "echo demo", "minute": ["1"], "ensure": "present",
"target": "root", "loglevel": "notice"}'
timeout: 30
shell: "/bin/bash"
ifshell: "/bin/bash"
ifcmd: |-
puppet yamlresource cron 'demo' '{"provider": "crontab", "command": "echo demo", "minute": ["1"], "ensure": "present",
"target": "root", "loglevel": "notice"}' --noop --color=false | grep -q ^Notice:
state: present
pollint: 1800
$ puppet module upgrade ffrank-mgmtgraph
$ puppet mgmtgraph print --code 'cron { "demo":
> command => "echo demo", minute => 1 }'
# simple resource:
$ puppet apply -e 'file { "/tmp/myfile": ensure => present, mode => "0644" }'
# faster to type:
$ puppet resource file /tmp/myfile ensure=present
# but what about
$ puppet apply -e 'cron { "clidemo": command => "echo hi", minute => [ 10, 40 ] }'
# these will not work
puppet resource cron clidemo command="echo hi" minute=10,40
puppet resource cron clidemo command="echo hi" minute='[10,40]'
# how to send this to `puppet resource`
$ puppet apply -e 'cron { "clidemo": command => "echo hi", minute => [ 10, 40 ] }'
# using ffrank-yamlresource
$ puppet yamlresource cron clidemo '{ command: "echo hi", minute: [ 10, 40 ] }'
$ puppet yamlresource receive
Notice: ready to receive resources in YAML representation (one per line)...
cron clidemo { minute: [ 10, 30, 50 ] }
{"resource":"Cron[clidemo]","failed":false,"changed":true,"noop":false,"error":false,"exception":null}
cron clidemo { minute: [ 10, 30, 50 ] }
{"resource":"Cron[clidemo]","failed":false,"changed":false,"noop":false,"error":false,"exception":null}
# also accepts JSON, useful when used as an API
{"type":"cron", "title": "clidemo", "params": "{ minute: [ 10, 40 ] }"}
{"resource":"Cron[clidemo]","failed":false,"changed":true,"noop":false,"error":false,"exception":null}
$ for i in `seq 1 100`
do
echo file /tmp/loop$i '{ ensure: file }'
done \
| puppet yamlresource receive
# vastly faster than
for i in `seq 1 100` ; do puppet resource file /tmp/loop$i ensure=file ; done
We aim to soon be ready to drop in mgmt
as a Puppet agent replacement/alternative
for testing and learning
We are "virtually ready" according to James
Come talk to us in #mgmtconfig on Freenode
For Science!
While you're still here - we're hiring
Thank you for
your kind attention
Questions?