Hacking Types and Providers

PuppetConf 2015, Portland OR

Who is that cat?

Felix Frank

  • Server Op from Berlin
  • Puppet user/contributor since 0.24
  • code contributor since ~3.4
  • Twitter @felis_rex
  • GitHub @ffrank

Caveat: Mostly self-taught

Focus will be on debugging
rather than designing

Agenda

  1. The Type class and DSL
  2. Providers
  3. Where to start
  4. Live demo

Types

Remember

  • lib/puppet/type(.rb)
  • lib/puppet/provider(.rb)

Also

  • lib/puppet/property(.rb)
  • lib/puppet/parameter(.rb)

Don't be mislead

all the Resource classes hardly relate

  • lib/puppet/parser/resource.rb
  • lib/puppet/parser/ast/resource.rb
  • lib/puppet/face/resource.rb
  • lib/puppet/resource.rb
  • lib/puppet/application/resource.rb

When reading type code

Keep some Ruby idiosyncracies in mind

Java

Class

Instance

https://en.wikipedia.org/wiki/Otter#/media/File:Fischotter,_Lutra_Lutra.JPG

https://en.wikipedia.org/wiki/Rooster#/media/File:El_Gallo_Ukraine.jpg

Ruby

Class

Instance

http://www.creationscience.com/onlinebook/LifeSciences13.html

https://en.wikipedia.org/wiki/Otter#/media/File:Fischotter,_Lutra_Lutra.JPG

Classes are instances


Class String ... end
						

same as


String = Class.new do ... end
						

Class Class : Module

An "overview"

                         +---------+             +-...
                         |         |             |
         BasicObject-----|-->(BasicObject)-------|-...
             ^           |         ^             |
             |           |         |             |
          Object---------|----->(Object)---------|-...
             ^           |         ^             |
             |           |         |             |
             +-------+   |         +--------+    |
             |       |   |         |        |    |
             |    Module-|---------|--->(Module)-|-...
             |       ^   |         |        ^    |
             |     Class-|---------|---->(Class)-|-...
             |       ^   |         |        ^    |
             |       +---+         |        +----+
obj--->OtherClass---------->(OtherClass)-----------...

http://ruby-doc.org/core-2.2.0/Class.html

Everything is very confusing

http://theawesomedaily.com/21-things-that-look-exactly-like-donald-trump/

What's important: Both classes and instances have state and behavior

Puppet types take advantage of that

Defining native types
creates subclasses

This code...


Puppet::Type.newtype(:cron) do
  # code!
end
						

...effectively gives you:


class Puppet::Type::Cron : Puppet::Type do
  # generated code!
end
						

Let's get meta

manifest resources
catalog resources
Resource Abstraction Layer
system entities

I: Manifest Resources


file { '/etc/motd':
    mode => 644
}
cron { 'break-all-the-things':
    command => '/opt/scripts/cleanup.rb'
}
						

II: Catalog Resources


{
  "type": "File",
  "title": "/etc/motd",
  "tags": ["file","class"],
  "file": "/tmp/example-manifest.pp",
  "line": 1,
  "exported": false,
  "parameters": {
    "mode": 644
  }
}

						

III: RAL resources

(resource abstraction layer)

  • complete representation of a system entity
  • properties and parameters
  • does all the interesting work

IV: system entities


-rw-r--r-- 1 root root 0 Feb  5  2011 /etc/motd
						

# HEADER: ...

# Puppet Name: break-all-the-things
1 1 2 * * /usr/scripts/cleanup.rb
						

http://www.theawl.com/2010/02/church-boring

Now back to the code

So what about this


class Puppet::Type::Cron : Puppet::Type do
  # generated code!
end
						

versus this

When reading the Puppet::Type code,
be mindful of class vs. instance scope

Ruby - know thyself

  • class method:
    
    def self.allattrs
    						
  • instance method:
    
    def retrieve
    						

Caution!


class Puppet::Parameter
  class <<self
    # ...
    def validate
      # ...
    end
  end
end
						

Puppet::Parameter::validate is a class method

Reading type code

DSL mainly consists of calls to
class methods of Puppet::Type


newproperty(:user) do
						

newparam(:name) do
						

It is generally safe to read
Type class methods to learn the DSL

Things like self.newparam or self.ensurable

Some are lifecycle methods though

self.instances is pretty neat

This talk is going great

http://devopsreactions.tumblr.com/post/129272206864/how-i-picture-the-thoughts-of-someone-saying-that

Interlude: properties and parameters

Mnemonic: Properties can
and will sync

Parameters can never do that

E.g. File content

Classic property: Is the content on disk?

E.g. install_options

Classic parameter: Unsyncable, since flags cannot be reset after the fact

Providers

Types vs. providers

Which does what?

For example: catalog


{
  "type": "Package",
  "title": "cowsay",
  "parameters": {
    "ensure": "installed"
  }
}
						

Should value

Package[cowsay]/ensure: present

Puppet::Provider::Package.prefetch

takes a hash argument

{ 'cowsay' => Package[cowsay] }

...where Package[cowsay] is an instance of Puppet::Type::Package

prefetch in turn calls

Puppet::Provider::Package::Dpkg.instances

assuming Dpkg or Apt is the selected provider

the instances hook produces
a list of Provider instances

Entities that cannot be enumerated or just not found get a "fresh" provider


Type::Package.provider(:apt).new
						

or in other words


Provider::Package::Apt.new
						

Finally properties can be checked
and synced one by one

Providers are
straight forward

...except when they inherit ParsedFile

lib/puppet/provider/parsedfile.rb
  • cron
  • mount
  • host
  • sshkey
  • ...

ParsedFile peculiarities

There is a specific DSL

See Util::FileParsing


text_line   :comment, :match => /^\s*#/
text_line   :blank,   :match => /^\s*$/
record_line :parsed,
  :fields   => %w{options type key name},
  :optional => %w{options},
						

ParsedFile peculiarities

Notion of records to represent resources


def self.prefetch_all_targets(resources)
  records = []
  targets(resources).each do |target|
    records += prefetch_target(target)
  end
  records
end
						

Targets are the managed files

ParsedFile peculiarities

Instance methods will often rely on class methods


def flush
  # ...
  self.class.flush(@property_hash)
end
						

ParsedFile peculiarities

Action is defered to flush

(true for some other providers as well)


def self.flush
  @modified.each do |target|
    flush_target(target)
  end
end
						

ParsedFile peculiarities

Flushing implies gathering
global state (records)


def self.flush_target(target)
  backup(target)
  records = target_records(target)
  target_object(target).write(to_file(records))
end
						

When debugging
parsedfile providers,
keep the convoluted
flow of logic in mind

Aside: Please avoid creating new parsedfile providers

Look at augeas providers instead

Summarizing

With types, keep class vs. instance level in mind

Provider instances get paired with
RAL resources, i.e. type instances

Parsedfile providers are
a can of worms

Learning their structure is rewarding but will not help you with
other providers at all

Where to start

How do I get started with resource type and provider development?

Read code, perhaps try and
fix a bug or three

Most of you will want
to enhance modules

Working on core Puppet
has some advantages

Test coverage is good

Weekly triage sessions

Bugs are plentiful

Almost there

Now some final words, then Q&A and demo!

The white dude ratio
is ours to choose

Recommend my book?

USA

Please don't make Mr. Trump
your president

Q&A