What is the Tlaloc Server Management Toolkit ?

Cloud Management

Tlaloc is a cloud-management application that kind of sits on top of the systems out there. It runs a little bit like Puppet or Chef, but it is built without dependencies on 1 central host.

Furthermore, right now Tlaloc is mostly a server-configuration program, but in the future it is meant to become a job-cluster organization tool where node configuration is just the first part. Along with the Tlaloc server-configuration processes are daemonizing processes that allows a programmer/sysadmin to create group/global work queues to be dispatched to running nodes from a Job Queue Cloud. A special client accesses the Job Queue Cloud for monitoring of activities and load, whereas individual nodes retrieve and/or inject work payloads for processing (or off-loading)

It bootstraps and its indempotent

Nodes are built with a bit of bootstrap code that auto-executes on node startup. This integrates quite easily with Amazon's EC2 user-data, so you can startup nodes manually with your own bootstrap overrides.

Nodes pickup their assigned recipe in the bootstrap procedure and self-install the necessary components.

Tlaloc node self-install and recipe running is completely indempotent, meaning you can re-run the script and it will always auto-detect if something already exists before adding (thus never duplicating users, groups, variables or configs)

Kill the Puppetmaster!

Nodes poll recipes from a central (redundant) location such as Amazon:S3 or your own internal HTTP repository (which can be rsynced redundantly)

This is meant to remove a nagging dependency on one central server for dispatching recipes (and compiling/running them as in the case of Puppet and Chef)

Tlaloc also moves the recipe compiling & running to the nodes

No Central host? How ?

Node agents participate in a Cloud of Task Queues, where Queues are replicated between Nodes instead of residing in a central location

When a Node boots up, it daemonizes a process dedicated to the Task Queue Cloud. It monitors the requests, its own replica of Task Queues (which may be different from the other's), memberships, announcements and is in charge of picking up Tasks and dispatching them to the local node for execution.

No Central host? Why ?

To better mimick real-life and team play. Basically, Tlaloc is modeled after a "gang" rather than a hierarchical structure. Tasks are injected by any process (coming from any Node), therefore, why waste resources on a control host ?

Basically, this allows a Node to ask for help or re-inforcements.

Another why is to avoid wasting additionnal resources when 1 control host doesn't suffice. And avoid the pitfalls of dependance on 1 host that could fail anytime.

Manage big !

Meant to manage a big bunch of clustered servers running short and long tasks, as well as daemon processes

Puppet and Chef don't quite cut it when you start playing with more than a hundred nodes. Tlaloc is an architectural answer to the problems that cluster-computing exacerbates when dealing with Big Data sets. Monitoring, data crunching and storage become complex ,involving more than 1 node at a time, and redundancy takes a sidestep. Tlaloc attempts to address this with intelligent cluster tasks that spawn more than 1 node.

Written in perl and PHP, it supports all kinds of shell-accessible scripts

Why ? Well... I really enjoy solutions like Chef and Puppet, but it pains me to always have to go back to Ruby coding when developping recipes and custom node code. Most system administrators build their own shell scripts, and I'm no exception.

I also investiguated performance on an existing cluster running Ruby code, then compared it to my own PHP code and found out that Ruby was a hog of an environment. Top this with Ruby's failing packages (maintaining working versions is a PAIN!) and its inability to connect to more than 1 DB structure and it meant Ruby was out for me.

Sure, writing daemons and sockets in Ruby is fun, but if I have to write monit scripts on top of them to keep them alive, bye bye! ;) (And don't get me on the JRuby or Enterprise-Ruby, they all fail miserably and leak badly)

I had a big library of code using PHP. And I personally enjoy PHP, for its C-style language. Still, when I felt PHP wasn't up to the task, I simply do it in Perl, C or Shell.

A bunch of Unix scripts are available to help tailor node scripts (apart from the Tlaloc management libraries and APIs)

API:Rest style baby !

The system provides a number of Rest interfaces accessible to developpers

Queue API (allows inserting and retrieving Queued items)

System Object (to see what's running, status of things and basic commands like reboot)

Monitoring API (a somewhat specialised node type for monitoring the workloads)

Currently supported Operating Systems

The plan is to evolve Tlaloc to all unix-based systems.

But currently, it supports OpenBSD, and other BSDs and partially Redhat derivatives (RedHat, CentOS, Ubuntu) (the amount of required work is minimal at this point, User object mostly).

How it works

a manual example of a node setup (getting to know the system manually is priceless)

• a) you "push" the Tlaloc node package using "push_to_node.sh username@server"
• b) get on your node, go under your username folder, and you should find the "Cluster_Runner" folder containing all the Tlaloc files
• c) execute "Install.php", which will install the barebone necessities for your OS and version (rsync and php)
• Your node is now ready to execute its self-installer recipe
• d) execute your favorite recipe (ie; "php Node_Setup.php")
• d.2) you could/should also execute your recipe from a startup script (/etc/rc.local for example), making sure it gets re-executed at each reboot, if you were to run disk-less nodes or following the next recipe for bootstrapping under Amazon.

An automatic installation through Amazon bootstrapping

• a) setup your favorite OS on a node, and follow the manual instructions above to have your basic kit installed
• b) image that Amazon EC node, and assign this image to your new nodes
• c) optionally configure the Amazon bootstrapping data to give your system a hostname identifiable by your recipe, or a role
• d) now when systems come online, they will boot, then execute your setup recipe (the system is fully indempotent, you can run it as many times as you want without affecting your running systems)

Et voilà !

Perhaps my goals are big, but the endeavor for Tlaloc is to become the baremetal management system that allows clouds to grow, without resorting to expensive

Recipe-making

As mentionned earlier, Tlaloc is loosely similar to Puppet and Chef, so the recipes look the same on the surface, and on purpose.

The system includes a number of objects to manage "Package", "File" and "Service"(s)

Custom-tailor your systems based on ereg rules using the hostname, or using bootstrapped variables (defining roles or names, up to you!)

Nothing beats a nice example, so here's one that installs and configures Nginx, php and php-fpm under OpenBSD 5.4+:

if (preg_match("/.*\.intra\..*/",$Node_Name) ) {
	print "We are an Intranet server (could be in the DMZ...)\n";
	// adding an additional route under OpenBSD
	$File->AddLine("/etc/rc.local","route add 192.168.0.0/24 -netmask 255.255.255.0 192.168.10.100");

	// execute the above script to add the extra route to the network routing tables
	$foo=`sh /etc/rc.local`;

	// Copy over the PHP config file
	$File->Source("php-5.3.ini","/etc/php-5.3.ini",null,0755,"root","wheel");

	// installing a number of simple packages
	$Package->Install("php-mysql");
	$Package->Install("php-gd");
	$Package->Install("php-curl");
	$Package->Install("php-mcrypt");
	$Package->Install("php-fpm");

	// Configuring NGinx and starting it
	// The following takes care of setting up your package for automatic start at boot-time
	$Service->Setup("nginx",null,null);
	// copy the nginx config file from the source repository
	$File->Source("nginx.conf","/etc/nginx/nginx.conf",null,0755,"root","wheel","nginx");
	// setup and copy your nginx virtual hosts (assuming they are separate files in an included folder by the main config)
	$File->Makedir("/etc/nginx/vhosts",0755,"root","wheel");

	// finally, setup the vhosts that would ONLY be under this node's
	// source files (repository/nodes/`$Node_Name`/nginx/vhosts)
	$File->Source("/nginx/vhosts/","/etc/nginx/vhosts/",null,0755,"root","wheel");

	// Install a package with some error catching
	if (! $Package->Install("mini_sendmail-chroot")) print "Error installing mini_sendmail package!\n";

	$Service->Setup("php-fpm");

	// kick-start the service daemons without having to reboot the server.
	$Service->Start("php-fpm");
	$Service->Start("nginx");

}