Belchak.com

Technology and Awesome

Making Sensu Work With Graphite

| Comments

Like many companies, I’m sure, we’re investigating using Sensu for monitoring and Graphite for metrics collecting. The docs for making these two commuicate together over AMQP are pretty good except for one thing.

Using the docs that I’ve read, when you start up Sensu first with its default options for AMQP, and then start up Graphite to read from that queue, Graphite will crash every time it tries to create the exchange. The reason for this, after lots of hair-pulling and debugging, is because Graphite is hard-coded to set up a durable exchange for its metrics, while Sensu’s default is to set up a non-durable exchange. RabbitMQ will not allow you to create an exchange with the same name and different options, so whichever piece of software created the exchange will work, but the other will blow up all over your face.

The solution to this is pretty simple, but I haven’t seen it documented anywhere. Simply add "durable": true to your exchange configuration in your Sensu config.json like so:

1
2
3
4
5
6
7
8
9
10
11
12
"handlers": {
    "graphite": {
      "exchange": {
        "name": "graphite",
        "type": "topic",
        "key": "metrics",
        "durable": true
      },
      "type": "amqp",
      "send_only_check_output": true
    },
}

Hope that helps!

Migrating From WordPress to Octopress

| Comments

WordPress is awesome, why’d you switch?

So, I’ve been really hating WordPress as a blog engine lately. Tons of spam, lots of emails from spam accounts signing up and trying to post comments, etc. We had some discussion at the office recently about what blog engine to use for our corporate blog, (we ended up using WordPress) but through the process I decided to evaluate Octopress as an approach to static content serving for my personal blog.

On top of the security issues that I’ve been dealing with (constantly updating the software, plugins and deleting spam), I host at Dreamhost. They are not known as the fastest hosting company around, and I was finding that my pages were loading so slowly that it was beginning to get frustrating.

Plus, I could never find a good WordPress theme that I wanted to look at and I never have time to spend customizing a complex WordPress theme.

Enter Octopress

I evaluated Octopress for about 10 minutes and knew I was hooked. I loved the simplicity and the beauty of its default theme (and since it’s themeable using SASS I can easily customize it) and the ease of deployment to any number of hosting platforms (rsync, github pages, heroku, etc) leaves no excuse not to use it! Also, since all the content is generated dynamically on your machine and then uploaded as static content, it is very fast to serve on the server, and is less prone to spam and security holes.

How I did it

I simply exported all of my posts and pages using the Export functionality in the WordPress administration tool and then ran it through exitwp. The result was phenomenal! I ended up with an export of all my pages and posts. Since I was already using Disqus all the comments came over automatically.

I did have to do some very basic formatting for some of my code snippets in order to get the syntax highlighting to work properly, but that was a simple matter of just setting the language in the generated markdown files.

Deploying

I use the Rsync deploy method to my Dreamhost shared server, but I am also considering migrating to github pages at some point. Deploying is as simple as:

1
2
$ rake generate
$ rake deploy

And that’s it! You can even streamline it more by only running one command:

1
$ rake gen_deploy

Overall, the migration was even more simple than I had expected, and I’m very happy with the results! Now the hard part is coming up with interesting content to put here!

JIRA and Campfire Integration

| Comments

I have been working on migrating my company’s Pivotal Tracker to JIRA. We love most of the features of JIRA, but we were really missing the nice updates to our Campfire room to tell us what is going on with our issues.

Sure, we could use an RSS feed plugin for Campy, but that is not real-time enough.

I didn’t want to write a huge plugin just to send messages to Campfire (it seems like such a simple thing to do!). I was poking around for solutions, and came across this plugin. It’s a plugin that runs scripts written in the Groovy dynamic language for Java. One nice thing about this plugin is that it lets you execute Python (Jython) or Ruby (Jruby). The only problem is that you can’t write a listener using Python, so I ended up having to learn Groovy.

Here’s what I came up with. It requires you to install the HTTPBuilder libraries for Groovy, and this script will require some customization to fit your environment. Right now it only supports sending messages when new issues are created and when a new comment is added to an existing issue, but other issue events would be easy enough to add - this is meant as an example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import com.atlassian.jira.event.issue.AbstractIssueEventListener
import com.atlassian.jira.event.issue.IssueEvent
import com.atlassian.jira.ComponentManager
import org.apache.log4j.Category
import groovyx.net.http.RESTClient
import static groovyx.net.http.ContentType.JSON
import static com.atlassian.jira.event.type.EventType.*

class CampfireListener extends AbstractIssueEventListener {

    @Override
    void workflowEvent(IssueEvent event) {
        def Category log = Category.getInstance("com.onresolve.jira.groovy.PostFunction")
        def campfire = new RESTClient('https://CAMPFIREID.campfirenow.com/room/ROOMID/')
        def issueBaseUrl = "http://yourjirahost:8080/path-to-jira/browse/"
        campfire.auth.basic 'CAMPFIRE AUTH TOKEN', 'X'
        switch (event.getEventTypeId()) {
            case ISSUE_COMMENTED_ID:
                def resp = campfire.post( path: 'speak.json',
                                      body: [ message: [ type: "TextMessage", body:
                                          String.format("%s added a comment to %s (%s%s):",
                                              event.getUser().getDisplayName(),
                                              event.issue.getKey(),
                                              issueBaseUrl,
                                              event.issue.getKey())] ],
                                      requestContentType: JSON)
                resp = campfire.post( path: 'speak.json',
                                      body: [ message: [ type: "PasteMessage", body:
                                          String.format("%s", event.getComment().getBody()) ] ],
                                      requestContentType: JSON)
                break
            case ISSUE_CREATED_ID:
                def resp = campfire.post( path: 'speak.json',
                        body: [ message: [ type: "TextMessage", body:
                            String.format('%s created a new issue: "%s" (%s%s):',
                                event.getUser().getDisplayName(),
                                event.issue.getSummary(),
                                issueBaseUrl,
                                event.issue.getKey()) ] ],
                        requestContentType: JSON)
                resp = campfire.post( path: 'speak.json',
                      body: [ message: [ type: "PasteMessage", body:
                          String.format("%s", event.getIssue().getDescription()) ] ],
                      requestContentType: JSON)
                break
        }
    }
}

Problems With IPv4/6 Hosts in OSX Lion

| Comments

If you’re like me, you end up doing a lot of testing in virtual machines. I run several VMs that do various different tasks for me so that I don’t have to sully my Mac with packages I rarely need, etc. So what I do is create a VM and give it a local hostname defined in my /etc/hosts file.

Recently, after my upgrade to OSX Lion, I started running into an issue where it would take over 5 seconds to connect to any of my VMs using any protocol (SSH, HTTP, etc). After several hours of troubleshooting, and several Wireshark sessions later, the truth of the matter was plain:

OSX is now doing IPv6 by default with no way that I have found to disable it yet, and so that means that if you have hosts that you want to reference with locally defined hostnames, you are going to have to create an IPv6 entry for them as well.

Here is a great site that I found that helps with the conversion: SubnetOnline.com

I am sure that there are better ways to do this, but I just wanted to share this quick little hack that I found with you. If you have a better way, please let me know!

Re-enabling Key Repeat in OSX Lion

| Comments

I have recently upgraded to OSX Lion, and I have to say that I love everything about it. Except for one thing. In many apps, the key repeat has been disabled in favor of the new press-and-hold popup for getting alternative characters.

This is fine for most apps, but for apps like PyCharm where I use vi key maps, it becomes very, very frustrating.

I came across this little tip to re-enable the key repeat, and my life is measurably better (first world problems, I know…).

Run the following in your Terminal.app:

1
$ defaults write -g ApplePressAndHoldEnabled -bool false

Then reboot, and you should be good to go!

Campy: The Python Campfire Bot

| Comments

We use a Campfire at Needle for our day-to-day communication between our tech team members and other business members. We really like Campfire’s ability to store files, conference call and keep transcripts even when you are not logged in. We get GitHub commit messages, Pivotal Tracker story updates and more scattered in our conversations.

These one-way updates are very nice, but I thought it would be nice to be able to have a bot that sat in the channel and could take commands from anybody and do more complex tasks for us without us ever having to leave Campfire. To solve this problem, I wrote Campy, a pure Python Campfire bot with an extensible plugin system.

The first plugin I wrote as a proof of concept is a Google image search plugin. You tell the bot some search parameters that you would like it to search for, and you get back an image inline in your chat transcript. Don’t worry, safe search is on by default so you don’t get any NSFW images back.

The second plugin I wrote was the all-important Pivotal Tracker plugin. This plugin lets you do the following:

  • pt story create “Title” “Description” bug|feature|chore|release – Create a story

  • pt getmine started|finished|delivered|accepted|rejected|unstarted|unscheduled – Get a list of all stories that belong to you

  • pt start #story_id – Start a particular story.

  • pt start|tell next bug|feature|chore [mine] – Start or tell the next story of a given type. Optionally supply ‘mine’ at the end of the message to only work on your own stories.

The plugins system is very extensible, so you can create a plugin to do pretty much anything!

You can get Campy here: https://github.com/bbelchak/campy

Ergotron WorkFit-S Standing Desk Review

| Comments

A few months ago, I started investigating the health benefits of standing while I work at my computer. The benefits are myriad, and include:

  • More calories burned. Standing at a desk while you work can actually burn around 300 calories per day more than sitting.

  • Standing at a desk helps people with back problems. Humans were not made for sitting, and doing so for long periods of time can be really hard on your hips and lower back.

  • Improved posture. Standing can help improve your posture, which can relieve pain in your lower and upper back, and neck.

  • More breaks. I find that I take more breaks, which makes me happier and healthier. While standing, it is easier to just walk away instead of hauling yourself out of a chair.

After my research, I decided to convert my cheap IKEA desk into a standing desk by raising the legs up as far as they could go and then stacking books and Rubbermaid bins to get the height on the keyboard and monitors right. Here’s what I ended up with:

As you can see, it is less than ideal. It forced me to stand 100% of the time, and in order to take a break from standing, I needed to disconnect my laptop from everything and take it to a chair. Not awesome.

I reverted back to a sitting desk, and kept wishing for a standing option that would allow me to quickly change to a sitting position if I got tired, without having to deal with completely rearranging my work area.

Then, I discovered the Ergotron WorkFit-S while searching around for standing desk options. I loved the youtube video I found here:

I bought the WorkFit-S Combo (laptop + LCD) on Amazon with overnight shipping with my Amazon Prime account, and it arrived very quickly. I was a bit worried because the tech specs indicated that the maximum monitor size is 20” while using a laptop. I spoke with one of their marketing people on Twitter, and they were very helpful, checking with their support staff and product managers to provide me with any tips that they could dig up while I anxiously awaited the FedEx delivery guy.

When I received it, I immediately installed it on my desk. It took approximately 15 minutes to get it built and installed on my desk. My 24” LCD fit perfectly on the right, with my laptop on the left. I think the concern with the size of the monitor is not necessarily the size, but the weight. I could probably fit a 27” on here without a problem if it was light enough. My LCD and laptop are probably right on the edge of the maximum weight. Since the up and down motion is achieved with a pulley system, I had to adjust the tension a bit to get the balance right.

I did have to move it after I’d installed it in the center of my corner desk because the mouse tray kept hitting the desk where it curves.

The thing will attach to virtually any surface, but they also make a wheeled cart that has similar features.

I am very happy with it so far, and I look forward to reaping the benefits of standing, while having the option to sit!

The Ergotron WorkFit retails for around $399 and you can find them on Amazon.com or elsewhere.

Unit Testing Django With a NoSQL Backend

| Comments

In my previous post about unit testing for django, I laid the groundwork for how to unit test any django application. One nice feature that django includes with its test framework is the test database syncing. Even better is if you are using South to do database migrations - it will run the migrations in  your test environment for you.

However, what if you are using a NoSQL database backend like MongoDB, Cassandra, CouchDB or something similar and you aren’t using the Django ORM? How do you handle setting up and tearing down the database environments?

The good news is that Python’s unittest framework makes this easy. You can override the setUp() and tearDown() on each TestCase that you build. Here is a snippet to get you started:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import pymongo
from django.test import TestCase

# It would be best to define this in a utility class somewhere
def get_db(db_name=None):
    """ GetDB - simple function to wrap getting a database
    connection from the connection pool.
    """
    return pymongo.Connection(
            host=settings.MONGO_HOST,
            port=settings.MONGO_PORT)[getattr(
                settings, "MONGO_DBNAME_PREFIX", "") +
                (db_name or settings.MONGO_DBNAME)]

class MyTestCase(TestCase):
    fixtures = ['test_data.json']
    def setUp(self):
        self.db = get_db('test')
        self.db.create_collection('mytestcollection')

    def test_doc_published(self):
        # Set up a document to save
        doc = dict(text="test",
                      user_id=1)
        self.db.mytestcollection.save(doc)
        self.assertEqual(self.db.mytestcollection.find_one(
            {'user_id':1})['text'], 'test')

    def tearDown(self):
        self.db.drop_collection('mytestcollection')

What this does is setUp() a collection in the mytestcollection collection, runs the my_doc_published test and then tears down the test database environment by dropping the mytestcollection collection.

Things to remember for setUp() and tearDown():

  • setUp() is called before every test method in your TestCase class.

  • tearDown() is called after every test method in your TestCase class.

  • tearDown() is called even if your test methods fail or error out.

And there you have it! Django makes testing even non-ORM datasources a snap, if you know how to wire it up. UPDATE: Some would say that database fixtures and setting up/tearing down database environments as part of your unit tests is not “unit testing”. This is not entirely accurate, because in order to do unit tests that rely on backend data, you must instantiate and tear down pristine database environments.

Unit Testing Your Django Application

| Comments

Unit testing is a very important part of any software project. It helps you know that the new code you are deploying works, and isn’t going to blow up in your face. It also helps you feel good about changing large chunks of code without destroying everything you’ve done for the last 3 years.

Unit testing with django is as simple as pie. The documentation is very good, and you can learn a lot about more advanced testing methods from the python documentation. In this blog post, I aim to show a quick way to get up and running with testing your django application.

First, if you are just starting out, make sure you put a high emphasis on testing your application, otherwise you are going to end up with a bunch of code that has never been tested and you will find yourself writing code for weeks just to get partial coverage on the code you’ve already written. Starting off on the right foot is a much better approach, and you will find life much more enjoyable.

Let’s get started…

First of all, you need to define your models:

1
2
3
4
5
6
7
8
9
10
11
12
from django.db import models
from django.contrib.auth.models import User

class Post(models.Model):
    title = models.CharField(max_length=50)
    body = models.TextField()
    author = models.ForeignKey(User, related_name="news_post")
    date = models.DateTimeField()
    users_read = models.ManyToManyField(User, related_name="users", blank=True)

    def __str__(self):
        return self.title

What we have done here is created a news post item. Let’s test it!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
class PostTestCase(TestCase):
    fixtures = ['test_data.json']
    def setUp(self):
        self.user = User.objects.create_user('newsposter',
                                             'newsposter@news.com', 'newspass')
        self.post = Post.objects.create(title="Test Post #1",
                body="Test Post #1 Body",
                author=self.user,
                date=datetime.datetime.now())
        self.c = Client()


    def test_post_creation(self):
        """
        Tests that we can create a Post
        """
        self.assertEqual(self.post.title, "Test Post #1")
        self.assertEqual(self.post.author, self.user)

    def test_user_can_read(self):
        """
        Tests that a user is allowed to read.
        """
        self.c.login(username='newsposter', password='newspass')
        response = self.c.get('/news/get_post/1/')
        self.assertEqual(response.status_code, 200)
        self.assertNotEqual(response.content, '{}')

One of the really cool thing about testing with django is that it comes with a testing client that allows you to make requests just like a real user would. As you can see in our test_user_can_read() method, we have used the client to make a GET request against a URL. You can make a POST request just as easily:

1
2
3
4
5
6
7
8
def test_i_read_this(self):
    """
    Tests a new user marking the story as read.
    """
    self.c.login(username='newsposter', password='newspass')
    response = self.c.post('/news/read/1/', {'add':True})
    self.assertEqual(response.status_code, 200)
    self.assertEquals(response.content, '{\n    "read": true\n}')

In the previous code sample, the client sends a POST request to /news/read/1/ with the {'add':True} data. This gets converted to form data and submitted via the POST. The request returns back JSON, which we match up against what we expect it to return.

Here are some things to remember when you are writing your test cases:

  • setUp() gets called before every test method in your TestCase.

  • tearDown() gets called after every test method in your TestCase.

  • Test methods must start with “test” otherwise they will not be executed. It is safe to have other methods in your TestCase that do not begin with “test” if you want to abstract functionality for multiple test methods into a single function.

  • Django creates a test database for you, populates it, runs any south migrations (if you are using south), and then destroys it.

  • Do not expect that data that is available in one of your test methods will be available in another. Each test method starts with a blank data slate. If you need data instantiated before your tests are run, consider using the setUp() and tearDown() methods, or using fixtures. You can specify fixtures other than your initial_data fixtures by adding fixtures = ['test_data.json'] to your TestCase class.

Here is in-depth documentation about what assert methods are available to you in different versions of Python in the official Python unittest documentation.

As you can see, testing with django is really really simple, but very powerful. In my next post, I will discuss how to test django with a MongoDB backend that does not use the ORM.

Code Completion (IntelliSense) in VIM

| Comments

VIM has been my editor of choice for at least 15 years. I love how fast I can edit files, perform menial tasks, and wreak general havoc on any code project I am working on at any given moment. One of the things that I have missed about VIM from an IDE perspective has been code completion (a.k.a. “IntelliSense”). I have spent a lot of time on websites and man pages trying to figure out syntax and function names for several types of languages, and just recently discovered a long-included feature of VIM called omni completion, or Omnicomplete.

Since my life is mostly centered around django these days, I will discuss how I’ve benefited from omnicomplete and how I’ve set it up in my own environment.

First, since django is a web development framework, I want to make sure that I can get omnicompletion for HTML, Python, JavaScript and CSS. Omnicompletion works for almost any programming language that VIM has syntax highlighting support for, and these languages are no exception.

Here’s what omnicomplete looks like for CSS files, for example:

"omnicomplete for vim"

Setting this up for your django project is simple as pie. It is helpful to have all your django projects in one parent directory for the following setup. You can obviously customize this to your needs, but this is the way I’ve set it up in my environment.

Add the following to a script in a directory of your choosing (~/bin/vim_wrapper is where mine is):

1
2
3
4
5
#!/bin/bash
export PROJECT=`python -c "import os; print os.path.basename(os.getcwd())"`
export PYTHONPATH="/path/to/your/projects/parent/directory/"
export DJANGO_SETTINGS_MODULE=$PROJECT.settings vim
$@

Then add the following line to your ~/.bash_profile or equivalent:

1
alias vi="vim_wrapper"

Or, you can call your vim_wrapper script by hand. (vim_wrapper file_to_edit.py)

Next, add the following lines to your ~/.vimrc file:

1
2
3
4
5
filetype plugin on
autocmd FileType python set omnifunc=pythoncomplete#Complete
autocmd FileType javascript set omnifunc=javascriptcomplete#CompleteJS
autocmd FileType html set omnifunc=htmlcomplete#CompleteTags
autocmd FileType css set omnifunc=csscomplete#CompleteCSS

I also prefer to re-map the default key binding () to , so I accomplish this by also adding the following line to my ~/.vimrc file:

1
inoremap <C-space> <C-x><C-o>

I also found this trick today while searching around…

What this function does is that if there is no completion that could happen it will insert a tab. Otherwise it checks to see if there is an omnifunction available and, if so, uses it. Otherwise it falls back to Dictionary completion if there is a dictionary defined. Finally it resorts to simple known word completion. In general, hitting the Tab key will just do the right thing for you in any given situation.

Add the following to your ~/.vimrc and you should be good to go. It works like a charm for me.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function! SuperCleverTab()
    if strpart(getline('.'), 0, col('.') - 1) =~ '^\s*$'
        return "\"
    else
        if &omnifunc != ''
            return "\\"
        elseif &dictionary != ''
            return "\"
        else
            return "\"
        endif
    endif
endfunction

inoremap <Tab> <C-R>=SuperCleverTab()<cr>

If you find yourself writing code in other languages, the following lines in your vimrc should be adequate:

1
2
filetype plugin on
set ofu=syntaxcomplete#Complete

You can now test that your installation works by changing directories to one of your django projects, firing up vim and running the following command:

1
:python from django import db

If you do not get a horrible error, you are good to go!

You can now access code completion by the following methods:

  • <C-p> - Shows a list of all local symbols. This is good if you do not have a tags file associated with the file you are editing.

  • <C-space> - Shows a list of all available symbols. You need to set up a tags file, which is outside the scope of this blog post

  • <C-x><C-o> - The original keystroke sequence that we re-mapped C-space to.

  • <Tab> - The all-powerful tab!`

I hope you enjoy your new-found power with vim as much as I do!