Quick and Dirty intro to Debian packaging

Required background

I assume you have installed a debian package atleast once in your life. And you are reading this because you want to know how they are created or you want to actually create one.

Back story

Over my career as a software engineer, there were several times I had to create a debian package. I always managed to avoid learning how to actually create it by sometimes using company internal tools and sometimes fpm.

Recently, I had the opportunity to create a debian package to deploy a project for a client, and I decided to learn how debian packages were “actually” created - “the whole nine yards”. Well, this is an account of that adventure. :)

As usual, I looked through the couple of blog posts on the internet. But most of them had the same “man page” look and feel. And I absolutely dread man pages. But without getting discouraged, I decided to plough through. I came across this page which finally gave me some much needed clarity.

Into the real stuff !

So, these are the things that I wanted to happen when I did dpkg -i on my package -

  1. Put the source files inside a “/opt/<project-name>/” folder.
  2. Put an upstart script inside the “/etc/init/” folder.
  3. Put a cron job in “/etc/cron.d/” folder.

The command that you use to build the debian package is

$ dpkg-deb --build <folder-name>

The contents of that folder is where the magic is.

Lets say that your folder is package. Inside package you need to have a folder DEBIAN. And then depending on the folder structure where you want your files to be, you have to create them accordingly. So in my case, I will have something like this -

$ tree -L 3 package/
package/
├── DEBIAN
│   ├── control
│   └── postinst
├── etc
│   ├── cron.d
│   │   └── cron-file
│   └── init
│       └── project_name.conf
└── opt
    └── <project-name>
        ├── main.js
        ├── folder1
        ├── node_modules
        ├── package.json
        ├── folder2
        └── helper.js

Consider the package folder to be the root(/). Don’t worry about the contents of the DEBIAN folder, we’ll come to that later.

After this, just run the command -

$ dpkg-deb --build package

Voila ! You have a debian package ready !

If you see any errors now, its probably related to the contents inside the DEBIAN folder. So, lets discuss it one by one.

  • control

If you just want to build the debian and get it done with, you only need to have the control file. Its kind of a package descriptor file with some fields that you need to fill up. Each field begins with a tag, followed by a colon and then the body of the field. The compulsory fields are Package, Version, Maintainer and Description.

Here’s how my control file looks -

Package: myPackage
Version: 1.0.0-1
Architecture: amd64
Depends: libcairo2-dev, libpango1.0-dev, libssl-dev, libjpeg62-dev, libgif-dev
Maintainer: Agniva De Sarker <agniva.quicksilver@gmail.com>
Description: Node js worker process to consume from the Meteor job queue
 The myPackage package consumes jobs submitted by users to the Meteor
 web application.

The Depends field helps you to specify the dependencies that your package might require to be pre-installed. Architecture is self-explanatory. (Small note on this - debian uses amd64 for 64 bit systems, not x86_64.)

For further info, see man 5 deb-control

  • preinst

If you want to run some sanity checks before the installation begins, you can have a shell script here. Important thing to note is that the packager decides the execution of the installation of the package depending on the exit code of the scripts. So, you should write “set -e” at the top of your script. Don’t forget to make it executable.

  • postinst

This is executed after the package is installed. Same rules apply as before. This is how my postinst looks -

#!/bin/bash
set -e

#Move the bootstrap file to proper location
mv /opt/myPackage/packaging/bootstrap.prod /opt/myPackage/.bootstraprc

#Clear the DEBIAN folder
rm -rf /opt/myPackage/packaging/DEBIAN
  • prerm

Gets executed before removing the package.

  • postrm

Gets executed after removing the package. You usually want to execute clean up tasks in this script.

Taking a step further

As you can figure, this entire process can be easily automated and made a part of your build system. Just create the required parent folders and put the source code and config files at the right places. Also have the files of the DEBIAN folder stored somewhere in your repo, which you can copy to the target folder.

Since, I had a Node project, I mapped it to my "scripts":{"build": "<command_to_run>"} in package.json file. You can apply it similarly for projects in other programming languages too.

TLDR

Just to recap quickly -

  1. Create a folder you will use to build the package.
  2. Put a DEBIAN folder inside it with the control file. Add more files depending on your need.
  3. Put the other files that you want to be placed in the filesystem after installation considering the folder as the root.
  4. Run dpkg-deb --build <folder-name>

Keep in mind, this is the bare minimum you need to create a debian package. Ideally, you would also want to add a copyright file, a changelog and a man page. There is a tool called lintian that you can use to follow the best practices around creating debian packages.

Hope this intro was helpful. As usual, comments and feedback are always appreciated !

Passing array buffer to Meteor gridFS

MongoDB has a document size limit of 16MB. To store larger file sizes, it is recommended to use GridFS.

Now, if you are a meteor user, you can very easily use the Meteor Collection-FS package to store and upload files. But it is slightly different when you actually want to store an object of size larger than 16MB which is not a file. Usually, these scenarios will come in the server side when you generate a large content and want to store that.

I was doing something like this -

//Collection initialisation
var Store = new FS.Store.GridFS("fileuploads");

FileUploads = new FS.Collection("fileuploads", {
  stores: [Store]
});

var buffer = new Buffer(JSON.stringify(jsonObj));
FileUploads.insert(buffer);

I found myself stuck with this error when I tried to use the insert function with the generated data. DataMan constructor requires a type argument when passed a Buffer

This is actually a mistake in the documentation here - https://github.com/CollectionFS/Meteor-CollectionFS#initiate-the-upload which says the insert function accepts a Buffer object at the server side. It doesn’t. Issue raised here. It accepts a file object with its data set as a buffer object along with a mime type.

Here is how to get it done-

var buffer = new Buffer(JSON.stringify(jsonObj));
var newFile = new FS.File();
newFile.attachData(buffer, {type: 'application/javascript'});
FileUploads.insert(newFile)

Now this will work :)

But we are not done yet !

How are we going to read the data back, if we are doing it at the client side ?

var fs = FileUploads.findOne({_id: fileId});
$.ajax({
  url: fs.url(),
  type: "GET",
  dataType: "binary",
  processData: false,
  success: function(data){
    var reader = new FileReader();
    reader.onload = function (event) {
      // event.target.result contains your data .. TADA!
      // console.log(event.target.result)
    };
    reader.onerror = function (event) {
      console.error(event.target.error);
    };
    reader.readAsBinaryString(new Blob([ data ],
      { type: 'application/octet-stream' }));
  }
});

Any comments and feedback is most appreciated

Update (May 25th, 2016): I just saw that the author of the repo has stopped from maintaining the project. Sorry to hear it. Its still a great library and I hope will help users who might still use this.

Enforcing git commit structure

I love git. Through and through. I love its distributed nature and the immense power it gives to the user. For someone who likes to be in complete control over his code, this was the real deal.

Naturally, I also started to follow the best practices around using git. Two good posts in this regard that I like are

A subject line, gap, then a nice body. Clean git commits. All nice and dandy.

But one thing was always nagging me at the back of my mind. If we are to enforce a process, there should be a way to automate this, and not have a newcomer memorize all the points before starting to make commits.

Git hooks to the rescue ! I had been planning to look into git hooks for some time but somehow didn’t get the time. Well, this was the perfect time.

After a bit of meddling around, I found that git provides hooks to various stages of the commit lifecycle. All at client side. And some at server side too. And guess what, you can run a script to check the commit message and if you don’t like it, you can choose to reject it. Time for some code :)

#!/bin/bash
cnt=0
while IFS='' read -r line || [[ -n "$line" ]]; do
  cnt=$((cnt+1))
  length=${#line}
  if [ $cnt -eq 1 ]; then
    # Checking if subject exceeds 50 characters
    if [ $length -gt 50 ]; then
      echo "Your subject line exceeds 50 characters."
      exit 1
    fi
    i=$(($length-1))
    last_char=${line:$i:1}
    # Last character must not have a punctuation
    if [[ ! $last_char =~ [0-9a-zA-Z] ]]; then
      echo "Last character of the subject line must not have punctuation."
      exit 1
    fi
  elif [ $cnt -eq 2 ]; then
    # Subject must be followed by a blank line
    if [ $length -ne 0 ]; then
      echo "Your subject line follows a non-empty line. Subject lines should always be followed by a blank line."
      exit 1
    fi
  else
    # Any line in body must not exceed 72 characters
    if [ $length -gt 72 ]; then
      echo "The line \"$line\" exceeds 72 characters."
      exit 1
    fi
  fi
done < "$1"

Here, I have just enforced a few basic rules which I feel should be followed by every repo. You can of course choose to modify or extend them.

Just put this code in a file called commit-msg. And drop it in the .git/hooks/ folder of your repo. Don’t forget to make it an executable. And you are done !

Proper gist link here - https://gist.github.com/agnivade/67b42d664ece2d4210c7

Mixing React and jQuery

Recently, I was doing a client project and got a chance to build a site. I decided this would be the perfect opportunity for me to build it in React and sharpen my React skills. Adventure time !

The site was coming up nice and smooth. But being a hardcore jQuery guy, I couldn’t resist myself from using jQuery here and there. The alternative was to spending more effort in reading React docs and implementing everything in React way. Well, you can guess what was about to happen.

I was stuck with this weird error - Invariant Violation: ..... This probably means the DOM was unexpectedly mutated (e.g. by the browser).

A quick search gave me links to

But none of these solved my issue. I just had a faint idea that it had to be something related to DOM manipulation and my intermixing of jQuery with React was the culprit.

After hours of playing cat and mouse with the bug, I finally located it. It was in these lines -

$("#poPreviewContainer").empty();
POPreviewRendered = React.render(<POPreviewDialog
    poData={this.props.poData}
    poItems={this.props.poItems} />,
    document.getElementById('poPreviewContainer'));

Basically, there is a dialog that I am dynamically populating everytime with new data and putting it inside a div to be shown in the browser. Now, being the jQuery user that I am, I did a .empty() on the div and did a React.render to re-render the new dialog. And it turns out React does not like someone else to empty divs without telling it.

The solution was to use React specific function calls to empty the div.

React.unmountComponentAtNode(document.getElementById('poPreviewContainer'));

A gentle reminder to those of you migrating from jQuery to React. Always think in React. And try to ditch jQuery completely. And if you can’t, have a look at posts like these to help you out :)