Programming is Debating
How difficult is it to start programming? By starting I mean producing real code, be it a personal project or something you will get paid for. But it is not written for the purpose of learning.
Most of the readers of this blog are professional programmers. I am sure for many of you starting was quite easy and natural. But for a lot of people it is an extremely difficult obstacle. I know people, more than 50 people, who studied CS or CE in university but has never written any real code. They get anxious when the possibility presents itself.
The problem with the start is usually the false expectations about programming. There are certain myths about heroic programmers writing incredible programs in impossible conditions with little or no effort. This is of course bullshit. But I have witnessed again and again people setting their expectations about the experience of writing code by these absurd standards. The actual experience inevitably fails to deliver these expectations and the person gives up. This doesn’t have be so. Programming is supposed to be fun.
When we open a book and start reading, the words we see are unlikely the author’s original words. The original, raw content get edited before printing. It is an iterative process. I don’t want to get into the details about publishing1 but beginners should study this process carefully. Because the process of programming is same, sans the magic.
So the act of programming is also an act of debating. You debate with your tests using your code. When all tests pass, the debate is over. When you are making design decisions, no matter how small they are, there is a debate going on between the requirements, the resources and your professional judgement. Therefore it would be unreasonable to expect programming to be a smooth, frictionless process. If you want to start you should be prepared for it.
I would like to share a few pointers that I hope will make the start more predictable if not easier:
- Try to find a real problem to work on. Forget about educational/theoretical problems. Solve a real, practical problem. There is one extremely important thing to remember here; the scope of the project must be as small as possible. A series of 2-day projects are much better than a 2 week project.
- Start by documenting the usage of the code. Write an example script that imports your hypotetical code and uses its functionality. In other words; design top-down, program bottom-up.
- Then flesh out the structure. Create files, classes, functions, comments. Writing code is good for warming-up to write some code.
- Divide and conquer. Implement incrementally, accomplish one thing at a time. If a functionality is giving you hard time, try to write something that produces some results but not exactly what you expect. Then continue iterating until you get it right. Don’t wait for the programming muse to come and light the way. Sometimes you need to invent all the wrong implementations before figuring out the correct one.
- Never hesitate to ask for help. Programming is debating, why not introduce your peers and mentors into the process. If you are asking for help make sure you have a concrete question. Input, expected output and your current code is usually enough. Even when you don’t have a problem or question, share your work with others and try to get as much feedback as possible.
- Do your research. There is no getting around the reading. I would be lying to you if I said otherwise. If you want to win the debate you need to be well prepared. It may be a little overwhelming in the beginning. But as you build up your knowledge (and experience) you will enjoy reading more.
I hope these pointers are helpful. But I think the most important thing to remember is programming is not a mechanical process but it is very human, I call it a debate, some call it art.
1: Nor am I an expert on the subject.
My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor
I have given a presentation about django_compressor at PyCon APAC 2011. Slides are below for everyone to see:
Drawing Gradients with PyGame
I have been learning myself some PyGame on my free time. Best way to learn new technologies, I believe, is to develop small project and doing a lot of reading. Everybody seems to agree on the former but most fail to put effort doing the latter.
I needed to draw gradients for the background of my little learning project. First I tried the naive approach: iterating over each pixel and setting the correct value. This takes forever as you could have guessed. I continued my research to find out the close connection between PyGame and NumPy. Yes, fast array operations. My gradient generating code using linspace and tile is below:
def generate_gradient(from_color, to_color, height, width):
channels = []
for channel in range(3):
from_value, to_value = from_color[channel], to_color[channel]
channels.append(
numpy.tile(
numpy.linspace(from_value, to_value, width), [height, 1],
),
)
return numpy.dstack(channels)
I create three 1D arrays for each channel and then tile them to get a 2D array. Then I stack those 2D arrays and get combined (r, g, b) values. Each 1D array contain values linearly sampled between given maximum and minimum values by linspace. I can then blip this array on a surface like this:
gradient = generate_gradient((63, 95, 127), (195, 195, 255), 1024, 600)
pygame.surfarray.blit_array(
some_surface,
pygame.surfarray.map_array(some_surface, gradient),
)
This works quite well. But I was curious whether there is a better method, so I did some googling. First I stumbled upon James Tauber’s gradient code. Arrays! I was a bit ashamed not remembering standard library had array module. James Tauber’s code builds an array with the correct values and then writes the image as a PNG file. It also has the ability to generate multi-gradients.
Then I found gradients package for PyGame. This package does much more than linear gradients. It creates a PyGame surface of 1 pixel thickness, fills it with the gradient values and then resizes it.
I made a small test to see which method runs faster. I had to strip file and compression related code from James Tauber’s gradient module. I run a simple method tha just generates a gradient image/array in memory and returns it 10 time, below is the average times:
- 320x240:
- numpy: 0.01469659805
- array: 3.6310561180
- pygame: 0.0069756984711
- 800x600:
- numpy: 0.1323119163
- array: 19.9719331265
- pygame: 0.0192141056061
Here numpy is my method above, array is the stripped version of James Tauber’s code and pygame is the gradiends package. If my benchmarking method doesn’t have dramatic errors fastest way to generate linear gradients is to fill a strip with correct values and then to resize it.
Now I’ll go back to work on my learning project.
How To Create A Debian VM With Qemu
I would like to post my notes as a little tutorial here. I am usually using these virtual machines as cheap staging servers. The first part of this tutorial, you hopefully need to do only once: creating a fresh Debian system. In the second part we will build on this image to create many different servers.
Creating A Base Debian System
We will create a Qemu machine and install Debian Lenny on it first:
# Download Debian image
wget http://debian.osuosl.org/debian-cdimage/current/i386/iso-cd/debian-504-i386-businesscard.iso
# Create base VM image
qemu-img create -f qcow2 debian.qcow2 2G
Our disk image will have a 2 Gigabyte size limit. You can pick a different size if you need.
Now we need to power on our VM and install Debian:
# Install Debian
qemu -enable-kvm -k tr -cdrom debian-504-i386-businesscard.iso -hda debian.qcow2 -boot d
You don’t need to allocate a large swap disk, 128MB should do just fine for a file/web server. Also I wouldn’t bother creating a seperate partition for /home/.
Next let’s log in as the user (www here) we have created to make final changes:
# logged in as user
dpkg-reconfigure console-data
aptitude install ssh sudo
echo "www ALL=(ALL) ALL" >> /etc/sudoers
At this point you might want to take a backup of debian.qcow2. (Even though we will open it only read-only from now on)
Creating The Actual VM
To save time and space we will use copy-on-write disks and re-use debian.qcow2.
# Create the actual VM's disk
qemu-img create -f qcow2 -o backing_file=debian.qcow2 actual.qcow2
Actually we are done. You can log in to your VM using the following command and start installing/configuring/running:
qemu -enable-kvm -k tr -hda actual.qcow2 -net user -net nic \
-redir tcp:5022::22 \
-redir tcp:9080::80
A few things to note about the command above:
- -enable-kvm is meaningful only if you have kvm kernel module installed. It improves performance a great deal, so it’s highly recommended.
- You probably need to change -k tr according to your keyboard’s layout.
- We are setting up two TCP redirections. 22 is for SSH and 80 is for HTTP. You can add more ports if you need.
Finally, I suggest you to prefer SSHing your VM instead of logging in directly:
# SSH into the VM
ssh -p 5022 www@localhost
I hope some of you find this useful.
How to Install MySQL with Fabric
I just want to share a small fabfile snipplet that installs mysql-server package on a Debian machine if it’s not already installed. Actually it should have been quite straightforward; just issue an apt-get command, right? But during configuration it displays a curses dialog for MySQL root password. This of course blows your automated configuration plans. We will seed this value to debconf database to avoid this dialog.
First let’s have a look at the supporting code:
from __future__ import with_statement
from fabric.api import *
from fabric.utils import warn
def apt_get(*packages):
sudo('apt-get -y --no-upgrade install %s' % ' '.join(packages), shell=False)
Our apt_get fabric command issues apt_get install that answers yes to all questions and does not upgrade if the requested package is already installed. Now let’s take a look at the main command that installs mysql-server
def install_mysql():
with settings(hide('warnings', 'stderr'), warn_only=True):
result = sudo('dpkg-query --show mysql-server')
if result.failed is False:
warn('MySQL is already installed')
return
mysql_password = prompt('Please enter MySQL root password:')
sudo('echo "mysql-server-5.0 mysql-server/root_password password ' \
'%s" | debconf-set-selections' % mysql_password)
sudo('echo "mysql-server-5.0 mysql-server/root_password_again password ' \
'%s" | debconf-set-selections' % mysql_password)
apt_get('mysql-server')
First we want to make sure mysql-server is not already installed. For this reason we issue dpkg-query --show mysql-server. Note that if this command fails then mysql-server is not installed and we can proceed otherwise we want to return. We hide the ugly error messages by running our dpkg-query command within a settings context:
with settings(hide('warnings', 'stderr'), warn_only=True):
result = sudo('dpkg-query --show mysql-server')
if result.failed is False:
warn('MySQL is already installed')
return
The rest of the code is pretty straightforward. We prompt to the user running fabric for the root password and seed its value twice into the debconf database:
sudo('echo "mysql-server-5.0 mysql-server/root_password password ' \
'%s" | debconf-set-selections' % mysql_password)
sudo('echo "mysql-server-5.0 mysql-server/root_password_again password ' \
'%s" | debconf-set-selections' % mysql_password)
apt_get('mysql-server')
Finally, having finished our little dance, we install mysql-server. I hope this helps some automated deployment believer out there.