Upstart can't read environment variables

I was trying to configure a gunicorn server with upstart for easily starting/stopping my service. But it turned out to be problematic since upstart does not read environment variables.

How did I encounter this error? I was trying to parse signed_request from Facebook canvas via the following line:

fb_request = facebook.parse_signed_request(signed_request, APP_SECRET)

where

APP_SECRET = os.environ.get('FB_APP_SECRET', 'jhdklu3sh4o8y4o8fh34')

I realized that it couldn't read the environment variable and was using the default app secret so couldn't decrypt the signed request. My environment variables were in .bashrc of both regular user and root but it did not read. So I added them to the upstart script:

script
     export HOME=/home/user/project
     cd $HOME
     export FB_APP_ID=859827u4fhk4vnf
     export FB_APP_SECRET=4ojfo92ufo2489fyuo482fy2
     . venv/bin/activate
     exec gunicorn myapp:app
end script

Embedding Skulpt Python Interpreter in Reveal.js slides

Lately, Slide libraries in HTML+JS+CSS became so popular. I liked reveal.js the most and wanted to prepare a presentation on Python. I wanted to perform demos throughout the presentation so I thought I could embed a Python interpreter (skulpt) on slides.

See sample HTML file: https://gist.github.com/aladagemre/9124007

Just place your html file inside reveal.js folder. Place skulpt js files in reveal.js/js folder.

There exists two slides with the interpreter embedded. Just copy and paste those sections for having more.

Flask, Facebook Canvas App, localhost and SSL

If you're developing a facebook canvas app with Flask, then you're asked to provide URLs for your app. Initially I used myapp.herokuapp.com address for http and https. But I realized that for testing local changes, I'd like to use localhost.

When I give localhost, it says it's not SSL supported. Then I had to create SSL certificates and install pyopenssl to use them. See: http://kracekumar.com/post/54437887454/ssl-for-flask-local-development

But what I saw was that connection was untrusted and I could not Add an Exception. I learned that browsers do not trust localhost for SSL. So I had to create an alias for my herokuapp. Added the following line to /etc/hosts

127.0.0.1 myapp.herokuapp.com

And set the canvas url as myapp.herokuapp.com for http and https.

Now when the app starts, tries to load https://myapp.herokuapp.com. It loads https://127.0.0.1:443 which is listened by the Flask app:

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 443))
    app.run('0.0.0.0', debug=True, port=port, ssl_context=('/home/user/projects/myapp/server.crt', '/home/user/projects/myapp/server.key'))

This way I could overcome the SSL localhost problem for facebook canvas apps.

UPDATE: If you don't want to run the app as root,  just forward port 443 to 3000 and listen 3000 port with the following command as root:

iptables -t nat -A OUTPUT -p tcp --dport 443 -j REDIRECT --to-port 3000

and to cancel it:

iptables -t nat -D OUTPUT -p tcp --dport 443 -j REDIRECT --to-port 3000

Note that forwarding 443, you won't be able to connect SSL web sites throughout your pc.

Maybe using vagrant could be more elegant.

Proxy Pattern for Extending non-extendable Python Class

graph_tool library is based on boost C++ library and provides Vertex class binding for Python. If we wanted to extend this Vertex class and add some attributes and methods, it wouldn't let us do that due to private constructor in C++ code.

RuntimeError: This class cannot be instantiated from Python

We can overcome this obstacle using Proxy pattern.

In the __getattr__ method, if the attribute(or function name) is not in the Subclass MyVertex, then it looks for attributes of Vertex object that is defined inside MyVertex.
 

 Here is the code:

http://code.activestate.com/recipes/578576-extending-non-extendable-c-based-python-classes/

Fix for SuspiciousOperation: Invalid HTTP_HOST header

With Django 1.5, HTTP_HOST header filter is applied to the requests.  If the HTTP_HOST header is not among the ALLOWED_HOSTS list in the settings.py, an error is raised, saying this is a suspicious operation.

Let me give an example. Someone (who is not Google), is trying to reach my IP address with the HTTP_HOST www.google.com as if I'm hosting the google.com homepage.:

SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): www.google.com


This happens frequently, leading to emails sent to the admins which is annoying. To overcome this, you can add the IP Address of the requester to /etc/hosts.deny file.

Indeed I thought I could put a hostname filter in nginx configuration, especially in the listen part but my configuration did not have an effect.

The attacker tries to exploit a vulnerability and performs a scan over the web. They seem to be from Vietnam.

ParallelPython vs multiprocessing

Today I'm working on parallelisation of a process I have written. Process is simply a text conversion. I have a translator class, organism directories and files inside these directories. What I want to do is to split the data among the processors and perform the operation faster.

To do this, I tested multiprocessing and ParallelPython modules of python. Without using these modules, it took 38 seconds to perform the task whereas with the help of these modules, it went down to 29 seconds(multiprocessing) and 30 seconds (ParallelPython). Not a great deal but better than nothing. By the way, ParallelPython is way too complicated compared to multiprocessing.

Here is the code for ParallelPython:

import translator
import os
from utils.pp import pp

base = "/some/path"
organisms = [ "organism1", "organism2", ...]

def convert_organism(base, organism):
t = translator.BiogridOspreyTranslator()
# uses os module here
t.translate()

if __name__ == '__main__':
job_server = pp.Server(ppservers=())
jobs = [(organism, job_server.submit(convert_organism, (base, organism,), (), ("os","translator",))) for organism in organisms]
for organism, job in jobs:
job()

ParallelPython requires you to tell him the modules the functions requires. I didn't like that.
And here is the code for multiprocessing:

from translator import *
import os
from multiprocessing import Pool

base = "/some/path"
organisms = [ "organism1", "organism2", ...]

def convert_organism(organism):
t = BiogridOspreyTranslator()
# uses os module here
t.translate()

if __name__ == '__main__':
pool = Pool(processes = 2)
pool.map(convert_organism, organisms)

QGraphicsItem.itemChange event and ItemPositionChange in Qt 4.6

Recently, I had a bug in Robinviz that I could not figure out where it come from. I was drawing a graph and nodes could be moved. Whenever I moved the node, edges connected to it used to come with it to the new position. But after upgrading to Qt 4.6, I saw that edges were not moving.

After some investigation, I realized that QGraphicsItem.itemChange event did not produce QGraphicsItem.ItemPositionChange but only selected, deselected signals. Googling for it, I hardly found that the problem was with the update. Due to performance reasons, Qt developers decided to stop emitting geometrical signals and wanted us to switch it on by supplying a flag for the QGraphicsItem:

  try:
           # available only in Qt 4.6
           self.setFlag( QGraphicsItem.ItemSendsGeometryChanges)
  except:
           # no need to do this in Qt 4.5
           pass

You can do this flag option in your constructor. I used try/except because I didn't know what might happen in 4.5 as there was no flag called ItemSendsGeometryChanges. In some other websites, following flag was suggested but it did not work for me:

self.setFlag(QGraphicsItem.ItemSendsScenePositionChanges, True)

For those who might be interested how I used itemChanged, I'll provide a portion of my code:

  def itemChange(self, change, value):  
     if change == QGraphicsItem.ItemPositionChange:
         self.updateEdges()

     return QVariant(value)

RobinViz Beta released

Things are going well and I've finished my first semester of my Masters. All of my grades are great and I'm just starting the second semester. On the other hand, I'm working on our project. When I joined the project in September 2009, one last step was missing. That was the Horizontal Coordinate Assignment Problem in Layered Graph Drawings. I've implemented Fast and Simple Horizontal Coordinate Assignment paper of Brandes et. al and we submitted our paper to International Symposium in Biocomputing (ISB 2010) in Calicut, India. Our paper was accepted and after that we decided to improve the GUI of our implementation. We developed a brand new GUI with PyQt4 and used the C++ code behind the scene to do the most of the scientific calculations. We published this novel software, RobinViz Beta version on http://code.google.com/p/robinviz.

With RobinViz, you can visualize PPI Networks and Gene Ontology as biclustered graphs. In these graphs, reliability of interactions are expressed with the thickness of the edges, h-value of the biclusters are expressed as node width etc. So the most important part of the data comes in front. I hope it shall be useful for all the scientists working on this subject.

This semester I'll be taking Algorithms, Data Mining and Advanced Java courses. I hope I'll learn a lot from them. By the way, while our professor presented our paper at the conference, I gave Introduction to Java tutorial lectures in his classes. That was also a nice experience.

Internship @ Pardus

I've entered my TOEFL exam and it was pretty good, except for the listening part in which I lost my attention and thought about the examination software (it was written in Java but the Cambridge, Longman etc. preparation CDs are prepared in Shockwave as I saw). I asked myself: "Why don't they write them in Java too so that I can use them on Linux? And guess what, at that moment, a lecture was being given on the computer and I lost lots of details :D

Nevermind, I started my third internship last week, in TÜBİTAK (National Scientific Research Center) on Pardus Linux Distribution Project. It's been very beneficial for me so far. I've learned lots of things about software, python, GUI design and opensource. I want to share what I've learned in this blog as much as I can do. You can see examples here. I've written a program called Sahip, which is an XML generator. It produces the installation settings for Yali (installation software of Pardus) to perform silent installs.

PyQt4 GUI Building
Let's start with PyQt4 GUI building. I used to think that we could not import GUI files from python modules and this caused us to refactor our python code whenever we wanted to change the gui, unlike GTK with Glade (you can import XML from python). But I was wrong. There was some options I could follow. One of them is the kdedesigner module for python. But I used the other one, inheriting the compiled GUI.

After you design a GUI on Qt Designer 4, you can compile it to Python code with:

pyuc4 gui.ui -o mygui.py -x

and after that we can import and inherit it as written below:

from PyQt4 import QtCore, QtGui
from sahip.usergui import Ui_UserDialog

class UserDialog(QtGui.QDialog):
  def __init__(self, caller=None, user=None):
    QtGui.QDialog.__init__(self, None)
    self.ui = Ui_UserDialog()
    self.ui.setupUi(self)
          
    self.ui.lineEdit.setText('test')
    # All the other stuff here or in other methods.

if __name__ == "__main__":
  import sys
  app = QtGui.QApplication(sys.argv)
  UserDia = UserDialog(None)
  UserDia.show()
  sys.exit(app.exec_())

That's all!

Opening up a new dialog
I imported UserDialog class above and used it as a dialog.

def slotUserNew(self):
      self.userDialog = QtGui.QDialog(self)
      self.userDialog.ui = UserDialog(self) # caller=self
      self.userDialog.ui.show()

Lists items with checkboxes

You can set all the items of a list have a checkbox near it with the following code:

for i in range(count):
  item = self.ui.groupList.item(i)
  item.setFlags(QtCore.Qt.ItemIsUserCheckable | QtCore.Qt.ItemIsEnabled)
  item.setCheckState(QtCore.Qt.Checked)

Handlers
In lists, I could only store the username of a user but the rest of the information should be stored somewhere else. That could be a dictionary, list etc. Dictionary was the most convenient for me but I had to define a dictionary for each list/combobox. That was a replication of code! So I wrote WidgetHandler, and specialized ListHandler, ComboBoxHandler to store them and defined addItem/removeItem, etc. methods to update both the dictionary and the widget. It was much more clear.


i18n

I have always wondered about internationalization and I saw that it was pretty easy. What you should do is to add the following code to the beginning of each of your files you want to be translated.

import gettext
__trans = gettext.translation('sahip', fallback=True)
_ = __trans.ugettext

and you will use _('string') instead of 'string' if you want that string to be translated. And here comes the pot file generation part. As you can see the example project on the link above, I have a tools, po and sahip directory. So I recommend you to create such directories. I put pot generator sh file into tools dir and po/pot files into po dir. After creating an empty po dir, you can execute the following script (above po dir)

#!/bin/bash
LANGUAGES=`ls po/*.po`
set -x

xgettext -L "python" -k__tr -k_ sahip/sahip sahip/*.py -o po/sahip.pot
for lang in $LANGUAGES
do
  msgmerge -U $lang po/sahip.pot
done

As you might understand, this script crawls through the given paths (sahip/sahip and sahip/*.py) and generates pot file by reading the py files and main (non-extensioned) sahip file. You will then find the pot file in your po directory. You can translate and rename it to lang.po format (tr.po, de.po, es.po). Then the python setup script will probably compile po file into mo file and copy it into proper place.


Setting Icon

1. Create a directory called images in the same directory of your codes.
   
2. Put an image file in it, such as icon.png
3. Create a qrc file such as resources.qrc and fill it with the content
   
   images/icon.png
   
   
4. Open your GUI with Qt4 Designer and click on ... button of windowIcon on the Property Editor when main form is selected.
5. A dialog will be shown, click on pen and then the open button (middle). Select the qrc file you created.
6. Select the icon appeared on the right side of the dialog and click OK.
7. Save the GUI.
8. On the console, apply the following command:
   

   
   pyrcc4 resources.qrc -o resources_rc.py
   

   
   
9. Generate your gui py file with pyuic4 and that's all!

Publishing the Code
I needed to copy YALI setup.py file and modify it for sahip. This file compiles po and qrc files, and then copy required files to the system for installation. Then I wrote a digestrelease.py script which copies the program directory to desktop and removes the unnecessary ones. Then the packager.py file compresses the directory and sha1sums it, updates a sample pisi pspec.xml file with the sha1sum, uploads the compressed targz file to ftp server and then build the pisi package, and install it.

It was a good way of automation for me. I could make modifications and try the result in seconds. I might have written this entry too confusing, sorry but I don't have much time. My next project is to develop a web site where files can be searched within the pisi packages so that for instance you can find which package the 'ls' file comes from. I'm currently writing the database generator and after I'll pass to the Django side.

New electrolyzers open up the way to Solar Century

In the earlier days of this week, while watching CNN International, I run into a wonderful news. The visitor was a professor from MIT and they were going to talk about a recent invention on storing solar energy.

As you may know, solar energy is known to be unlimited as long as the sun lives but limited as it can't be used efficiently because not only the solar panels can't absorb all the energy but also the energy can be obtained only during the daytime. It seems that the absorption is still a problem but the recent news promise in storing the excess energy.

The scientists at MIT, have found a new method to store the excess energy by dividing the water into its ingredients, Hydrogene and Oxygene. Heey, this is already being done worldwide, you might say. But this operation needed high maintenance costs and abnormal operation conditions(temperature, pressure, etc.). This new method, inspired by the photosynthesis, uses a new catalyst consisting of cobalt metal, phosphate and an electrode to produce Oxygen and another catalyst like platinum to produce Hydrogen and does not require any special conditions for the reaction to start.

As this is an easy-implementing method of storing energy, it is tought to change the world, contributing to the works on solutions for the global warming problem. But as the proffessor says, it will take 8 years for us to have these sets on our roofs. I wish it was closer if it was that easy to implement but maybe the other (efficient absorption) problem needs to be solved in order these products to be more efficient.

By the way, I have finished my internship. Nowadays I'm preparing for my TOEFL exam and working on Python as usual. My Python presentation went very well and I saw that most of the aspects of Python could be introduced in 2 hours! Such a beautiful language...

BOLO and Python

We worked on BOLO for some time but we found it very complex. The code is unnecessarily too much (less code could achieve the same functionality). One manager for each POJO is defined although those manager methods could be defined in POJOs. So, crawling in the code is a funny(!) way of spending time for us (me and Melih). We tried to implement the search by status functionality but got stuck at the enumarator types for the Status class properties. Showing them on JSF and selecting them is a problem. Then we tried to add a timestamp on the CVs uploaded. But finding the code where the file is uploaded, where the filepath is inserted into the database were the problem in that case. Moreover, the service had been giving "Out of memory" errors after 10 minutes of run.

Getting bored from all these stuff, I tried to write the model in Python/Django. It was a good try. I found a model to UML Diagram script and generated a database model diagram. Showing the diagram, Melih was impressed by the easiness and practicality. Then he wanted me to show my work to the team leader and I showed what I did:But he did not welcome my work as much as I expected: "Good, but when your internship ends, there's nobody to maintain it here. I don't want my staff to learn Python for such a thing. Because nobody uses Python". I didn't know what to fell sorry for. For my effort, or the misknowledge for Python. Nevermind, I'm spending my time with some trying on that BOLO stuff, and some with fixing some errors in my Python projects.

By the way, some colleagues wanted me to give a talk about Python and how it is used. I'm going to get prepared for it and give a brief presentation called "Python for Java Geeks". So, first of all, I have to find some resources on the differences between Java and Python so that making it easier to understand, I can go faster. Maybe in 3 hours, instead of 8 hours that I had given at school for programming newbies.

Google Code Jam

Last Thursday, I saw a blogpost on a Python related RSS, mentioning about the Google Code Jam. I saw that it was a programming contest and took a look at it. Reading the rules, I saw that there was a 4 - 8 minute limit. At first I thought that it was the time to solve a problem that is given to you. I thought it was a crazy idea to solve a problem in 4 minutes and code it. As I was at the company, I put it off to the evening for details. When I came home, I did some stuff I needed to do and when I realized that the limit is not for solving but submitting the output after downloading the input, I was angry with myself. Why didn't you read it! There had been 4 hours left. It was 10 pm and the contest was going to end by 2 am. So I started to look at the questions. There were 3 questions of which I had to solve 1 of them correctly to qualificate to the first round. I needed to code to solve the problem and then download two (small and big) inputs and submit them.

I started with the first question, here you are a summary: "In a planet, there are some search engines. When you search the name of a search engine on itself, the planet explodes. So you shouldn't search for Gogool on the Gogool engine. Scientists developed a central system to prevent this. This system forwards the search requests to the engines other than the engine with the name in the request. The question is that with the least switch between the engines, how many switches are required to serve all the search requests."

I developed a greedy algorithm. First, I was going to determine the indexes of the engines where they were first seen. Then I was going to find the furthest one and serve the request before that with this engine. For example below, the furthest first seen engine is NSM. So I should handle the first 4 requests with NSM:

Yehhaw
Yehhaw
Gogool
Dont Ask
NSM
Gogool
DontAsk
DontAsk
Gogool
Yehhaw

Then I had to remove the indices below 5 which belongs to NSM. Then I looked ahead like having a new beginning from NSM:
Gogool
DontAsk
DontAsk
Gogool
Yehhaw

So, it is now obvious that the index 11, (i.e. Yehhaw) is the engine to be used for serving 5-10 (NSM-Gogool). And then the previous indices would be removed after that and so on. The input files were large with the 20 different cases in the number of various engines and the number of searches.

I coded this algorithm on Python and when I ran the sample input, it was successful. Then I downloaded the input file but it told me that my output was incorrect. After the contest, I learned that there were only 3 digits wrong, possibly caused by boundary checking or exceptional conditions.

After the wrong answers, I decided to look for the other questions and started the second question, the train question. There were two towns, A and B between which there were train routes. Input file consisted of the turnaround time for the trains, the timetable from A and timetable from B. The question is that with how many trains at minimum you can handle these timetables such a way that a train coming from A, after completing the turnaround can handle the closest train route on the timetable B to go back to A.

A: 9.00 - 12.30
Turns around in 5 minutes
B: 13.00 - 15.00
The turned around train undertakes the 13.00 route and goes to A. So there's no need to have an additional train to go from B to A at that time.

At first I did not have a clear algorithm but I thought it would be great if I could match the trains as complement and show the complement pairs on the screen so that I could draw inferences on it. Maybe I could develop a matching algorithm after marking them as complements. But the things were not the same as they seemed to be. When marking as complement, I fell into a infinite loop, the train from A matched B and the train from B matched A and A did the same, etc ... They matched each other again and again. Then I realized the time was nearly up. I could not do anything but to post my code and the small output. They were incorrect, of course. But at least a hope that they would look at the codes.

So, I could not qualificate to the first round. I wish I could know it before. For months, they had been practicing for the contest and I hadn't ever heard about it. It was the contest day (last day) when I saw it. So all I can do is this in four hours. Maybe I can do better next year, using all the 24 hours. :)