Adding external libraries (jar files) to an ivy project (Nutch)

I wanted to use my Apache Giraph fork in Apache Nutch 2.1. The Problem was that Giraph used mvn and Nutch used ivy. Although Nutch could fetch jar files from maven repositories, my customized jars are in none of those repos. So I had to find a way to use my local jars.


Add the following line to ivysettings.xml of Nutch:

<module organisation="org.apache.giraph" name=".*" resolver="internal"/>

Add the following lines to ivy.xml of Nutch:

    <dependency org="org.apache.giraph" name="giraph" rev="1.1.0-SNAPSHOT"
      conf="*->default" />
    <dependency org="org.apache.giraph" name="giraph-hbase" rev="1.1.0-SNAPSHOT"
      conf="*->default" />
    <dependency org="org.apache.giraph" name="giraph-examples" rev="1.1.0-SNAPSHOT"
      conf="*->default" />

Now create put jar files under .ivy2 subfolders:

/home/emre/.ivy2/local/org.apache.giraph/giraph/1.1.0-SNAPSHOT/jars/giraph.jar

/home/emre/.ivy2/local/org.apache.giraph/giraph-hbase/1.1.0-SNAPSHOT/jars/giraph-hbase.jar

/home/emre/.ivy2/local/org.apache.giraph/giraph-examples/1.1.0-SNAPSHOT/jars/giraph-examples.jar  

Now when you run ant runtime and start a crawling, you will see that your local jar files are binded to your app:

SLF4J: Found binding in [jar:file:/home/emre/tmp/apache-nutch-2.1/runtime/local/lib/giraph-1.1.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/emre/tmp/apache-nutch-2.1/runtime/local/lib/giraph-examples-1.1.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/emre/tmp/apache-nutch-2.1/runtime/local/lib/giraph-hbase-1.1.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]

Proxy Pattern for Extending non-extendable Python Class

graph_tool library is based on boost C++ library and provides Vertex class binding for Python. If we wanted to extend this Vertex class and add some attributes and methods, it wouldn't let us do that due to private constructor in C++ code.

RuntimeError: This class cannot be instantiated from Python

We can overcome this obstacle using Proxy pattern.

In the __getattr__ method, if the attribute(or function name) is not in the Subclass MyVertex, then it looks for attributes of Vertex object that is defined inside MyVertex.
 

 Here is the code:

http://code.activestate.com/recipes/578576-extending-non-extendable-c-based-python-classes/

Latex and Bibtex

Using bibliography in latex might be headache sometimes. While I was writing my MSc thesis, I had taken some notes. I'd like to share them with you.

For a while, I was wondering what is the difference between bib and bbl. I learned that when you execute bibtex, bib files are converted to bbl.

Steps for running BIBTEX with LaTEX

1. Run LaTEX, which generates a list of \cite references in its auxiliary file, .aux.
2. Run BIBTEX, which reads the auxiliary file, looks up the references in a database
(one or more .bib files, and then writes a file (the .bbl file) containing the formatted
references according to the format specified in the style file (the .bst file). Warning
and error messages are written to the log file (the .blg file). It should be noted that
BIBTEX never reads the original LaTEX source file.
3. Run LaTEX again, which now reads the .bbl reference file.
4. Run LaTEX a third time, resolving all references.

File icon extensions (emblems) like of Dropbox

I have been wondering how Dropbox could put a icon on the file indicating it's synchronizing or synchronized. I digged its code and the files of gnome itself and found the way for that.

You can find some examples for these icons in /usr/share/icons/gnome/scalable/emblems/ which belongs to Gnome.

If you want to have a custom icon, then you can put your files in /usr/share/pixmaps

An example filename: emblem-my-synced-symbolic.svg

I had the impression that the file name should start with emblem but I'm not really sure.

After having your file in /usr/share/pixmaps, you can activate this icon for a file:

gvfs-set-attribute -t stringv /home/emre/sample.txt metadata::emblems 'my-synced-symbolic'



Now your file has the svg icon on it.

Fix for SuspiciousOperation: Invalid HTTP_HOST header

With Django 1.5, HTTP_HOST header filter is applied to the requests.  If the HTTP_HOST header is not among the ALLOWED_HOSTS list in the settings.py, an error is raised, saying this is a suspicious operation.

Let me give an example. Someone (who is not Google), is trying to reach my IP address with the HTTP_HOST www.google.com as if I'm hosting the google.com homepage.:

SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): www.google.com


This happens frequently, leading to emails sent to the admins which is annoying. To overcome this, you can add the IP Address of the requester to /etc/hosts.deny file.

Indeed I thought I could put a hostname filter in nginx configuration, especially in the listen part but my configuration did not have an effect.

The attacker tries to exploit a vulnerability and performs a scan over the web. They seem to be from Vietnam.

Incorporating request variable in django forms dynamically

In some cases, you may need to change form contents according to the user's session information. Just like I'd like to display the content for a specific DJANGO_SITE in one of the fields: categories. User should see only the categories of the site he belongs to. There should be a filtering but we can't use request data in Forms! One way is like I explained before. But this requires the form to be called with additional parameters. What if you can't modify those parameters because the view calling it is a third-party module? Then you may wrap the form in a function and create a new form definition according to the request with the session info in it:

def get_edit_profile_form(request):
    class EditProfileForm(BaseEdit):
        def __init__(self, *args, **kw):
            super(BaseEdit, self).__init__(*args, **kw)
            self.fields['categories'].queryset = Category.objects.filter(site=request.site)


def my_profile_edit(request, username):
    # do anything before performing the original profile edit
    response = userena_views.profile_edit(request, username=username, edit_profile_form=get_edit_profile_form(request))
    # do anything after performing the original profile edit.
    return response

This way we don't have to modify userena which is a third party module.

Designing Django, PHP and HTML Compatible

You may want to design a web site and sell or distribute it. Headers, navigation bars and footers of the pages will be the same for most of the pages. So you may want to re-use these fragments in each page so that when you make a change in the design, it's reflected in all of those pages. Template engines come in handy in these cases. I've been using Django for my projects. Django uses template engine similar to jinja2.

But when I wanted to create a static site, I used Hyde to compile the fragments into complete html files. But this required re-compilation of all the project whenever you make a change in one single file. Perhaps compiling only the changed is possible with some kind of console arguments. Then I thought that'd be nice if there were a hyde server like django development server which recognizes which file has changed and only compiles it and serves it on HTTP. Then I found Hyde web server. I was really excited. Then I thought if I wanted to use this design on a PHP-supporting server, what would I do? I found Twig for this purpose. Wonderful! These are excellent resources for a web designer.