Monday, August 22, 2011

Python Multi-Build for Python Extensions Packaging - Report 5

Hi everybody,

This is the 5th and the final report of my Google Summer of Code project for Debian,
Python Multi-Build for Python Extensions Packaging.

During the Google's Summer of Code 2011 period, I coded a tool to packaging Python extensions which can be built by distutils, autotools, cmake and scons. I finished tool for Python3.X (py3build), its CDBS files and working on integrating it with debhelper's sequencer and the tool for Python2.X (pybuild) and its BDS/DebHelper files. I will surely continue after GSoC finished and will continue improving the tool. The tool's git repository is here[1].

I'm a bit behind of my schedule. I finished all plugins for Python3.X except packaging module. Instead of that I worked on CDBS and DebHelper files. I finished CDBS files, also write a DebHelper but we decided to do DH file a different way. So I need to change it. I will port the tool to Python2.X.

In git repository there is contrib/ directory to contain CDBS and DebHelper files. CDBS files are in contrib/cdbs/ directory and debhelper files are in contrib/debhelper/ directory. I put original files if they exists (with .orig suffix) and modified files. I also created a new file for CDBS, called python-py3build.mk.in for handling all build systems. And modified python-distutils.mk.in and python-autotools.mk.in files. All they are in git repository.

I'm still working on DebHelper file. After that I'll work on pybuild. And surely testing all of them and if any bug or problem occurs I will fix it.

Also, I need to improve my build system detection algorithm. At first, I thought that if there is multiple build system matches then show which are usable and then exit but I realized that in CDBS and DebHelper I need one build system. So even if there is multiple matches, it should return something. I'm also working on that.

For now that's all. Thanks everybody. It is a pleasure to contribute to Debian and participating from Google's Summer of Code. I will be in touch with Debian Project as a contributor.

Sunday, July 31, 2011

Python Multi-Build for Python Extensions Packaging - Report 4

Hello,

This is the fourth report of my GSoC project for Debian. It has been a long time
since I wrote third report. So past four weeks a lot of things are implemented.

Firstly, I've changed tool's main script's name to py3build (removed .py suffix)
and make it executable so after you cloned my git repository you can use tool
with "./py3build".

I fixed a bug which occurs when the tool is invoked from somewhere except
its directory. If you invoke it like "./path/to/py3build/py3build" then the tool give
errors (f.e. could not find plugins etc.). Now this is fixed.

When I'm designing the API, I think if there is given build system (with
--system parameter) there is no need to invoke detect method but when
I'm writing autotools plugin, I realized that I need to learn which files are
exists and according to that information I'll invoke right thing (f.e. autoconf
or directly configure). So even if there is a given build system with --system
parameter its detect method invokes.

When I'm implementing plugins to tool, I need clean args (f.e. parameters
which are using with make clean) and install args (f.e. parameters with
make install) and I added it.

I added --dest-dir parameter this is passing to distutils plugin's root parameter.
In distutils plugin --install-dir parameter is passing as --install-platlib,
--install-purelib and --install-scripts so that maintainer can override it
with py3build's --install-args parameter.

Now, I implemented distutils, autotools, cmake plugins and working on
scons and packaging plugins. During the writing period of plugins I'm
testing them. After I finished these plugins I'm planning to write a patch
to dh and cdbs. After that I will write the tool and plugins for Python2.X.

Sunday, July 3, 2011

Python Multi-Build for Python Extensions Packaging - Report 3

Hello,

This is the third report of my GSoC project for Debian. During last two weeks
I've done most of things about the main tool. Before that weeks I have been
working on project's API. API is almost finished. Still I find missing/wrong
parts during the coding and changing the API design.

Like I said, main tool's coding process is nearly finished. For now It is doing
its job for building Python3.X packages. You can find the source code at [1].

I also coded an example plugin (actually only return values) and plugins/__init__.py
file which contains like Base class, exception classes and plugin listing method.

When I started coding I realized that I need extra parameters. I've added --install-dir
parameter. You can specify the install directory if there is no install directory the tools
pretends that the install directory is "./debian/tmp". After that it checks whether if the
directory exists. If not it creates the directory.

I've changed py3build.py a bit. First, I've thought for every process I invoke py3build's
methods but they are redundant. I've removed them. Now they are called in main
method of py3build.

I use logging module for printing messages, errors etc.

get_build_system() method retieves list from plugins/__init__.py file. In that file there is
scan_for_plugins() method. It finds all plugins in plugins/ directory which files match for
"[a-zA-Z].py". It keeps the list in plugins_list variable and returns that. When I'm writing
scan_for_plugins() method I've read codes (f.e. [2]) about that and decided to name all
plugins' class name as *BuildSystem*. Now, detecting method works with that way.
(all plugins' class' are called BuildSystem) And I've changed base class name to
BuildSystemBase.

get_build_system() method is used by determine_build_system() method. If there is given
build system (by parameter) it tries to use that. If that build system is not usable then give
error and exits. If there is no given build system. It invokes get_build_system() and iterates
that methods return value which is a list. When iterating It create an instance of that build
system class (if can not create then it gives warning and skips that build system plugin)
and invokes detect method and append plugin name and detect method's return value to
the bs_list which contains plugin list which are available. And then sorts it. If there is error
while sorting list or there is no usable plugin then give error and exits. If there is multiple
highest detect values then lists that plugins and exists. (maintainer should select one of
these plugins to build extension.)

In main method, after getting arguments, it checks for verbose mode (or 'DH_VERBOSE'
environment variable) and sets log level (logging.DEBUG or logging.WARN)

If there is no given build_dir then it pretends it is "./".

After that It try to determine build system. If can not then prints error and exits.

And finally it starts to build extension. Firstly it checks if there is given version list. If not,
it uses default values and iterates build loop with that values.

First cleans, configures, builds (if there is no --test-only arg) and tests (if there is --test-only
arg or there is not --no-tests argument)

If there is error (raises plugin.Error) while invoking that method then prints error and exits.

In verbose mode tool prints every process. (like "configuring for PythonX.Y" etc.)

I've started writing a plugin for detecting *distutils*. For now, I only wrote its detect method. It
returns between 0-1.0 floating point value. If everything is OK it returns 1.0. Other methods does
nothing, only returns True but this week I'll finish this plugin and work for others.

For detecting distutils I've used the code at [3]. First I try to import that module if not, I've copied
required methods to plugins/_packaging.py file, I use them.

I don't use is_distutils() method directly. For give probability I invoke what it invokes and give them
a number. (f.e. _has_setup_py has 0.5, _has_pkg_info has 0.25 and _has_distutil_text has 0.25.
But if _has_setup_py returns False then detect method doesn't look other methods since they
require setup.py.)

For now that's all. See you at next time.


--
Mesutcan Kurt

Sunday, June 19, 2011

Python Multi-Build for Python Extensions Packaging - Report 2

Hello,

This is second report of my GSoC project for Debian. Since my last
report I have done most of things about my project's design. Me and my
mentor are thought about projects API, design etc. this two weeks. Since
preparing API is the most important task of this project two week is not
a long time, and I don't say that API design is over. According to your
comments or requests I can change the API.

If you want to see how the tool's API look like or tool's source code you
can clone project's git repository at [1]. In repository, under template/
directory you can see that.

Under the directory you will see one file and one directory. Only file is
the tool's main file, called py3build. It is a python3 script. It will
build extensions for Python3.

Firstly It will parse arguments. Arguments are:

--system args: (extension will be build for which build system)
--test: (for only test the extension)
--no-clean: (don't try to call clean method)
--configure-args args: (extra configure args)
--build-args args: (extra build args)
--test-args args: (extra test args)
--quiet: (quiet mode)
--verbose: (verbose mode)
--python-version args: (build for that version)
--build-directory args: (build in given directory)

If there is no given Build System then It will try to get Build System
list. This list will be generated from plugins/ directory which you can
find under my git repository's template/ directory.

I want to describe what will BuildSystem's look like.

Under plugins/ directory you will see __init__.py file. It will contain
base BuildSystem class, main Exception class, and other method specific
Exception classes. (f.e. ConfigureError) Main Exception class will be
derived from Exception class and other Exception classes will be derived
from this class, called Error.

Base BuildSystem will be derived from all other Build System classes. It
has __init__ method which gives build_dir, configure_args, build_args,
test_args for parameter. It will set them to self.{parameter_name}. So,
if they will be needed from another method, can be retrieved from
self.{parameter_name}, no need to pass every method.

One of the important BuildSystem class method is detect method. This
method will detect the extension's build system. If given directory has
files for building extensions for this build system (or other magical
detecting ways) it will return true.

After that with configure method we can configure extension for the build
system. This method and other methods in build system classes' will also take version parameter. This parameter will
determine that this extension will be build for which Python version. If there is
any error occurs then it will raise ConfigureError exception.

Clean method will cleans (f.e. removing build files) given directory for
that build system. If there is any error occurs then it will raise
CleanError.

Build method will build extension for the build system. It will use
self.version to determine which version it will be built for. If there is
any error, then raises BuildError.

Last method is called test. This method will test the extension for
self.version Python version. If there is any error occurs then raises
TestError.

Every build system plugin should have one class, same name with file. The
class should extend BuildSystem class and override this class' methods
for working correctly.

I also wrote an example plugin, called example. For show how it will be done (It is under
plugin/ directory under project's git repository).

py3build's get_build_system method will retrieve a BuildSystem list. This
method will be used by determine_build_system method. That method will
determine which build system is usable for this extension. After that
this method also returns a list. List is in order to higher probablity to detect.

Main method will invoke clean_for_build_system with a BuildSystem object
argument for clean for that BuildSystem, configure_for_build_system with
a BuildSystem object argument for configure for that BuildSystem,
make_for_build_system with a BuildSystem object argument for build for
that BuildSystem and test_for_build_system a BuildSystem object argument
for test for that BuildSystem. The call sequence is like that.

Every method will return True if everything is OK.

We are still working on API and this one isn't completed, and
discussing about various problems.

[1] git://git.debian.org/~mesutcan-guest/py3multibuild.git

Sunday, June 5, 2011

Python Multi-Build for Python Extensions Packaging - Report 1

Hi,

This report is first official one for Debian Google Summer of Code. My project is creating a tool to build Python extensions for all Python versions supported by Debian. I've added my project proposal to Debian Wiki before I submitted. You can see it at [1]. I'm also writing a blog, [2]. You can see my posts from Debian Planet.

I didn't start coding for the tool until today. This week I'll start coding. My final exams are ending at this Friday (10 June). This is why I didn't started coding. I don't want to start coding at the beginning of my exams because I can't fully concentrate on coding.

Before writing and during writing the tool I'll decide this tool's
methods and what they do with my mentor Piotr and Debian-Python list
[3].

For investigating what CDBS and dh sequencer are doing right now I read

* /usr/share/cdbs/1/class/python-*
* /usr/share/perl5/Debian/Debhelper/Buildsystem/python_distutils.pm

files. They give me enough information what is going on right now.

I won't try to feel developers strange when they using this tool via
CDBS and/or dh sequencer.

For first, I'll write a tool to create Python 3 extensions. I think
creating Python 3 extensions have a bit higher priority.

Coding period is something like that:

* Writing a method to detect which build system will be used.
* After detecting build system, check everything is sane.
* Start building package:
* Make tests which are supported by build system. (f.e. in
autotools, *make check*)
* If needed, compile.
* Build package.

I decided to start writing for Python distutils first. I think this one is more important than other build systems. So, after I finished building extensions using distutils I want to give it to developers who can test the tool, and if they find bugs or something should be added, I will add these things. After I finished and give developers to test tool for each one of build system, I will start writing for next build system.

I'll use PEP8 standards for coding. I'll develop my project using GIT, my repository is under Alioth. You can clone this project's repository. For now I created only one repository for building python3 extensions [4]. Firstly It will be written in Python3. I'll write a py3to2 converter and make running this tool using >= python2.6.

I didn't do much things, especially didn't started coding and naturally report is short, but next report will be much better (more detailed and more technical) then this one.

[1] http://wiki.debian.org/SummerOfCode2011/PythonMultibuild/mesutcankurt
[2] http://mesutcaneng.blogspot.com
[3] http://lists.debian.org/debian-python/
[4] git://git.debian.org/~mesutcan-guest/py3multibuild.git

Tuesday, May 24, 2011

Hello from Debian GNU/Linux GSoC Student

Hello everyone,

Until today (24 May), I didn't blog regularly in *English*. I think after being accepted to GSoC writing a blog is a good (hmm, great) idea. By the way, I'm already writing a blog in *Turkish* at [1] address.

I'm too happy to be accepted to Google's Summer of Code. More important thing is I've been accepted to Debian GNU/Linux! This is unbelievable. Even after results are announced I can't believe that. Wonderful! I want to be more involved in Debian after GSoC.

My proposal is at [2]. And my project is about writing a tool to build Python extensions.

You can follow my GSoC progress from this blog or debian-python mailing list [3], I will write to this list about my project regularly.

Here is some decisions we took with my mentor Piotr:

* I will develop my project with GIT (under alioth), the address will be mentioned in my next mail.

* I will blog[2] about my progress (an email will be send here as well) every two weeks. (maybe more often)

* I will write the code in Python 3 and convert it to Python 2.6 (current Squeeze Python version) using Python 3 to 2 converter which I will write after I finished some important tasks on py3multibuild.

* I will focus on building Python extensions for Python 3 then I will backport the tool to Python 2.

This post is meant as introduction, my next one will contain more technical details.

Please do not hesitate to comment to my posts.

[1] http://mesutcank.blogspot.com
[2] http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/mesutcank/2001