tag:blogger.com,1999:blog-14197836731552842912024-03-13T20:43:34.728+03:00Mesutcan Kurt's Blog SiteMesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-1419783673155284291.post-16684871334382498132011-08-22T04:40:00.001+03:002011-08-22T04:41:18.589+03:00Python Multi-Build for Python Extensions Packaging - Report 5<meta equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="font-family: arial; font-size: small; "><div>Hi everybody,</div><div>
<br /></div><div>This is the 5th and the final report of my Google Summer of Code project for Debian, </div><div>Python Multi-Build for Python Extensions Packaging.</div><div>
<br /></div><div>During the Google's Summer of Code 2011 period, I coded a tool to packaging Python extensions which can be built by distutils, autotools, cmake and scons. I finished tool for Python3.X (py3build), its CDBS files and working on integrating it with debhelper's sequencer and the tool for Python2.X (pybuild) and its BDS/DebHelper files. I will surely continue after GSoC finished and will continue improving the tool. The tool's git repository is here[1].</div><div>
<br /></div><div>I'm a bit behind of my schedule. I finished all plugins for Python3.X except packaging module. Instead of that I worked on CDBS and DebHelper files. I finished CDBS files, also write a DebHelper but we decided to do DH file a different way. So I need to change it. I will port the tool to Python2.X.</div><div>
<br /></div><div>In git repository there is contrib/ directory to contain CDBS and DebHelper files. CDBS files are in contrib/cdbs/ directory and debhelper files are in contrib/debhelper/ directory. I put original files if they exists (with .orig suffix) and modified files. I also created a new file for CDBS, called <a href="http://python-py3build.mk.in/" target="_blank">python-py3build.mk.in</a> for handling all build systems. And modified <a href="http://python-distutils.mk.in/" target="_blank">python-distutils.mk.in</a> and <a href="http://python-autotools.mk.in/" target="_blank">pyt<wbr>hon-autotools.mk.in</a> files. All they are in git repository.</div><div>
<br /></div><div>I'm still working on DebHelper file. After that I'll work on pybuild. And surely testing all of them and if any bug or problem occurs I will fix it.</div><div>
<br /></div><div>Also, I need to improve my build system detection algorithm. At first, I thought that if there is multiple build system matches then show which are usable and then exit but I realized that in CDBS and DebHelper I need one build system. So even if there is multiple matches, it should return something. I'm also working on that.</div><div>
<br /></div><div>For now that's all. Thanks everybody. It is a pleasure to contribute to Debian and participating from Google's Summer of Code. I will be in touch with Debian Project as a contributor.</div><div>
<br /></div><div>[1] git://<a href="http://git.debian.org/~mesutcan-guest/py3multibuild.git" target="_blank">git.debian.org/~<wbr>mesutcan-guest/py3multibuild.<wbr>git</a></div></span>Mesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com1tag:blogger.com,1999:blog-1419783673155284291.post-2259131013877746072011-07-31T11:45:00.001+03:002011-07-31T11:45:57.413+03:00Python Multi-Build for Python Extensions Packaging - Report 4Hello,<br /><br />This is the fourth report of my GSoC project for Debian. It has been a long time<br />since I wrote third report. So past four weeks a lot of things are implemented.<br /><br />Firstly, I've changed tool's main script's name to py3build (removed .py suffix)<br />and make it executable so after you cloned my git repository you can use tool<br />with "./py3build".<br /><br />I fixed a bug which occurs when the tool is invoked from somewhere except<br />its directory. If you invoke it like "./path/to/py3build/py3build" then the tool give<br />errors (f.e. could not find plugins etc.). Now this is fixed.<br /><br />When I'm designing the API, I think if there is given build system (with<br />--system parameter) there is no need to invoke detect method but when<br />I'm writing autotools plugin, I realized that I need to learn which files are<br />exists and according to that information I'll invoke right thing (f.e. autoconf<br />or directly configure). So even if there is a given build system with --system<br />parameter its detect method invokes.<br /><br />When I'm implementing plugins to tool, I need clean args (f.e. parameters<br />which are using with make clean) and install args (f.e. parameters with<br />make install) and I added it.<br /><br />I added --dest-dir parameter this is passing to distutils plugin's root parameter.<br />In distutils plugin --install-dir parameter is passing as --install-platlib,<br />--install-purelib and --install-scripts so that maintainer can override it<br />with py3build's --install-args parameter.<br /><br />Now, I implemented distutils, autotools, cmake plugins and working on<br />scons and packaging plugins. During the writing period of plugins I'm<br />testing them. After I finished these plugins I'm planning to write a patch<br />to dh and cdbs. After that I will write the tool and plugins for Python2.X.Mesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com0tag:blogger.com,1999:blog-1419783673155284291.post-12457960817920076392011-07-03T22:27:00.001+03:002011-07-03T22:27:51.240+03:00Python Multi-Build for Python Extensions Packaging - Report 3<meta equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="font-family: arial; font-size: small; "><div>Hello,</div><div>
<br /></div><div>This is the third report of my GSoC project for Debian. During last two weeks </div><div>I've done most of things about the main tool. Before that weeks I have been</div><div> working on project's API. API is almost finished. Still I find missing/wrong </div><div>parts during the coding and changing the API design.</div><div>
<br /></div><div>Like I said, main tool's coding process is nearly finished. For now It is doing </div><div>its job for building Python3.X packages. You can find the source code at [1].</div><div>
<br /></div><div>I also coded an example plugin (actually only return values) and plugins/__init__.py </div><div>file which contains like Base class, exception classes and plugin listing method.</div><div>
<br /></div><div>When I started coding I realized that I need extra parameters. I've added --install-dir</div><div> parameter. You can specify the install directory if there is no install directory the tools</div><div> pretends that the install directory is "./debian/tmp". After that it checks whether if the</div><div> directory exists. If not it creates the directory.</div><div>
<br /></div><div>I've changed py3build.py a bit. First, I've thought for every process I invoke py3build's </div><div>methods but they are redundant. I've removed them. Now they are called in main </div><div>method of py3build.</div><div>
<br /></div><div>I use logging module for printing messages, errors etc.</div><div>
<br /></div><div>get_build_system() method retieves list from plugins/__init__.py file. In that file there is </div><div>scan_for_plugins() method. It finds all plugins in plugins/ directory which files match for</div><div> "[a-zA-Z].py". It keeps the list in plugins_list variable and returns that. When I'm writing </div><div>scan_for_plugins() method I've read codes (f.e. [2]) about that and decided to name all </div><div>plugins' class name as *BuildSystem*. Now, detecting method works with that way. </div><div>(all plugins' class' are called BuildSystem) And I've changed base class name to </div><div>BuildSystemBase.</div><div>
<br /></div><div>get_build_system() method is used by determine_build_system() method. If there is given </div><div>build system (by parameter) it tries to use that. If that build system is not usable then give </div><div>error and exits. If there is no given build system. It invokes get_build_system() and iterates</div><div> that methods return value which is a list. When iterating It create an instance of that build </div><div>system class (if can not create then it gives warning and skips that build system plugin) </div><div>and invokes detect method and append plugin name and detect method's return value to </div><div>the bs_list which contains plugin list which are available. And then sorts it. If there is error </div><div>while sorting list or there is no usable plugin then give error and exits. If there is multiple </div><div>highest detect values then lists that plugins and exists. (maintainer should select one of </div><div>these plugins to build extension.)</div><div>
<br /></div><div>In main method, after getting arguments, it checks for verbose mode (or 'DH_VERBOSE' </div><div>environment variable) and sets log level (logging.DEBUG or logging.WARN)</div><div>
<br /></div><div>If there is no given build_dir then it pretends it is "./".</div><div>
<br /></div><div>After that It try to determine build system. If can not then prints error and exits.</div><div>
<br /></div><div>And finally it starts to build extension. Firstly it checks if there is given version list. If not, </div><div>it uses default values and iterates build loop with that values.</div><div>
<br /></div><div>First cleans, configures, builds (if there is no --test-only arg) and tests (if there is --test-only </div><div>arg or there is not --no-tests argument)</div><div>
<br /></div><div>If there is error (raises plugin.Error) while invoking that method then prints error and exits. </div><div>
<br /></div><div>In verbose mode tool prints every process. (like "configuring for PythonX.Y" etc.)</div><div>
<br /></div><div><div>I've started writing a plugin for detecting *distutils*. For now, I only wrote its detect method. It </div><div>returns between 0-1.0 floating point value. If everything is OK it returns 1.0. Other methods does </div><div>nothing, only returns True but this week I'll finish this plugin and work for others.</div></div><div>
<br /></div><div>For detecting distutils I've used the code at [3]. First I try to import that module if not, I've copied </div><div>required methods to plugins/_packaging.py file, I use them.</div><div>
<br /></div><div>I don't use is_distutils() method directly. For give probability I invoke what it invokes and give them </div><div>a number. (f.e. _has_setup_py has 0.5, _has_pkg_info has 0.25 and _has_distutil_text has 0.25. </div><div>But if _has_setup_py returns False then detect method doesn't look other methods since they </div><div>require setup.py.)</div><div>
<br /></div><div>For now that's all. See you at next time.</div><div>
<br /></div><div><span style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; ">[1] <span style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; ">git://<a href="http://git.debian.org/~mesutcan-guest/py3multibuild.git" target="_blank" style="color: rgb(28, 81, 168); ">git.debian.org/~<wbr>mesutcan-guest/py3multibuild.<wbr>git</a></span></span>
<br />[2] <a href="http://svn.berlios.de/viewvc/griffith/trunk/lib/plugins/extensions/" target="_blank">http://svn.berlios.de/<wbr>viewvc/griffith/trunk/lib/<wbr>plugins/extensions/</a></div><div>[3] <span style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; "><a href="http://hg.python.org/cpython/file/a7e00934baf1/Lib/packaging/util.py#l1343" target="_blank" style="color: rgb(28, 81, 168); ">http://hg.python.org/<wbr>cpython/file/a7e00934baf1/Lib/</a></span><span style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; "><a href="http://hg.python.org/cpython/file/a7e00934baf1/Lib/packaging/util.py#l1343" target="_blank" style="color: rgb(28, 81, 168); "><wbr>packaging/util.py</a></span></div><div><span ><span style="border-collapse: collapse; ">
<br /></span></span></div><div>--
<br />Mesutcan Kurt
<br /></div><div>
<br /></div></span>Mesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com0tag:blogger.com,1999:blog-1419783673155284291.post-19916818271306752972011-06-19T18:03:00.001+03:002011-06-19T18:06:09.968+03:00Python Multi-Build for Python Extensions Packaging - Report 2Hello,<br /><br />This is second report of my GSoC project for Debian. Since my last<br />report I have done most of things about my project's design. Me and my<br />mentor are thought about projects API, design etc. this two weeks. Since<br />preparing API is the most important task of this project two week is not<br />a long time, and I don't say that API design is over. According to your<br />comments or requests I can change the API.<br /><br />If you want to see how the tool's API look like or tool's source code you<br />can clone project's git repository at [1]. In repository, under template/<br />directory you can see that.<br /><br />Under the directory you will see one file and one directory. Only file is<br />the tool's main file, called py3build. It is a python3 script. It will<br />build extensions for Python3.<br /><br />Firstly It will parse arguments. Arguments are:<br /><br />--system args: (extension will be build for which build system)<br />--test: (for only test the extension)<br />--no-clean: (don't try to call clean method)<br />--configure-args args: (extra configure args)<br />--build-args args: (extra build args)<br />--test-args args: (extra test args)<br />--quiet: (quiet mode)<br />--verbose: (verbose mode)<br />--python-version args: (build for that version)<br />--build-directory args: (build in given directory)<br /><br />If there is no given Build System then It will try to get Build System<br />list. This list will be generated from plugins/ directory which you can<br />find under my git repository's template/ directory.<br /><br />I want to describe what will BuildSystem's look like.<br /><br />Under plugins/ directory you will see __init__.py file. It will contain<br />base BuildSystem class, main Exception class, and other method specific<br />Exception classes. (f.e. ConfigureError) Main Exception class will be<br />derived from Exception class and other Exception classes will be derived<br />from this class, called Error.<br /><br />Base BuildSystem will be derived from all other Build System classes. It<br />has __init__ method which gives build_dir, configure_args, build_args,<br />test_args for parameter. It will set them to self.{parameter_name}. So,<br />if they will be needed from another method, can be retrieved from<br />self.{parameter_name}, no need to pass every method.<br /><br />One of the important BuildSystem class method is detect method. This<br />method will detect the extension's build system. If given directory has<br />files for building extensions for this build system (or other magical<br />detecting ways) it will return true.<br /><br />After that with configure method we can configure extension for the build<br />system. This method and other methods in build system classes' will also take version parameter. This parameter will<br />determine that this extension will be build for which Python version. If there is<br />any error occurs then it will raise ConfigureError exception.<br /><br />Clean method will cleans (f.e. removing build files) given directory for<br />that build system. If there is any error occurs then it will raise<br />CleanError.<br /><br />Build method will build extension for the build system. It will use<br />self.version to determine which version it will be built for. If there is<br />any error, then raises BuildError.<br /><br />Last method is called test. This method will test the extension for<br />self.version Python version. If there is any error occurs then raises<br />TestError.<br /><br />Every build system plugin should have one class, same name with file. The<br />class should extend BuildSystem class and override this class' methods<br />for working correctly.<br /><br />I also wrote an example plugin, called example. For show how it will be done (It is under<br />plugin/ directory under project's git repository).<br /><br />py3build's get_build_system method will retrieve a BuildSystem list. This<br />method will be used by determine_build_system method. That method will<br />determine which build system is usable for this extension. After that<br />this method also returns a list. List is in order to higher probablity to detect.<br /><br />Main method will invoke clean_for_build_system with a BuildSystem object<br />argument for clean for that BuildSystem, configure_for_build_system with<br />a BuildSystem object argument for configure for that BuildSystem,<br />make_for_build_system with a BuildSystem object argument for build for<br />that BuildSystem and test_for_build_system a BuildSystem object argument<br />for test for that BuildSystem. The call sequence is like that.<br /><br />Every method will return True if everything is OK.<br /><br />We are still working on API and this one isn't completed, and<br />discussing about various problems.<br /><br />[1] git://git.debian.org/~mesutcan-guest/py3multibuild.gitMesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com0tag:blogger.com,1999:blog-1419783673155284291.post-21024970369928130242011-06-05T15:28:00.002+03:002011-06-05T16:10:04.271+03:00Python Multi-Build for Python Extensions Packaging - Report 1Hi,<br /><br />This report is first official one for Debian Google Summer of Code. My project is creating a tool to build Python extensions for all Python versions supported by Debian. I've added my project proposal to Debian Wiki before I submitted. You can see it at [1]. I'm also writing a blog, [2]. You can see my posts from Debian Planet.<br /><br />I didn't start coding for the tool until today. This week I'll start coding. My final exams are ending at this Friday (10 June). This is why I didn't started coding. I don't want to start coding at the beginning of my exams because I can't fully concentrate on coding.<br /><br />Before writing and during writing the tool I'll decide this tool's<br />methods and what they do with my mentor Piotr and Debian-Python list<br />[3].<br /><br />For investigating what CDBS and dh sequencer are doing right now I read<br /><br />* /usr/share/cdbs/1/class/python-*<br />* /usr/share/perl5/Debian/Debhelper/Buildsystem/python_distutils.pm<br /><br />files. They give me enough information what is going on right now.<br /><br />I won't try to feel developers strange when they using this tool via<br />CDBS and/or dh sequencer.<br /><br />For first, I'll write a tool to create Python 3 extensions. I think<br />creating Python 3 extensions have a bit higher priority.<br /><br />Coding period is something like that:<br /><br />* Writing a method to detect which build system will be used.<br />* After detecting build system, check everything is sane.<br />* Start building package:<br /> * Make tests which are supported by build system. (f.e. in<br />autotools, *make check*)<br /> * If needed, compile.<br /> * Build package.<br /><br />I decided to start writing for Python distutils first. I think this one is more important than other build systems. So, after I finished building extensions using distutils I want to give it to developers who can test the tool, and if they find bugs or something should be added, I will add these things. After I finished and give developers to test tool for each one of build system, I will start writing for next build system.<br /><br />I'll use PEP8 standards for coding. I'll develop my project using GIT, my repository is under Alioth. You can clone this project's repository. For now I created only one repository for building python3 extensions [4]. Firstly It will be written in Python3. I'll write a py3to2 converter and make running this tool using >= python2.6.<br /><br />I didn't do much things, especially didn't started coding and naturally report is short, but next report will be much better (more detailed and more technical) then this one.<br /><br />[1] http://wiki.debian.org/SummerOfCode2011/PythonMultibuild/mesutcankurt<br />[2] http://mesutcaneng.blogspot.com<br />[3] http://lists.debian.org/debian-python/<br />[4] git://git.debian.org/~mesutcan-guest/py3multibuild.gitMesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com0tag:blogger.com,1999:blog-1419783673155284291.post-20616897615639276842011-05-24T10:28:00.005+03:002011-06-03T00:37:10.679+03:00Hello from Debian GNU/Linux GSoC StudentHello everyone,<div>
<br /></div><div><div>Until today (24 May), I didn't blog regularly in *English*. I think after being accepted to GSoC writing a blog is a good (hmm, great) idea. By the way, I'm already writing a blog in *Turkish* at [1] address.</div></div><div>
<br /></div><div>I'm too happy to be accepted to Google's Summer of Code. More important thing is I've been accepted to Debian GNU/Linux! This is unbelievable. Even after results are announced I can't believe that. Wonderful! I want to be more involved in Debian after GSoC.</div><div>
<br /></div><div>My proposal is at [2]. And my project is about writing a tool to build Python extensions. </div><div>
<br /></div><div>You can follow my GSoC progress from this blog or debian-python mailing list [3], I will write to this list about my project regularly.</div><div>
<br /></div><div><div>Here is some decisions we took with my mentor Piotr:</div><div>
<br /></div><div>* I will develop my project with GIT (under alioth), the address will be mentioned in my next mail.</div><div>
<br /></div><div>* I will blog[2] about my progress (an email will be send here as well) every two weeks. (maybe more often)</div><div>
<br /></div><div>* I will write the code in Python 3 and convert it to Python 2.6 (current Squeeze Python version) using Python 3 to 2 converter which I will write after I finished some important tasks on py3multibuild.</div><div>
<br /></div><div>* I will focus on building Python extensions for Python 3 then I will backport the tool to Python 2.</div><div>
<br /></div><div>This post is meant as introduction, my next one will contain more technical details.</div></div><div>
<br /></div><div>Please do not hesitate to comment to my posts. </div><div>
<br /></div><div>[1] http://mesutcank.blogspot.com</div><div>[2] http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/mesutcank/2001</div><div>[3] <a href="http://lists.debian.org/debian-python/">http://lists.debian.org/debian-python/</a></div><meta equiv="content-type" content="text/html; charset=utf-8">Mesutcanhttp://www.blogger.com/profile/14480791836738882031noreply@blogger.com0