Planet Bugzilla

I got an email this morning from the maintainer of Testopia to inform me that Testopia 2.5 is finally released! This is the first version to work with Bugzilla 4.2.1 (if you are still running Bugzilla 4.0.x or 4.2, upgrade to 4.2.1 first). The announcement has not be made official yet, but this should be done in the coming hours, hopefully. Meanwhile, you can already download Testopia 2.5. Note that there is no need to apply any patch anymore to make it work (which is why you need Bugzilla 4.2.1 instead of e.g. 4.2, because 4.2.1 has some changes included in it to avoid to patch the core code to make Testopia work). And if you find any problem which hasn’t been reported yet, feel free to file a new bug (bugs only, no support questions). Have fun!

UPDATE: The announcement has finally been made here.

May 30, 2012 01:05 PM

During the week-end, I received a pertinent email from a former Bugzilla developer who replied to an email I sent to all reviewers about the pretty low activity in the Bugzilla project during the current development cycle. He argued that one of the reasons which made him go away, and which probably took some other contributors away as well, is our high code quality standards we have in this project. The point is that we deny review if submitted patches do not follow our guidelines or are poorly written compared to what we expect in our codebase. He suggested that reviewers should accept and commit lower quality patches, and file follow-up bugs to clean up the new code. I then brought this discussion on IRC with other core developers, and we realized how hard it is to define the right level of code quality. The problems are:

• Some new features require some rewrite of some parts of the code to be easily extendable in the future and/or to reuse some existing code. These features could also be implemented without the code rewrite, which is easier to do for the contributor, but then would be harder to maintain, or would require to duplicate some existing code, which we try to avoid as much as possible. This is how Bugzilla 2.x was written, and it took us two full development cycles (Bugzilla 2.22 + 3.0) to clean up and rewrite the codebase to be able to implement new features in a reasonable way (the 2.x codebase became too ugly and complex to easily implement anything new on top of it). So the risk to accept such patches is that the codebase would become more messy again. But on the other hand, the risk is also to make contributors leave if we keep our high standards, because a rewrite is generally not something trivial nor exciting, and contributors generally do not understand why we reject a patch which does the job.
• Accepting patches which are of lower quality and filing follow-up bugs to fix the new code before the next release seems reasonable only if the community around the project is large enough to have the manpower to do the job in the expected timeframe. The current Bugzilla team is very small, maybe 4-5 core developers + some occasional contributors helping with some not too hard bugs. This isn’t large enough to expect these follow-up bugs to be fixed promptly. But we also cannot release a new version with poor code in it, some of the reasons being security implications, performance issues, or possible regressions in some use cases. As the community is small, fixing these bugs would delay the next major release a lot, which is not a desirable effect.
• Depending on how the submitted patch is written, it would take twice the time to review + rewrite the new code to match our standards compared to the time it would take for a core developer to write the patch himself. The lack of time being the main enemy, I personally wouldn’t want to spend more time on such or such feature because it’s not written as we would like it to be. The time spent to rewrite bad code means less time to work on other stuff.

So what’s the right threshold to not make new and occasional contributors go away without badly impacting our codebase? Which rules do other Mozilla and non-Mozilla projects use to solve this problem? Please share your experience with us.

May 15, 2012 05:56 PM

We released Bugzilla 4.3.1, 4.2.1, 4.0.6 and 3.6.9 a few hours ago, and they all contain 2 security fixes. All installations are highly encouraged to upgrade to these new releases. Bugzilla 3.6.9 and 4.0.6 only contain the two security fixes mentioned in the security advisory. Bugzilla 4.2.1 also contains a pretty large list of bug fixes, which makes it a good candidate to upgrade to the 4.2.x series if you didn’t upgrade to 4.2 yet. Note that Bugzilla 4.2.1 is also the first release to work with Testopia without needing to be patched first. A new release of Testopia should be available soon, which will take advantage of the improvements and new hooks available in 4.2.1.

We also released Bugzilla 4.3.1 which is the first development snapshot of the 4.4 series. I’m going to list some of the new features and improvements available:

• A new user preference lets you decide if you should be CC’ed when someone requests something from you via a flag. For instance, requesting someone for review would automatically CC the reviewer to the bug.
• Unset flags are now hidden by default, in bug reports.
• Duplicates are now listed at the top of the bug report, close to the dependency lists.
• URLs pointing to GitHub (github.com) are now accepted in the See Also field in bug reports.
• Personal tags can now be displayed in buglists. In Bugzilla 4.3.2, you will also be able to tag bugs from the bug reports themselves. This means the ugly box at the very bottom of every page will go away.
• You can now hide the description displayed at the top of buglists containing all the arguments passed to the query.
• You can now use pronouns (%reporter%, %user%, etc.) with the flag setter and requestee fields in boolean charts.
• A new email preference has been added to get emails when the product or the component of a bug changes.
• Bugzilla now keeps track of the last time a user used his account to access Bugzilla. This will let admins know which accounts are inactive.
• Bugzilla now supports SSL connections to SMTP servers.
• Target milestones can now be 64 characters long (instead of 20) to match the same limit as versions.
• Whine emails now correctly take the sort order of buglists into account.
• Several new WebServices methods: User.update, Product.update, Group.update, Bug.update_tags, Bugzilla.parameters (read-only), Bugzilla.last_audit_time.
• Several existing WebServices methods have been improved to return more data.
• The documentation is now generated using xmlto instead of jade.
• Several performance improvements have been made when viewing large bug reports.
• The Bugzilla source code now uses the MPL2.0 license, instead of the MPL1.1 one.

We are one month away from the freezing date for new features in Bugzilla 4.4. So if some of you really want something for 4.4, you have exactly one month left to submit patches, or find a kind developer to write the patch for you. Also note that Bugzilla 4.4 will be the last release to support Perl 5.8.x. The next major release, Bugzilla 4.6, will require Perl 5.10.1 or better.

April 19, 2012 01:10 AM

You probably noticed that there has been no activity related to the development of Testopia, a Bugzilla extension, for more than a year. The reason is that its maintainer, who was the single contributor to this project, had a new job and has no time to work on it anymore. Consequently, the latest version of this extension, Testopia 2.4, which was released in October 2010 (!), only works with Bugzilla 3.6, but not 4.x.

You will be happy to hear that with the help of some new contributors wanting to make Testopia work with Bugzilla 4.2, I committed several patches to the bzr repository which make it work pretty decently with Bugzilla 4.2.1 (probably also with Bugzilla 4.0.x, but I didn’t test). There is no official release yet (it should be named Testopia 2.5), but you can download updated files using bzr. Make sure to revert changes made to the core code of Bugzilla first before applying the new code.

To make things clear: I’m not the new maintainer of this extension, and I have no plan to take this role. I’m busy enough with my role as assistant project lead + reviewer + QA lead + bug triager + patch writer for the Bugzilla project. I simply decided to help the new contributors who jumped in these last few days and used my commit access to the Testopia repository to commit some patches. The Testopia maintainer is still alive, and emailed me this morning. So he is still the one who will take the decision to release 2.5 when it’s ready.

Update: Starting with Bugzilla 4.2.1, you no longer need to patch the source code of Bugzilla to make it work with Testopia 2.5! If you are upgrading from Testopia 2.4 or older, make sure to revert the changes made to the source code first.

April 09, 2012 05:58 PM

This is just a quick note to let you know that once we branch for Bugzilla 4.4 in May, I will commit a patch which will make Bugzilla 4.5 and newer to require Perl 5.10.1 as a minimum. This means Bugzilla won’t support Perl 5.8.x anymore. So Bugzilla 4.4 will be the last release to support the oldish Perl 5.8.x. This new requirement is tracked in bug 655477.

March 29, 2012 09:24 PM

There has been some complains these last few days on IRC and in the support mailing-list/newsgroup that admins couldn’t upgrade their Bugzilla installation to 4.2 due to the lack of a CVS mirror for this branch. As announced 3 years ago in the developers mailing-list and on b.m.o., and 1.5 years ago on the bugzilla.org website, the Bugzilla team switched from the old CVS to the more modern Bazaar (or bzr for short) VCS. If you use our tarballs to download Bugzilla, then you don’t really care about this change, and the process to upgrade won’t change for you. If you use CVS and you wonder how to upgrade using Bzr, here is how you can do it:

1. For now, there is no need to shut down Bugzilla. We will do this when we start the upgrade process itself. If you made changes to the Bugzilla code itself (instead of using its extension system), you will have to port these changes to the new version, else they will be lost. If you made no changes, or all changes are contained into extensions, you can jump to step 2 directly. To generate a patch with all the changes you made, go into your bugzilla/ directory and run this command:
   cvs diff -puN > patch.diff
   The file patch.diff will contain all the changes you made to your current installation.
2. Let’s download the new version of Bugzilla into a separate directory, to not mess with the current installation. Now you will need bzr. All Linux installations have it; have a look at your package manager. For instance, Fedora 16 has bzr-2.4.2-1.fc16.i686.rpm. On Windows, you can download it from canonical.com. The standalone application is recommended; for instance: bzr-2.4.2-1-setup.exe. Once bzr is installed, run this command to download Bugzilla 4.2:
   bzr co bzr://bzr.mozilla.org/bugzilla/4.2 bugzilla42
   Here, bzr co means “checkout”, i.e. it’s the first download of 4.2 with bzr. Then you have the URL to our Bugzilla 4.2 repository, and finally you have the name of the local directory into which to dowload the source code. If you don’t like the name bugzilla42, feel free to choose another one.
3. Now go into the new bugzilla42/ directory and run the following command:
   ./checksetup.pl –check-modules
   The –check-modules part is important for two reasons. First of all, we didn’t copy the configuration files from the old bugzilla/ directory into the new one. The advantage to do so is to prevent Bugzilla 4.2 from finding where your DB is located and start interacting with it. Remember that we did no backup of your DB yet! And the second reason is that we first want to make sure that we have all the required Perl modules installed. Between major releases of Bugzilla, the requirements may change, either by requiring new Perl modules, or by requiring a newer version of an existing Perl module. If you have missing or too old Perl modules, you will have to install or upgrade them, ideally using your package manager. On Windows, ActivePerl 5.12 and newer have all the required modules available, so that’s not a problem. On Linux, only old distros may have some missing or too old Perl modules. If this happens, you can use the install-module.pl script which is in the same directory as checksetup.pl to install them. The commands you need to execute are given by checksetup.pl –check-modules itself. So open your eyes. Note that you don’t need to install all the optional Perl modules. The reason is that they are…. optional! Only install those which are relevant to your installation.
4. If you made changes to your current Bugzilla installation, then you have to apply patch.diff you created at step 1 to your new installation. If you made no changes, you can jump to step 5 directly. If you are on Windows and you don’t have patch.exe, you can download it from sourceforge.net. Once downloaded, you must copy patch.exe into the Windows directory. At this point, it’s very likely that your changes will conflict with the new code, unless you are lucky or your changes are very minor. So we first check if there are conflicts or not. To do that, copy patch.diff into your new bugzilla42/ directory, and run this command from here:
   patch -p0 –dry-run < patch.diff
   –dry-run means that we made no changes to files; it was only a test. If all you get are messages such as “Hunk #1 succeeded at 79 (offset -26 lines)” or “Hunk #1 succeeded at 31 with fuzz 1 (offset 2 lines)”, then you are fine. But if you get messages such as “Hunk #1 FAILED at 27″ and “1 out of 2 hunks FAILED”, then you are in trouble. And don’t blame the Bugzilla team and Bzr, you would get the same errors with CVS! If the conflicts are minor, you can easily fix them manually, else you probably will have to rewrite your changes directly into the new code. If you have no conflicts, you can drop the –dry-run argument and apply your patch for real:
   patch -p0 < patch.diff
5. Now we are ready to start the upgrade process itself. The first thing to do is to shut down Bugzilla, because the DB must not be accessed by anyone but you during the upgrade process. To do that, go to Administration > Parameters > General > shutdownhtml, and add some explanation of what’s happening. For instance: “This installation is being upgraded to Bugzilla 4.2. The downtime should last approximatively 2 hours.”, and click the “Save Changes” button at the bottom of the page. I recommend that you don’t leave this page during the upgrade process, because all other pages will be deactivated besides this one. At this point, when someone tries to access Bugzilla during the downtime, this message will be displayed to them instead, so that you can upgrade your installation without having some users still interacting with it.
6. Then, and this rule applies all the time when upgrading to a newer major version: do a backup of your database! The reason is that once you start the upgrade process (i.e. when you run checksetup.pl), there is NO way to downgrade later. So if the upgrade process fails for some reason (most of the time because someone hacked the source code or the DB schema) and you made no backup, you are lost. To backup your DB with MySQL, run the following command, where you must replace $db_user and $db_name by their value set in your bugzilla/localconfig file (by default: $db_user = bugs; $db_name = bugs):
   mysqldump -u $db_user -p $db_name > db_backup.sql
7. Copy the whole data/ directory and the localconfig file from your old bugzilla/ directory into the new bugzilla42/ one. data/ contains your parameters in the data/params file, your local attachments in data/attachments/ as well as data for Old Charts in data/mining/. And localconfig contains all the parameters to access the DB, including its password, the name of the web server group, and some other useful configuration information. If you wrote some extensions, now is a good time to also copy them into bugzilla42/extensions/. Only copy the ones you wrote, not the existing ones such as BmpConvert, Example, OldBugMove or Voting, which are maintained by the Bugzilla team.
8. It’s now time to upgrade your DB to work with Bugzilla 4.2. From the bugzilla42/ directory, run ./checksetup.pl with no arguments (so no –check-modules anymore) and let it run the upgrade process for you. As you copied the configuration and parameters files, it knows where the DB is, its password, and everything else it should know about. Depending on the size of your DB and from which version you are upgrading, this may take from a few minutes to several hours. But typically, it’s of the order of a few tens of minutes for a DB with several tens of thousands of bugs. If everything went well, the last message displayed by checksetup.pl must be “checksetup.pl complete” displayed in green. Else this means something went wrong at some point. If half of the messages are in red, then there is no doubt about this.
9. The upgrade is now complete, and it’s time to let your users admire this new version. Replace your old bugzilla/ directory by the new one, and re-enable Bugzilla. To do that, you must remove the text you wrote for the shutdownhtml parameter. As we replaced the old directory by the new one, the URL pointing to Bugzilla remains unchanged. You now have a fully-working bzr-based Bugzilla installation.

March 02, 2012 03:04 PM

Now that Bugzilla 4.2 is released, I could finally focus on enhancements instead of fixing blockers and doing QA. This week, I decided to spend some time to improve the performance of Bugzilla. My main focus was show_bug.cgi, i.e. bug reports. I plan to look at other CGI scripts soon, hopefully within 2 weeks.

You will be happy to hear that in the last 3 days, I managed to divide the time spent to load large bug reports such as bmo bug 38862 (235 comments, 32 attachments and 55 CC’ed users) or bmo bug 18574 (760 comments, 63 attachments and 170 CC’ed users!!) by a factor of 2 when using mod_cgi (-50%), and by 30% with mod_perl. The exact load time in seconds depends on the bugs being viewed and on your hardware, but these percentages seem pretty consistent with the different tests done this week. All my patches have been checked in upstream (see rev. 8133 – 8142), and Bugzilla 4.3.1 will be the first release to benefit from them all. Two of them have been backported to 4.2.1 as they are well contained and fix obvious problems, and the other ones are either too invasive for a stable branch or have unclear benefits (the perf win varies between a few 1/1oth of second to 2-3 seconds depending on the test installation).

Now talking about bmo specifically, I gave a look at the InlineHistory extension for the first time, and I proposed a patch which highly decreases the load time of bug reports. dkl did some testing for me with and without mod_perl, and he found these results:

for bug 38862: 5.18 s (unpatched) -> 3.01 s (patched)

for bug 18574 + mod_cgi: 8.01 s (unpatched) -> 4.57 s (patched)

for bug 18574 + mod_perl: 5.76 s (unpatched) -> 4.06 s (patched)

Now I hope the same gain will be visible once these patches are applied to bmo, but the hardware + software configurations of bmo are so different from our test environments that it’s hard to say for sure till the changes are committed and pushed to production. Fingers crossed!

March 02, 2012 12:49 AM

We released Bugzilla 4.2 today, exactly one year after our previous major release, 4.0! Bugzilla 4.2 now supports SQLite, lets you create attachments simply by pasting text into a text field, can send bug changes notifications in HTML format, supports more complex queries, lets you disable old target milestones, versions and components (so that you don’t need to delete them, but also don’t let users report new bugs to them), has accessibility improvements, and much more…

This release also means that Bugzilla 3.4.x is no longer supported. Installations still running 3.4.14 or older are highly encouraged to upgrade to 4.2, especially to benefit from the security improvements made in newer versions. This also means that Bugzilla 4.0.x will now only get security fixes, and other bug fixes won’t be accepted on this branch anymore, unless they fix critical flaws, such as upgrade issues or dataloss.

The Bugzilla team will now focus on the next major release, Bugzilla 4.4, which we expect to release before the end of the year. We expect to release the first development snapshot (4.3.1) in a few weeks. New features will be accepted for the next two months, till the end of April. Then we will focus on stabilization to prepare Bugzilla 4.4rc1.

If you are interested in helping with the development of Bugzilla, now is a good time to join the team and contribute with new features and/or bug fixes. Due to other activities and because life can sometimes make you very busy, some core developers had to stop their contributions to the Bugzilla project in the last few months and so we would be very happy to see new faces. Bugzilla needs to be faster, nicer, more user friendly, and all this is only possible with your help, your ideas and your feedback. So even if you aren’t a Perl expert, there is a lot of place for everyone (you  can do a lot with HTML + JS + CSS only, think about the User Interface!). If you are not sure about how to contribute or help, feel free to join us on IRC in the #bugzilla channel. There is always someone around to answer your questions.

February 22, 2012 11:50 PM

We released Bugzilla 4.2rc2, 4.0.4, 3.6.8 and 3.4.14 a few minutes ago. They all contain various security fixes which are described in the Security Advisory. 4.2rc2 should be our last release candidate before 4.2 final, which we expect to release in the 2nd half of February. On the other end, 3.4.14 is very likely our last release for the 3.4 branch. Once 4.2 final is released, we won’t support 3.4 any longer. This means that admins still running 3.4.x or older are highly encouraged to upgrade. Users should pester their admins to upgrade if they don’t do it themselves.

Now is a good time to explain (again) why upgrading is not only about getting new features and bug fixes, but also to keep your installation secure. Below are some security fixes and/or enhancements made to various releases:

X-XSS-Protection header

Since Bugzilla 4.4, the X-XSS-Protection header is used to block simple XSS attacks.

CSRF vulnerabilities in attachment.cgi and post_bug.cgi

Till recently, no token check was done before accepting new bug submissions or before uploading an attachment to an existing bug. The rationale behind this was that in older versions of Bugzilla there was no easy way to do it from the WebServices API, and we didn’t want to break existing 3rd-party tools which were legitimately interacting with Bugzilla. As the lack of token validation could be used by attackers to submit unwanted new bugs or attachments, it has been decided that a token was required in these cases too, and not only when updating a bug or an attachment. But to not break 3rd-party tools, these token checks have been implemented in Bugzilla 4.2 only, meaning that Bugzilla 4.0 and older are still vulnerable to these attacks. If you want your installation to be protected against this kind of vulnerabilities, upgrade to 4.2!

Configurable password complexity for user accounts

Bugzilla 4.2 has a new parameter which lets admins decide how complex a password must be to be accepted by Bugzilla. Up to 4.0, Bugzilla accepted all passwords which were long enough (min 6 characters by default). Now you can enforce the complexity: uppercase + lowercase characters, letters + numbers, etc… If you want this feature, upgrade to 4.2!

Clickjacking in attachments with the text/html MIME type

As Bugzilla accepts all attachments independently of their MIME type, it was possible to attach HTML files which could try to abuse users using a method known as clickjacking. To prevent this, the “Details” page of attachments now display the source code of these HTML files instead of rendering them. This security enhancement has been implemented in Bugzilla 4.0.4 and newer (including 4.2). If you want it, upgrade!

X-Frame-Options: sameorigin header

Since Bugzilla 4.0, the X-Frame-Options: SAMEORIGIN header is sent for all pages (besides attachments when delivered from their alternate host). This prevents to load a Bugzilla page from within a frame outside Bugzilla itself. This, combined with the clickjacking protection above, prevents an attacker to create an HTML page with malicious code in it to force a user to make undesired changes to Bugzilla. If you want this, upgrade!

Strict-Transport-Security (STS) header

Since Bugzilla 4.0, a new parameter lets admins enable the Strict-Transport-Security header to force the communication between Bugzilla and the user to be made over SSL only. This prevents data to be sent unencrypted.

X-Content-Type-Options: nosniff header

To prevent Internet Explorer 8 and newer from sniffing the content of attachments, Bugzilla 4.0 and newer now pass the X-Content-Type-Options: nosniff header to avoid some malicious attachments to be rendered as HTML files.

Lockout policy to prevent brute force

Since Bugzilla 3.6, if you try to guess someone else’s password and you fail 5 consecutive times, your IP is blocked for the next 30 minutes. If you still run Bugzilla 3.4 or older, Bugzilla will accept all your attempts to crack the victim’s password, severely increasing the risk that the attacker manages to do it.

And much more…

Those of you who are still running very old versions of Bugzilla (older than 3.2) don’t have their login cookies protected against JavaScript as they don’t have the HttpOnly attribute set. This means a malicious HTML page could steal your login cookies very easily and they could then be used to impersonate your user account. These same cookies also do not have the Secure attribute set which prevents them to be transmitted on unsecure connections. And last but not least, tokens used by Bugzilla 3.2.9 and older are not generated using a cryptographically secure pseudorandom number generator. This means that with some effort, your installation could be abused by an attacker who managed to guess how tokens are generated in your installation.

Now I think you know enough about security features implemented in Bugzilla to decide what you want to do. If security matters to you, you should upgrade to at least Bugzilla 4.0.4, and seriously plan to upgrade to 4.2 once it’s released later this month.