FreshPorts News

News and announcements regarding FreshPorts, The Place For Ports.

Dan Langille

spacer

Note to self: dev ain’t working

 Development  No Responses »
Apr 062015
 

I noticed a few days ago: dev.freshports.org/ is out of date with respect to vuxml.

That’s because the input processing for that website takes place in one jail, and the website (the output) takes places in another jail. There is no way for one to tell the other that something is ready for processing. This is a design issue.

Through the magic of jails, I plan to move dev.freshports.org, or at least, the website that the public sees, to another website.

My current home lab is configured with Nginx sitting behind pfSense, and in front of a few instances of Nginx and Apache. This allows me to put my webservers wherever I want, and tie them all together with Nginx as a proxy.

This move-around-of-stuff may afford me an opportunity to configure a FreshPorts installation via Ansible. I may not get to this until June or July, because of BSDCan and PGCon.

Of course, what may delay me entirely may be the naming of the new jail. Or, I could just move it all into the jail I’m now using for processing the input. That would be cleaner.

 Posted by Dan Langille at 2:21 pm

Bad portrevision

 Bug fixes  No Responses »
Jan 192015
 

mandree reported a problem with security/openvpn-auth-ldap.

spacer

Bad broken & revision

Looking in the database:

Looking at the database, the webpage is displaying what is there:

[sourcecode]
freshports.org=# select id, last_commit_id, name, version, revision, broken from ports_active where name = ‘openvpn-auth-ldap';
id | last_commit_id | name | version | revision | broken
——-+—————-+——————-+—————+———-+—————————————————————————————————–
21504 | 557197 | openvpn-auth-ldap | 2.0.4.0.s1379 | 0 | Untested SVN-based port, report success on https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=190497
(1 row)
[/sourcecode]

Let’s looking at that commit in the database:

[sourcecode]
freshports.org=# select id, message_id, message_subject, date_added from commit_log where id = 557197;
id | message_id | message_subject | date_added
——–+———————————————+————————————————————-+——————————-
557197 | 201501192036.t0JKaMNA032759@svn.freebsd.org | svn commit: r377473 – head/security/openvpn-auth-ldap/files | 2015-01-19 20:42:15.216586+00
(1 row)
[/sourcecode]

OK, that’s not the commit which changed the BROKEN flag. Let’s look at that commit:

[sourcecode]
freshports.org=# select id, message_id, message_subject, date_added from commit_log where message_id = ‘201501192037.t0JKbXZO033083@svn.freebsd.org';
id | message_id | message_subject | date_added
——–+———————————————+——————————————————-+——————————-
557194 | 201501192037.t0JKbXZO033083@svn.freebsd.org | svn commit: r377475 – head/security/openvpn-auth-ldap | 2015-01-19 20:41:18.731784+00
(1 row)
[/sourcecode]

The the date_added fields? These commits were processed out of order. No doubt, they arrived out of order because of today’s DDoS.

As you can see, we record the date the commit was processed (date_added) and we also store the svn_revision. It would be fairly easy to locate such commits which were processed out of order and then replay them.

 Posted by Dan Langille at 4:54 pm

bug with Fetch, Patch, and Extract dependencies

 Bug fixes  1 Response »
Jan 032015
 

Today I noticed a bug with the recently added Fetch, Patch, and Extract dependencies. The are correctly displayed on the port page, but not on the dependent port page.

The problem

If you look at sysutils/libchk, you will see lang/ruby20 listed under Extract dependencies. However, at lang/ruby20, there is no mention of Patch, Fetch, or Extract under This port is required by. These three entries are missing from all pages.

On the FreshPorts dev server, I have fixed the problem.

If you look at dev for lang/ruby19, you will see the entries for sysutils/libchk under both Extract and for Patch.

The causes

The problem is caused by two issues:

  1. not loading the port_dependencies table with values for Fetch, Patch, and Extract
  2. not displaying the Fetch, Patch, and Extract dependencies on the web page

In the next section, I’ll show you the patches for the above problems.

loading port_dependencies

The loading of the port_dependencies table occurs during the commit processing. As each commit arrives at the FreshPorts server, the working copy of the ports tree is updated via subversion, and the various values are extracted via make -V.

In this case, the problem was one of inconsistent field names, which led to incorrect code.

You will notice that while I have corrected the field names, they are still inconsistent with other fields. When I first added the Build, Run, and Library depends, I put the depends_ prefix on each field. I like that approach, but the field names are not the same as in the ports tree, where they are known as BUILD_DEPENDS, RUN_DEPENDS, and LIB_DEPENDS, respectively.

When I added the new fields, I used the same name in FreshPorts as used in the ports tree. However, when it came to the following section of code, I fell back to my old ways.

The following patch fixes that.

[sourcecode]
$ svn di
Index: port.pm
===================================================================
— port.pm (revision 4699)
+++ port.pm (working copy)
@@ -1118,9 +1118,9 @@
$this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_build} ), ‘B’ ); # build
$this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_run} ), ‘R’ ); # runtime
$this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_lib} ), ‘L’ ); # library
– $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_fetch} ), ‘F’ ); # fetch
– $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_extract} ), ‘E’ ); # extract
– $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{depends_patch} ), ‘P’ ); # patch
+ $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{fetch_depends} ), ‘F’ ); # fetch
+ $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{extract_depends} ), ‘E’ ); # extract
+ $this->update_depends_helper( $CommitBranch, $this->depends_stripper( $this->{patch_depends} ), ‘P’ ); # patch
}

sub depends_type_long {
[/sourcecode]

Displaying dependencies

The next bug was an act of omission. I neglected to tell the front end that there we new types of values to display. The patch for this bug is:

[sourcecode]
$ svn di
Index: port-display.php
===================================================================
— port-display.php (revision 4700)
+++ port-display.php (working copy)
@@ -490,8 +490,9 @@
$HTML .= freshports_depends_links($this->db, $port->extract_depends);
$HTML .= "\n</ol>\n";
}
+
+ # XXX when adding new depends above, be sure to update the array in ShowDependencies()

– $HTML .= $this->ShowDependencies( $port );
$HTML .= $this->ShowDependencies( $port );
}

@@ -599,7 +601,7 @@
$HTML = ”;

$PortDependencies = new PortDependencies( $this->db );
– $Types = array( ‘B’ => ‘Build’, ‘L’ => ‘Libraries’, ‘R’ => ‘Run’ );
+ $Types = array( ‘B’ => ‘Build’, ‘E’ => ‘Extract’, ‘F’ => ‘Fetch’, ‘L’ => ‘Libraries’, ‘P’ => ‘Patch’, ‘R’ => ‘Run’ );
foreach ( $Types as $type => $title )
{
$NumRows = $PortDependencies->FetchInitialise( $port->id, $type );
[/sourcecode]

That comment at the top is to remind me the next time I add depend links of a new type, there is an array below which needs updating. The first code segment outputs the dependencies for the port you are viewing. The second code segments outputs the This port is required by section of each port page.

Dev now being updated

As I type this, the set-patch-extract-fetch-depends-uses.pl script is running. That should finish by about 0100 UTC on 4 January.

I will post to Twitter when that update is completed.

 Posted by Dan Langille at 2:25 pm

Fetch, Patch, & Extract dependencies and Uses

 Branches, Bug fixes, New ideas  1 Response »
Dec 242014
 

FreshPorts is not magic. It grabs the stuff it needs by running make -V with a number of parameters. New fields are added from time to time, usually at the prompting of a user. Recently, the following fields were added to the dev website:

  1. FETCH_DEPENDS
  2. PATCH_DEPENDS
  3. EXTRACT_DEPENDS
  4. USES

More information on these fields can be found in the Dependencies section of the FreeBSD Porter’s Handbook.

Other recent changes, already in production include:

  1. Find issues related to this port
  2. Report an issue related to this port
  3. The format used to display dependencies has been changed to show the file/pkg in question (e.g. python2) and the port/pkg it is supplied from (e.g. lang/python2)

The rest of this post will outline some of the changes made to dev, and a CATEGORIES bug discovered during the database population phase of these changes.

Broken CATEGORIES

While testing the above changes, I noticed 24 ports had a NULL value for the categories column of the ports table:

[sourcecode]
freshports.org=# select name, category, element_pathname(element_id) from ports_active where categories is null;
name | category | element_pathname
—————————————+————+————————————————————————
rubygem-inflecto | devel | /ports/head/devel/rubygem-inflecto
py-wcwidth | devel | /ports/head/devel/py-wcwidth
py-Products.GenericSetup | devel | /ports/branches/RELENG_9_1_0/devel/py-Products.GenericSetup
py-eggtestinfo | devel | /ports/branches/RELENG_9_1_0/devel/py-eggtestinfo
rubygem-request_store | devel | /ports/head/devel/rubygem-request_store
calligra | editors | /ports/branches/RELENG_9_1_0/editors/calligra
xzx | emulators | /ports/branches/RELENG_9_1_0/emulators/xzx
gexiv2 | graphics | /ports/branches/RELENG_9_1_0/graphics/gexiv2
gource | graphics | /ports/branches/RELENG_9_1_0/graphics/gource
squirrelmail | mail | /ports/branches/RELENG_9_1_0/mail/squirrelmail
devede | multimedia | /ports/branches/RELENG_9_1_0/multimedia/devede
jstrack | science | /ports/branches/RELENG_9_1_0/science/jstrack
bro | security | /ports/branches/RELENG_9_1_0/security/bro
php53-openssl | security | /ports/branches/RELENG_9_2_0/security/php53-openssl
createrepo | sysutils | /ports/branches/RELENG_9_1_0/sysutils/createrepo
p5-Corona | www | /ports/branches/RELENG_9_1_0/www/p5-Corona
p5-POE-Component-Server-PSGI | www | /ports/branches/RELENG_9_1_0/www/p5-POE-Component-Server-PSGI
p5-Plack-Handler-AnyEvent-ReverseHTTP | www | /ports/branches/RELENG_9_1_0/www/p5-Plack-Handler-AnyEvent-ReverseHTTP
py-requests-toolbelt | www | /ports/head/www/py-requests-toolbelt
py-rhodecode | www | /ports/branches/RELENG_9_1_0/www/py-rhodecode
py-satchmo | www | /ports/branches/RELENG_9_1_0/www/py-satchmo
razorback-dispatcher | security | /ports/branches/RELENG_9_1_0/security/razorback-dispatcher
rtorrent-devel | net-p2p | /ports/branches/RELENG_9_1_0/net-p2p/rtorrent-devel
php53 | lang | /ports/branches/RELENG_9_2_0/lang/php53
(24 rows)

freshports.org=#
[/sourcecode]

These errors came to light when running a script to populate the newly created, and initially empty, fields: fetch_depends, extract_depends, patch_depends, and uses. Those errors were:

[sourcecode]
Use of uninitialized value in split at ..//port.pm line 861.
Use of uninitialized value in concatenation (.) or string at ..//port.pm line 874.
Use of uninitialized value in substitution (s///) at ..//port.pm line 824.
[/sourcecode]

It took me a while to track it down using the log file created during the processing of each commit. This bug was introduced during the coding and is not present in production. This sounded much more interesting while I was writing up, but I didn’t know then that the bug was so recent and isolated.

You can see the issue here:

[sourcecode]

— port.pm (revision 4692)
+++ port.pm (working copy)
@@ -821,6 +821,7 @@
# convert all whitespace to a single space
# This arose from 200609130717.k8D7HpNc057638@repoman.freebsd.org
#
+ $this->{categories} = $categories;
$this->{categories} =~ s/\s+/ /g;

$result = $this->_Validate();
[/sourcecode]

Being branch aware

While coding these changes, I realized that the code had to be much more branch aware than previously designed. The code for processing a commit is aware of what branch the commit is done, and updates only those ports on that branch. However, the code for refreshing the database, when new fields are added, was not.

Getting the Branch from the path name

When processing the ports for refresh, a process not often done, the SQL has been altered to return the full pathname of the port. This is used to determine the branch we are upgrading. Two new functions were created: one to grab the branch from the path, and the other to set the branch in the database so it knows what to do.

[sourcecode]
Index: branches.pm
===================================================================
— branches.pm (revision 4692)
+++ branches.pm (working copy)
@@ -109,4 +109,52 @@
}
}

+sub GetBranchFromPathName($)
+{
+ my $pathname = shift();
+
+ # convert /ports/branches/2014Q1/archivers/hs-tar to 2014Q1
+ # convert /ports/head/archivers/hs-tar to head
+ # convert /ports/head/accessibility/accerciser to head
+
+ # the variable names used here assume HEAD
+ #
+ my ($emptyLeadingSlash, $subtree, $branch, $category, $port) = split/\//,$pathname, 5;
+
+ # example result based on above
+ # undefined ‘ports’ ‘head’ ‘archivers’ ‘hs-tar’
+
+ print "GetBranchFromPathName finds: ‘$subtree’, ‘$category’, ‘$port’\n";
+ if ($subtree ne $FreshPorts::Constants::PORTS) {
+ die("Unrecognized structure for pathname(‘$pathname’): $emptyLeadingSlash, $subtree, $category, $port");
+ }
+
+ if ($branch eq $FreshPorts::Constants::HEAD)
+ {
+ return $branch;
+ } else {
+ if ($branch eq ‘branches’)
+ {
+ return $category;
+ } else {
+ die("Unable to determine branch for ‘$pathname’\n");
+ }
+ }
+
+ die("Faulty code logic. We should never get here in GetBranchFromPathName\n");
+}
+
+sub SetBranchInDB($$) {
+ my $dbh = shift();
+ my $branch = shift();
+
+ $sql = ‘select freshports_branch_set(‘ . $dbh->quote($branch) . ‘)';
+ $sth = $dbh->prepare($sql);
+ if (!$sth->execute()) {
+ FreshPorts::Utilities::ReportError(‘warning’, "SetBranchInDB: Could not set branch:" . $dbh->errstr, 1);
+ }
+
+ $sth->finish();
+}
+
1;
[/sourcecode]

Lost time

I had trouble figuring the Branch problem. The update process kept adding dependencies to ports on the wrong branch, resulting in duplicate key errors. I incorrectly assumed the problem was related to adding duplicate and wrote some code to keep track of what had been added and not add duplicates. The problem was actually related to adding dependencies to ports not under consideration (i.e. the right port, but in the wrong branch). I’m re-running the update/population script now, with the duplicate-avoidance code removed.

Beta testing

This code will run on dev for a while before moving to production. Suggestions & comments always welcome.

 Posted by Dan Langille at 5:15 pm

Improving date search

 PostgreSQL  No Responses »
Nov 082014
 

I noticed the server load was high. I saw that several instances of search by date where still running.

Looking at the query in question:

[code language=”sql”]
freshports.org=# explain
freshports.org-# SELECT count(DISTINCT CL.id) AS count
freshports.org-# FROM commit_log_ports CLP, commit_log CL JOIN commit_log_branches CLB ON CL.id = CLB.commit_log_id
freshports.org-# JOIN system_branch SB ON SB.branch_name = ‘head’ AND SB.id = CLB.branch_id
freshports.org-# WHERE CL.commit_date BETWEEN ‘2014/11/07′::timestamptz + SystemTimeAdjust()
freshports.org-# AND ‘2014/11/07′::timestamptz + SystemTimeAdjust() + ‘1 Day’
freshports.org-# AND CLP.commit_log_id = CL.id;
QUERY PLAN
——————————————————————————————————————————————————————————————————————————————————-
Aggregate (cost=684.40..684.41 rows=1 )
-> Merge Join (cost=43.20..684.40 rows=1 )
Merge Cond: (clb.commit_log_id = cl.id)
-> Nested Loop (cost=2.55..3125771.74 rows=4875 )
Join Filter: (clb.commit_log_id = clp.commit_log_id)
-> Index Scan using commit_log_ports_pkey on commit_log_ports clp (cost=0.00..544853.97 rows=630110 )
-> Materialize (cost=2.55..618.00 rows=273 )
-> Hash Join (cost=2.55..616.64 rows=273 )
Hash Cond: (clb.branch_id = sb.id)
-> Seq Scan on commit_log_branches clb (cost=0.00..485.26 rows=33626 )
-> Hash (cost=2.54..2.54 rows=1 )
-> Seq Scan on system_branch sb (cost=0.00..2.54 rows=1 )
Filter: (branch_name = ‘head'::text)
-> Sort (cost=40.61..40.63 rows=9 )
Sort Key: cl.id
-> Index Scan using commit_log_commit_date on commit_log cl (cost=0.01..40.47 rows=9 )
Index Cond: ((commit_date >= (‘2014-11-07 00:00:00+00′::timestamp with time zone + ’00:00:00′::interval)) AND (commit_date <= ((‘2014-11-07 00:00:00+00′::timestamp with time zone + ’00:00:00′::interval) + ‘1 day'::interval)))
(17 rows)

freshports.org=#
[/code]

The analysis for this query is available at explain.depesz.com/s/uCT

I redesigned the query to make use of explicit JOINs:

[code language=”sql”]
freshports.org=# explain
freshports.org-# SELECT count(DISTINCT CL.id) AS count
freshports.org-# FROM commit_log CL JOIN commit_log_ports CLP ON CLP.commit_log_id = CL.id
freshports.org-# AND CL.commit_date BETWEEN ‘2014/11/07′::timestamptz + SystemTimeAdjust()
freshports.org-# AND ‘2014/11/07′::timestamptz + SystemTimeAdjust() + ‘1 Day’
freshports.org-# JOIN commit_log_branches CLB ON CL.id = CLB.commit_log_id
freshports.org-# JOIN system_branch SB ON SB.branch_name = ‘head’
freshports.org-# AND SB.id = CLB.branch_id;
QUERY PLAN
——————————————————————————————————————————————————————————————————————————————————————-
Aggregate (cost=692.70..692.71 rows=1 )
-> Nested Loop (cost=40.58..692.70 rows=1 )
-> Nested Loop (cost=40.58..652.23 rows=1 )
-> Hash Join (cost=40.58..651.95 rows=1 )
Hash Cond: (clb.commit_log_id = cl.id)
-> Seq Scan on commit_log_branches clb (cost=0.00..485.26 rows=33626 )
-> Hash (cost=40.47..40.47 rows=9 )
-> Index Scan using commit_log_commit_date on commit_log cl (cost=0.01..40.47 rows=9 )
Index Cond: ((commit_date >= (‘2014-11-07 00:00:00+00′::timestamp with time zone + ’00:00:00′::interval)) AND (commit_date <= ((‘2014-11-07 00:00:00+00′::timestamp with time zone + ’00:00:00′::interval) + ‘1 day'::interval)))
-> Index Scan using system_branch_pkey on system_branch sb (cost=0.00..0.27 rows=1 )
Index Cond: (id = clb.branch_id)
Filter: (branch_name = ‘head'::text)
-> Index Scan using commit_log_ports_pkey on commit_log_ports clp (cost=0.00..40.25 rows=18 )
Index Cond: (commit_log_id = cl.id)
(14 rows)

freshports.org=#
[/code]

The analysis for that query is at explain.depesz.com/s/p3wn.

I’ll let readers describe what happened here…

 Posted by Dan Langille at 12:32 pm

New categories

 Bug fixes  No Responses »
Oct 052014
 

New categories in the FreeBSD ports tree do not come along often, but when they do, FreshPorts is ready to pull the details in and get the category name right.

def ProcessCategoryNew():
  syslog.syslog(syslog.LOG_NOTICE, 'We have a new category')

  urllib.urlretrieve("www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD;content-type=text%2Fplain", '/usr/websites/freshports.org/dynamic/caching/tmp/categories');
  Touch(WWWENPortsCategoriesFlag)
  Touch(JOBWAITING)

No. It’s not.

That’s cvs. Won’t work.

It will redirect to https://svnweb.freebsd.org/ports/head/en/ports/categories and nothing of value will be retrieved.

Oops.

See also news.freshports.org/2007/10/12/new-categories/

 Posted by Dan Langille at 5:54 pm

cache invalidation

 Branches, Bug fixes, Caching, Full pathnames  No Responses »
Oct 052014
 

Today, kwm mentioned that www.freshports.org/Mk/bsd.port.mk was out of date, showing the most recent commit as back in March. My first concern was the system had stopped recording such commits. To check on that, I looked at https://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk and searched my email for the most recent commit message. Extracting the message id from that email, I quickly found that FreshPorts was will processing such

gipoco.com is neither affiliated with the authors of this page nor responsible for its contents. This is a safe-cache copy of the original web site.