Alternative PHP Cache (APC) saves the day

Posted by Mikko at 5 February 2012

Category: BiliBid

During the peak hours of BiliBid last Friday night, I had to heavily monitor our servers since it was the first time that we’re expecting higher than usual loads since I changed the set-up of our servers. From the usual 2.50 to 3.00 on normal days, it spiked to as high as 25.00-level. The server was consuming more RAM than usual that it had to use the swap space (virtual RAM). Fortunately, the server was able to survive the night.

Right after all the auctions, I immediately searched for ways on how to reduce server load and RAM usage. I had to find a solution right away since we’re expecting a much higher load the following night.

The first thing I had in mind was to get another server on the same datacenter, then do a load-balancing between two web servers while maintaining a single database server for data consistency. Obviously, the trade-off would be the cost.

I was about to deploy another server when I read an forum post by someone who share the same problem with me. I read the entire thread and read a suggestion to use the Alternative PHP Cache (APC). The thread starter confirmed that APC lowered down his CPU usage so I thought of giving it a try, too.

Right after installation, I immediately saw a drastic drop on the load. I played with APC’s configuration for some time then  restarted the server to free up some RAM in case I have memory leaks. After the restart, I saw promising figures, increasing my confidence on the server.

Then came Saturday night, the time I was expecting the heaviest load. During the time I had maximum number of requests on the server (even more than the max no of requests on Friday night), the load only played from 1.50 to 2.00, a huge leap from Friday night’s 25.00.

The server has officially survived the heaviest load (in terms of requests) for Saturday night without hiccups and with spare resources for moooore load. Thanks APC for saving the day!

Now, I have more time to optimize my UFC 143 live stream post. Hehehe.

The art of scaling

Posted by Mikko at 1 February 2012

Category: BiliBid, Programming

When we started BiliBid, it ran on an Apache web server. Our server’s system load played from 0.50 to 2.00 depending on the number of users online. On the first few days, everything ran well but eventually it came to a point when it was hogging too much RAM.

Just to give you an idea, I’ll enumerate the HTTP requests the BiliBid server is constantly receiving:

• The pages. They are the requests for the front page, winners page, single bid page, redeem PIN page, etc. It is requested once per pageview.
• The static files. The pages contain static files (Javascript and LESS files). Multiple requests per pageview, only requested when the page is loaded.
• Balance requests. When a user is logged in, the user’s browser requests for the bid balance once every two seconds.
• Timer requests. When a user is in the front page or a single bid page, the user’s browser requests for the timers of the items in the page once every second.
• Bid requests. Everytime a user places a bid, a request is sent to the server.

Multiply that to the number of users online!

Switch from Apache to nginx + php-fpm

One day around early September, I tried researching on nginx (engine-x) since I’ve heard that it has a smaller and more predictable memory (RAM) footprint. I also read stuff on php-fpm. I tried to weigh things since I’ll have to lose some Apache functionality if I switch to this one. This means no more .htaccess for me, but obviously I can convert my .htaccess file to something that nginx would understand. Fortunately, I found out that it wasn’t that hard to do it.

The same day, I installed nginx on the server. Before killing Apache and starting nginx, I fixed all my configuration files first. Then came the moment of truth, I killed Apache and started nginx. I was surprised that everything went so well that I couldn’t believe that BiliBid was already running in nginx.

Routing problems

Then came late October, people started reporting having high latencies with BiliBid. I think it was also the same period when I decided to put the Connection Quality indicator (before I did it, I know no other penny auction site with that feature – until I received a tip from a bidder that a penny auction site in the Philippines [starts with B and ends with A] implemented it, too). Before the problem occurred, I made no major change on BiliBid therefore I concluded that there might have been routing problems from some of our bidders’ end to our server.

This time, I decided to move our server to a nearer location to reduce latency and hopefully solve the routing problem. The move had trade-offs. It was more cost-efficient but to the expense of number of usable processor threads. On our previous server, I had 16 processor threads compared to four (4) on the new server.

I decided to review most of the SQL queries I was making and optimize it. I was able to reduce the execution time of my complex queries, some went down to up to 50%.

Everything went well until we started our Facebook campaigns. We received double the amount of concurrent online users. Our server load went crazy to up to 20.00, then one time it went as high as 70.00 during the auction of a big item. We had to postpone one of the big items to distribute the load over time. I tried experimenting different configurations on nginx, php-fpm and MySQL but to no avail.

Going back to our old server

This time, I decided to go back to our old server (16 usable processor threads). The problem with the server load disappeared. Everything was okay until recently, we received an overwhelming number of reports that translate to latency problems. I checked them one by one, around 1/2 were valid while the other third I consider invalid – just riding with the flow (and I hate it).

The most recent server switch, adding another server, MariaDB, etc

So I decided to go back to nearer server with four (4) processor threads. Before switching, I tried reading stuff to optimize the server to prevent high loads. I decided to change SQL server from MySQL to MariaDB. I think I first heard of MariaDB during a DevCon in UP. I experimented on the different storage engines offered by MariaDB until I settled with Aria.

Then came the switch. The latency was significantly lowered down (reduced by more than 50% on most users). The only problem was the higher-than-usual server load (as compared to our old server with 16 processor threads) which I fear would slow down the SQL queries (which is bad for a realtime app like a penny auction site) on peak hours.

Fearful of the slowdown, I decided to get another server on the same location. I configured the second server as the SQL server while the other one will be used solely as a web server. Since they are on the same location, I set up a private network for the two machines so that they exist in one logical network and one LAN.

With the new set-up (dedicated machine for SQL and another one for web serving), I was able to reduce the server load on the web server. The SQL server was also running very fine, in fact based on my tests the execution time for my most complex query was reduced to as much as 40%.

Other stuff I did

Since BiliBid is a time-sensitive web application, there is no room for slowdown. I had to:

• Optimize my cron job that runs every minute. I had to tweak it to only update information that needs to be updated, ignore others. Previously, it processes all my entities. Since the number of entities grow over time (new signups daily, new auctions, etc), I found it inefficient.
• Remove all system cron jobs. On a fresh install of most Linux distros, it automatically adds cron jobs for maintenance tasks. I had to remove them since a maintenance task could possibly slow down BiliBid. This equates to disaster if it runs on a peak hour or if there’s an active auction. I decided to manually run this maintenance tasks during off-peak hours.
• Made RRDTool graphs on the latency to make monitoring easier.

I really hope the stuff I did recently will work well. Until now, I continue to experiment different configurations to check the most efficient one.

The Cebu Experience

Posted by Mikko at 28 November 2011

Category: BiliBid, Travel

Tags: bilibid, cebu

I first stepped on Visayas’ soil last October 11, 2011 when the kids of BossChief Inc went to Cebu to promote our very own penny auction site, BiliBid. We ‘launched’ BiliBid in Cebu last October 13 at Handuraw Pizza located at Gorordo Avenue in Cebu City. Around 20 bloggers came up. Dinner was served, then the ‘program’ followed. It was a simple program. After dinner, we introduced BossChief Inc, the people behind it and BiliBid. The mechanics of BiliBid were explained, followed by a live bidding demo then the Q&A portion.

We spent most of our stay in Cebu inside our hotel room at Alba Uno Residencia, just behind IT Park. Our usual waking time was 1pm, which made it difficult for us to roam around the city since we had to fix ourselves before we can go out. Aside from our stay in the hotel, we were able to:

• hangout at IT Park (walking distance from the hotel),
• eat for dinner at The Ranch in Banilad,
• eat for lunch at Matias BBQ in A.S. Fortuna,
• ride the zipline at Papa Kit’s in Liluan,
• have our massage at Thai Boran near Crowne Regency Hotel just before leaving Cebu.

We weren’t able to explore Cebu much due to time constraints, but never did I know I’ll be coming back to Cebu after a month.

Last week, a BossChief and I came back to Cebu to meet some friends and talk about our upcoming business in Cebu. Tonight will be my last night in Cebu and tomorrow morning, I’ll be going back to Manila. Just like last time, we still checked in at Alba Uno Residencia.

This time, I was able to enter a mall in Cebu to attend the Visayas Blogging Summit. We had our ‘business meeting’ on a Sunday at Cebu Yacht Club. After our meeting, we went to Portofino Beach Resort beach resort, at last natikman ko na rin ang dagat ng Cebu. I was unprepared for our impromptu visit at the beach – but that did not stop me from dipping on the beach.

I’m looking forward to my next visit in Cebu. Sana tuloy-tuloy na once a month ako makakabalik ng Cebu.

Trying to solve an upcoming problem

Posted by Mikko at 23 September 2011

Category: BiliBid, Programming

Tags: bilibid

Since BiliBid started, I’ve encountered and foreseen problems that may affect the experience of the bidders. I’ve done preventive measures to counter the problems I’ve identified.

As BiliBid continues to grow, I foresee new problems that may arise. Most of them are due to concurrency (users accessing the same resource at the same moment). I have to tweak and optimize my code and queries from time to time to make it as efficient as possible, while maintaining correctness.

Testing the changes is even harder. It’s hard to replicate a potential problem, since first and foremost it does not entail a single request only. It’s a combination of continuous requests (for the timer, messages, and bid history) and user-triggered requests (placing a bid) from multiple clients. At times, there’s no assurance that the changes I’ve made will prevent those problems from happening. What I do is just convince myself that the changes I’ve made were for the better. Once convinced, I deploy it to the production server and hope for the best.

Maintaining BiliBid is not as easy as it may seem. It requires attention especially during the first bidding night after changes were committed. I have to trust on my gut feeling to decide whether to rollback the changes or not.

On the lighter note, I see this strenuous task of maintaining BiliBid as a learning experience. I learned a lot from BiliBid alone. For the love of BiliBid, I’ve read a lot of articles on server hardening, tweaking database and web servers and other related stuff.

Three weeks of BiliBid

Posted by Mikko at 16 September 2011

Category: BiliBid

Tags: bilibid

It’s been a rough ride for the BossChiefs (incorporators of BossChief Inc) for the last two months. It was early July when we thought of creating BiliBid. We first planned to open BiliBid to the public on July 25. Then we thought of moving it on August 6, until it was finally pushed to August 22.

It was a rough ride before we actually opened. Programming BiliBid was a breeze (lucky for us we had a good programmer, yours truly), but designing it took a lot of time. It took me five (5) designs before I was finally satisfied by my work. It took us countless debates on the designs, until they gave up since I always fought for what I wanted – and since I was the one doing it, it was always my call. Continue reading “Three weeks of BiliBid” »