Alternative PHP Cache (APC) saves the day
Posted by Mikko at 5 February 2012
Category: BiliBid
During the peak hours of BiliBid last Friday night, I had to heavily monitor our servers since it was the first time that we’re expecting higher than usual loads since I changed the set-up of our servers. From the usual 2.50 to 3.00 on normal days, it spiked to as high as 25.00-level. The server was consuming more RAM than usual that it had to use the swap space (virtual RAM). Fortunately, the server was able to survive the night.
Right after all the auctions, I immediately searched for ways on how to reduce server load and RAM usage. I had to find a solution right away since we’re expecting a much higher load the following night.
The first thing I had in mind was to get another server on the same datacenter, then do a load-balancing between two web servers while maintaining a single database server for data consistency. Obviously, the trade-off would be the cost.
I was about to deploy another server when I read an forum post by someone who share the same problem with me. I read the entire thread and read a suggestion to use the Alternative PHP Cache (APC). The thread starter confirmed that APC lowered down his CPU usage so I thought of giving it a try, too.
Right after installation, I immediately saw a drastic drop on the load. I played with APC’s configuration for some time then restarted the server to free up some RAM in case I have memory leaks. After the restart, I saw promising figures, increasing my confidence on the server.
Then came Saturday night, the time I was expecting the heaviest load. During the time I had maximum number of requests on the server (even more than the max no of requests on Friday night), the load only played from 1.50 to 2.00, a huge leap from Friday night’s 25.00.
The server has officially survived the heaviest load (in terms of requests) for Saturday night without hiccups and with spare resources for moooore load. Thanks APC for saving the day!
Now, I have more time to optimize my UFC 143 live stream post. Hehehe.
The art of scaling
Posted by Mikko at 1 February 2012
Category: BiliBid, Programming
When we started BiliBid, it ran on an Apache web server. Our server’s system load played from 0.50 to 2.00 depending on the number of users online. On the first few days, everything ran well but eventually it came to a point when it was hogging too much RAM.
Just to give you an idea, I’ll enumerate the HTTP requests the BiliBid server is constantly receiving:
- The pages. They are the requests for the front page, winners page, single bid page, redeem PIN page, etc. It is requested once per pageview.
- The static files. The pages contain static files (Javascript and LESS files). Multiple requests per pageview, only requested when the page is loaded.
- Balance requests. When a user is logged in, the user’s browser requests for the bid balance once every two seconds.
- Timer requests. When a user is in the front page or a single bid page, the user’s browser requests for the timers of the items in the page once every second.
- Bid requests. Everytime a user places a bid, a request is sent to the server.
Multiply that to the number of users online! 
Switch from Apache to nginx + php-fpm
One day around early September, I tried researching on nginx (engine-x) since I’ve heard that it has a smaller and more predictable memory (RAM) footprint. I also read stuff on php-fpm. I tried to weigh things since I’ll have to lose some Apache functionality if I switch to this one. This means no more .htaccess for me, but obviously I can convert my .htaccess file to something that nginx would understand. Fortunately, I found out that it wasn’t that hard to do it.
The same day, I installed nginx on the server. Before killing Apache and starting nginx, I fixed all my configuration files first. Then came the moment of truth, I killed Apache and started nginx. I was surprised that everything went so well that I couldn’t believe that BiliBid was already running in nginx.
Routing problems
Then came late October, people started reporting having high latencies with BiliBid. I think it was also the same period when I decided to put the Connection Quality indicator (before I did it, I know no other penny auction site with that feature – until I received a tip from a bidder that a penny auction site in the Philippines [starts with B and ends with A] implemented it, too). Before the problem occurred, I made no major change on BiliBid therefore I concluded that there might have been routing problems from some of our bidders’ end to our server.
This time, I decided to move our server to a nearer location to reduce latency and hopefully solve the routing problem. The move had trade-offs. It was more cost-efficient but to the expense of number of usable processor threads. On our previous server, I had 16 processor threads compared to four (4) on the new server.
I decided to review most of the SQL queries I was making and optimize it. I was able to reduce the execution time of my complex queries, some went down to up to 50%.
Everything went well until we started our Facebook campaigns. We received double the amount of concurrent online users. Our server load went crazy to up to 20.00, then one time it went as high as 70.00 during the auction of a big item. We had to postpone one of the big items to distribute the load over time. I tried experimenting different configurations on nginx, php-fpm and MySQL but to no avail.
Going back to our old server
This time, I decided to go back to our old server (16 usable processor threads). The problem with the server load disappeared. Everything was okay until recently, we received an overwhelming number of reports that translate to latency problems. I checked them one by one, around 1/2 were valid while the other third I consider invalid – just riding with the flow (and I hate it).
The most recent server switch, adding another server, MariaDB, etc
So I decided to go back to nearer server with four (4) processor threads. Before switching, I tried reading stuff to optimize the server to prevent high loads. I decided to change SQL server from MySQL to MariaDB. I think I first heard of MariaDB during a DevCon in UP. I experimented on the different storage engines offered by MariaDB until I settled with Aria.
Then came the switch. The latency was significantly lowered down (reduced by more than 50% on most users). The only problem was the higher-than-usual server load (as compared to our old server with 16 processor threads) which I fear would slow down the SQL queries (which is bad for a realtime app like a penny auction site) on peak hours.
Fearful of the slowdown, I decided to get another server on the same location. I configured the second server as the SQL server while the other one will be used solely as a web server. Since they are on the same location, I set up a private network for the two machines so that they exist in one logical network and one LAN.
With the new set-up (dedicated machine for SQL and another one for web serving), I was able to reduce the server load on the web server. The SQL server was also running very fine, in fact based on my tests the execution time for my most complex query was reduced to as much as 40%.
Other stuff I did
Since BiliBid is a time-sensitive web application, there is no room for slowdown. I had to:
- Optimize my cron job that runs every minute. I had to tweak it to only update information that needs to be updated, ignore others. Previously, it processes all my entities. Since the number of entities grow over time (new signups daily, new auctions, etc), I found it inefficient.
- Remove all system cron jobs. On a fresh install of most Linux distros, it automatically adds cron jobs for maintenance tasks. I had to remove them since a maintenance task could possibly slow down BiliBid. This equates to disaster if it runs on a peak hour or if there’s an active auction. I decided to manually run this maintenance tasks during off-peak hours.
- Made RRDTool graphs on the latency to make monitoring easier.
I really hope the stuff I did recently will work well. Until now, I continue to experiment different configurations to check the most efficient one.
The Cebu Experience
Posted by Mikko at 28 November 2011
Category: BiliBid, Travel
Tags: bilibid, cebu
I first stepped on Visayas’ soil last October 11, 2011 when the kids of BossChief Inc went to Cebu to promote our very own penny auction site, BiliBid. We ‘launched’ BiliBid in Cebu last October 13 at Handuraw Pizza located at Gorordo Avenue in Cebu City. Around 20 bloggers came up. Dinner was served, then the ‘program’ followed. It was a simple program. After dinner, we introduced BossChief Inc, the people behind it and BiliBid. The mechanics of BiliBid were explained, followed by a live bidding demo then the Q&A portion.
We spent most of our stay in Cebu inside our hotel room at Alba Uno Residencia, just behind IT Park. Our usual waking time was 1pm, which made it difficult for us to roam around the city since we had to fix ourselves before we can go out. Aside from our stay in the hotel, we were able to:
- hangout at IT Park (walking distance from the hotel),
- eat for dinner at The Ranch in Banilad,
- eat for lunch at Matias BBQ in A.S. Fortuna,
- ride the zipline at Papa Kit’s in Liluan,
- have our massage at Thai Boran near Crowne Regency Hotel just before leaving Cebu.
We weren’t able to explore Cebu much due to time constraints, but never did I know I’ll be coming back to Cebu after a month.
Last week, a BossChief and I came back to Cebu to meet some friends and talk about our upcoming business in Cebu. Tonight will be my last night in Cebu and tomorrow morning, I’ll be going back to Manila. Just like last time, we still checked in at Alba Uno Residencia.
This time, I was able to enter a mall in Cebu to attend the Visayas Blogging Summit. We had our ‘business meeting’ on a Sunday at Cebu Yacht Club. After our meeting, we went to Portofino Beach Resort beach resort, at last natikman ko na rin ang dagat ng Cebu. I was unprepared for our impromptu visit at the beach – but that did not stop me from dipping on the beach.
I’m looking forward to my next visit in Cebu. Sana tuloy-tuloy na once a month ako makakabalik ng Cebu.
(29 Sept) More PAL flights push through, majority still cancelled
Posted by Mikko at 29 September 2011
Category: Travel
Tags: pal
It’s a strike three for the Philippine Airlines!
The Philippine Airlines is still struggling to return to normalcy two days (2) after the Philippine Airlines Employees’ Association (Palea) went on a sit-back strike.
Most of the airlines flights are cancelled for the third consecutive day already. PAL posted on their Facebook page that the following international and domestic flights will push through today (29 Sept) until further notice. All flights not listed below are considered cancelled. Continue reading “(29 Sept) More PAL flights push through, majority still cancelled” »
(28 Sept) PAL continues some flights, most still cancelled
Posted by Mikko at 28 September 2011
Category: Travel
Tags: pal
The Philippines’ flag carrier, the Philippine Airlines (PAL), said on its Facebook page that they will be operating select domestic and international flights today, September 28, 2011. 62 domestic flights and 40 international flights shall remain canceled today.
Only the following flights from Manila will operate today:
DOMESTIC PR847 0600H CEB PR291 0700H DGT PR321 0855H KLO PR811 0900H DVO PR853 0900H CEB PR323 1330H KLO PR293 1340H DGT PR849 1530H CEB PR813 1530H DVO
INTERNATIONAL PR896 0720H TPE PR300 0800H HKG PR438 1330H NGO PR408 1425H KIX PR432 1430H NRT PR306 1450H HKG PR116 1850H YVR PR102 2200H LAX PR104 2230H SFO
This comes a day after members of the ground crew union Palea (Philippine Airlines Employees’ Association) went on a surprise sit-back strike. The employees refused to perform their duties such as facilitating disembarkment of baggage, refuelling aircrafts, checking in passengers, among others. Continue reading “(28 Sept) PAL continues some flights, most still cancelled” »
Trying to solve an upcoming problem
Posted by Mikko at 23 September 2011
Category: BiliBid, Programming
Tags: bilibid
Since BiliBid started, I’ve encountered and foreseen problems that may affect the experience of the bidders. I’ve done preventive measures to counter the problems I’ve identified.
As BiliBid continues to grow, I foresee new problems that may arise. Most of them are due to concurrency (users accessing the same resource at the same moment). I have to tweak and optimize my code and queries from time to time to make it as efficient as possible, while maintaining correctness.
Testing the changes is even harder. It’s hard to replicate a potential problem, since first and foremost it does not entail a single request only. It’s a combination of continuous requests (for the timer, messages, and bid history) and user-triggered requests (placing a bid) from multiple clients. At times, there’s no assurance that the changes I’ve made will prevent those problems from happening. What I do is just convince myself that the changes I’ve made were for the better. Once convinced, I deploy it to the production server and hope for the best.
Maintaining BiliBid is not as easy as it may seem. It requires attention especially during the first bidding night after changes were committed. I have to trust on my gut feeling to decide whether to rollback the changes or not.
On the lighter note, I see this strenuous task of maintaining BiliBid as a learning experience. I learned a lot from BiliBid alone. For the love of BiliBid, I’ve read a lot of articles on server hardening, tweaking database and web servers and other related stuff.
Three weeks of BiliBid
Posted by Mikko at 16 September 2011
Category: BiliBid
Tags: bilibid
It’s been a rough ride for the BossChiefs (incorporators of BossChief Inc) for the last two months. It was early July when we thought of creating BiliBid. We first planned to open BiliBid to the public on July 25. Then we thought of moving it on August 6, until it was finally pushed to August 22.
It was a rough ride before we actually opened. Programming BiliBid was a breeze (lucky for us we had a good programmer, yours truly), but designing it took a lot of time. It took me five (5) designs before I was finally satisfied by my work. It took us countless debates on the designs, until they gave up since I always fought for what I wanted – and since I was the one doing it, it was always my call. Continue reading “Three weeks of BiliBid” »
Best Summer Yet
Posted by Mikko at 2 June 2011
Category: Personal
Tags: college
Hello, I’m back at my blog after four (4) months. I haven’t blogged for the entire summer term, not because I don’t have time and absolutely not because I don’t have anything to share.
A lot of things happened during the past months. I’ve been to a four-day basic masses integration and art workshop in Hacienda Luisita in Tarlac, finished yet another weight training PE, transformed my laptop, explored haircuts, had fun with my classes and my classmates and a lot more.
After finishing my third weight training class in UP, I can say that this summer’s weight training under Sir Marvin Sabado was the most challenging. I’m very proud that I still got a 1.50 from him even though I missed two days of classes during our trip in Hacienda Luisita. All those strenuous days of workout was definitely worth it. Until now, I still continue the abdominals regimen I’ve learned from the class from time to time.
PE 2 WT-M W7 class
My English 10 class under Miss Loujaye Sonido is so far the best GE I’ve taken in UP. I enjoyed the class plus the outside-the-classroom experiences. I really enjoyed the company of my classmates and our beautiful teacher. My thesis paper partner and I also got a 1.00 on our thesis paper.
Our class went on a four-day basic masses integration and art workshop (Lakbay Sining) with Sinagbayan (Sining na Naglilingkod sa Bayan) at Hacienda Luisita in Tarlac. There I learned to asarol (hoe) the soil at Ka-Gerry’s place. I also enjoyed pumping the poso (handpump) for different people (mostly strangers) for three straight days; which was an instant workout for me in preparation for the upcoming weight training practicals then. Did that for three days and fortunately my triceps didn’t sore up. During the art workshop, we were asked to choose our class. I chose the art workshop as I don’t have a talent in playing musical instruments, dancing and making songs.
Ang pag-aasarol
During the “last day of summer classes” inside the classroom and even after it, my classmates and I still went out from time to time during deadline of submission of papers and the benefit gigs we had to attend such as the Barikada: Tulong para sa Corazon de Jesus at Hacienda Luisita. Among all of them, I enjoyed Barikada the most. Barikada was a night filled with progressive music and booze – for a cause. I never imagined an activist gig to be that fun.
At Clerisy with English 10 classmates after last day of classes (inside the classroom)
English 10 class picture at Barikada
More Barikada photos
The English 10 kids who stayed ’til the end plus Miss Loujaye
Being a BLKD fan, I did not miss the opportunity at Barikada to have a fan photo with him. To my surprise, he recognized me even though we only met at Hacienda Luisita. More info about BLKD on at fuckyeahblkd in tumblr.
Fan photo with BLKD
Even more BLKD fanboys
As compared to my previous summers, I spent lesser chunk of my time in front of my laptop. But that didn’t stop my laptop from giving up. Too bad for my laptop, his owner loves tinkering stuff. And this is how it looked like while I was tinkering (stripping) it:
I decided to give it some decent outfit so I bought filing cases from CDRKing. Eventually, it now looks like this now:
As the summer classes were about to end, I also made a remake of a video I uploaded before. Here’s the remake:
I would like to share moooooooore photos and videos but my body is already giving up. Disclaimer: the next links are potentially offensive and may be NSFW. I’d be wrapping this up with a pull-ups shot taken by a classmate, and a couple of vanity shots I took supposedly for my own consumption.
Blah blah blah
Posted by Mikko at 5 February 2011
Category: Personal
Tags: fitness
Hey guys. It’s been more than a month since I last blogged. I just feel ranting now after cleaning the entire room. The last time we cleaned was around last October so we had to clean more than three (3) months worth of mess. It took us more than five (5) hours to finish it.
For the past few days, I’ve been keeping myself busy with my fitness. LOL. I tried to exercise more, started to jog and significantly decrease my alcohol consumption. I guess my effort paid off. Last November 2010, my body fat percentage was still around 17%. Yesterday, the interns at the College of Human Kinetics measured it for us again for our fitness test and now it’s down to 6.2%. My resting heart rate is now 48bpm, enough for me to self-diagnose myself with sinus bradycardia. Both my resting heart rate and body fat percentage say I qualify as an athlete, now I need to find a sport to play. My next goal is to increase my BMI from 21.22 to 23 while still maintaining my body fat percentage to below 10%.
By now, I should be keeping myself busy WordPress-izing a website but I’m still not in the mood of doing it.
Our almost 16-hour delay with Zest Air
Posted by Mikko at 3 January 2011
Category: Travel
Tags: zest air
Summary (all times were approximates, GMT+8)
- Passengers of this flight, you may answer this form so that we can coordinate in case someone’s filing a civil case against Zest Air.
- Scheduled flight Zest Air Z2393 (ETD: January 2, 2011 1915h) bound for Manila from Davao.
- No announcement of delay before ETD.
- 1945h, 30 minutes after ETD, ground crew flight will be delayed due to additional servicing of aircraft. No new ETD was given.
- No announcements were made until 2230h, 3 hours 15 minutes after ETD. People from Zest Air were “hiding” most of the time as they weren’t there to assist us.
- 2230h, a “supervisor” named Emmanuel Aldea, announced that flight is cancelled. A new flight will be held the next morning at 0625h, in conjunction with an existing 0625h Zest Air flight. No hotel accommodations, just free dinner.
- New ETD is now January 3, 2011 0625h.
- No announcements from Zest Air until around 1000h the next day. Not even an announcement of delay of our new flight. Announced that new ETD will be January 3, 2011 1030h.
- The other 0625h flight (but delayed) pushed through but carried passengers scheduled for the flight. People started complaining since we weren’t prioritized even if we were delayed for more than 11 hours already.
- The 1030h flight pushed through. The Zest Air management issued no official statement regarding the delay.
- In-flight, they served our “breakfast” at 1200h. To add insult to injury, the food pax were not enough for everyone in the fight. There was also a lack of utensils. Some people at their food pax with their hands.
- Original ETA: January 2, 2011 2100h. Actual time of arrival: January 3, 2011 1240h.
- Total delay: 15 hours and 40 minutes.
Happy New Year everyone. I started writing this post on my tenth hour at Davao International Airport. My sister and I were here more than two hours before the ETD of our Zest Air flight as a check-in requirement.
We checked-in together with my sister’s batchmates (which we saw at the airport). For me and my sister’s ticket, a Zest Air ground staff asked for a photocopy of the credit card used to purchase the ticket. Since we had none, we were asked the first and last four digits of the credit card. We were able to provide it right away since it was stored on my phone. As the crew gave us our boarding passes, we immediately notice that there were no names on them. Just passenger and seat numbers, not even flight or gate number. Continue reading “Our almost 16-hour delay with Zest Air” »