Archive for the ‘technology’ Category

Debugging the Android Battery

Does android battery suck, or is it usage? In short it does suck compared to other popular alternatives(the i range). However the purpose of this post is not to start a witchhunt or rant but to arm you with tools to debug your battery usage. I will help you track down your battery hog(s) and then suggest some methods to tackle them.

There are 4 main causes of battery hogs:

  1. Cell Phone Usage
  2. Display
  3. CPU usage
  4. Sensors like (GPS, Wifi etc)

Lets start with the easy one:

Cell Phone Usage

Cell tower is the one that provides you connectivity, EDGE and 3G. Unfortunately weak signal can cause this to use exorbitant amount of  battery. If you are experiencing full battery drain in about 2-3 hours then this is most probably the culprit. First thing to do is go Settings -> About Phone -> Battery Use -> Cell Phone standby. Look @ Time spent without signal. Any number other than 0 is very dangerous. There are no practical solutions to this problem. Search for network and fix it. This will prevent phone from searching network when signal is weak. However this may not work when you go on roaming and if you spend most time out of signal this will still consume battery.


The only downside of having a large screen is that it consumes exponential charge. If you using an LCD screen, your phone is only likely to survive 4-5 hrs of screen on time. This is assuming that you are not performing any CPU intensive task during that period like gaming, video etc. It is highly likely that you will consume 4 hrs of usage out of your phone in the entire day hence its unlikely that your phone will survive more than a day. The only aspect to debug here is if some freaky software or buggy ROM is keeping your screen up. Gingerbread provides you clean interface to check your screen on period. You can use Juice Plotter on non gingerbread phones but it has been known to consume a lot of battery itself. Make sure you only use to to debug your screen and not on a regular basis.

Have a look at your Screen ON time and make sure you are getting around 4 hrs of usage. Any thing less than that means that you have issues with your battery. Look through the CPU Usage and Sensor usage area  debug those issues.


  1. Choose an AMOLED screen phone: AMOLED screens consume very little battery in comparison to LCD Screens.
  2. If you have an AMOLED screen then choose darker themes while browsing/surfing/wallpapers. A light color pixel consumes about 5.8 times more energy than dark pixel. This method will not work for LCD phones. You can see the impact of changed wall paper using Current Widget. This will tell you how much charge your phone is consuming(wait for couple of min for it to refresh when you change the wall paper).
  3. Choose  lower screen brightness in general. Rely on Power control widgets to increase screen brightness when required.
  4. Reduce display timeout to 30sec-1min. Adjust that to a time that is comfortable to you.

CPU usage

Today’s CPU are beast. This means not only they are fast and powerful they consumAndroid allows apps to ask for CPU while not using the screen. e millions of amperes of precious juice(okay maybe not millions). CPU usage is a bit tricky to debug in pre-Gingerbread phones. You are mostly led to superstitious advices to control power usage. You can see as much as 10-20% battery usage overnight when you were not using the screen, this is because  Android allows apps to ask for CPU while not using the screen and poorly designed apps continue to update even when they have not been used in a while draining CPU. Enough rant now to function, to debug CPU:
  1. Launch Spare Parts. Fortunately this is bundled in Gingerbread phones, for Froyo you can try out this market app and hope it will work.
  2. Look at Battery history inside Spare Parts. Select “Other Usage” in first drop down.
  3. Check out Running time vs Screen On time.
If these two numbers are widely different then you have a problem on your hands. This means that some app is keeping CPU awake even when your screen is off. To find out which app
  1. Launch Spare Parts
  2. Look at battery history and select “Partial Wake Usage”
  3. Now you will see which app is the culprit and eating up CPU.
Now comes the hard part . You can reduce your running time by disabling this app. If the app manufacturer is good, he would have provided you means to reduce its battery usage(Disabling sync, changing sync from push to hourly, reducing data refresh etc). If not then you should uninstall the app and use something else for that purpose. Keep repeating this exercise till your running time is almost equal to your screen on time.  You will see radical improvements in your standby time with this exercise which in turn can reduce your charge cycles to as much as 2 days.
Other than choosing good apps which use CPU wisely some suggestions:
  1. Use less push sync in general. True push is only available with Blackberry everybody else is doing a very frequent pull to emulate push, so make sure you don’t use push where its not required.
  2. In my opinion disable auto sync all together and do a manual sync when needed. You can put the auto sync button in your power widget and flick it on and off whenever needed
  3. Use Juice Defender it will give you good usage patterns(disabling sync at night etc).
Solution for Advanced Users: if you need all the apps that sync but still want to reduce your battery usage you can try Set CPU to underclock your CPU. You can also use the inbuilt underclocking baked into the CM7 ROM. This solution will only work if you have rooted your phone and your kernel supports underclocking. Default android kernel comes with OnDemand CPU clocking. CPU almost instantly scales to max frequency(like 1Ghz) from its min frequency(like 245Mhz) when required. You will need this kind of speed for gaming/browsing etc however its not required for sync and other background tasks. Set CPU lets you configure so that
  1. Your CPU scales slowly(Conservating  scaling/Power intensive), especially when you have very little battery left
  2. Reduce highest CPU frequency to something like 700 Mhz especially during the night or when the screen is off
Keep in mind however that these solutions can impact your phones performance while rendering pages, gaming etc, so do not reduce cpu frequency or scaling when your screen is on to prevent frustrating experiences.

Sensor usage

If you have reached here then the problem is quite simple shut off all your sensors :). Well Android provides various ways of debugging sensor usage, have a look at your Spare parts => Battery History => Sensor Usage/GPS Usage. If you see a lot of devices are using GPS Sensor you are likely to see a lot of battery loss. Its unfair that android does not let us control which device has access to my location and which does not, I hope they will add that in future versions of android. Meanwhile if you see useless apps tryto access location(mostly to serve relevant ads etc) then you can adopt the following solutions:
  1. Disable both ways to determine your location in Settings => Location and Security. If you really need location then go for “use wireless location” instead of GPS for location. Use GPS only in critical situations where precise location is very necessary
  2.  Change Wifi sleep policy (in Settings => Wireless and Networks => Wifi Settings => Menu key => Advanced => Wifi Sleep policy => When screen is off) This will switch off your wifi and turn on  mobile network when you are not using the screen. This way your background apps can use your slow mobile network to sync while saving wifi juice.
  3. Turn on “Only 2G network” in (Settings => Wireless and Networks =>  Mobile Networks) . This will switch off your 3G network and use only 2G which uses much less battery and is much slower. You can also use 2G/3G switcher in power widget to control this if you rely on 3G for browsing.
Got any other suggestions?? Let me know in comments…

Apache Mahout 0.2 Released – Now classify, cluster and generate recommendations!

Apache Mahout

Apache Mahout

For the past two years, I have been working with this amazing bunch of people whilst, being paid by Google in their summer of code program in a project called Mahout. And like the name says, it is trying to tame the young beast known as Hadoop. I have received a lot from the community. Being part of the project, I have got some real exposure to Java, data mining, machine learning and hands on experience over distributed systems like Hadoop, Hbase, Pig.  The project is still in its infancy, but, its ambitions are high in the sky. I am happy to announce the second release of the project, and proud to be a part of it. I hope people will adapt it in their projects and that it becomes the defacto standard machine learning library the way lucene and hadoop has become in their respective focus areas.

If you are already excited and want to take it for a ride, read Grant’s article on IBM developerworks here
The release announcement below

Apache Mahout 0.2 has been released and is now available for public download at

Up to date maven artifacts can be found in the Apache repository at

Apache Mahout is a subproject of Apache Lucene with the goal of delivering scalable machine learning algorithm implementations under the Apache license.

Mahout is a machine learning library meant to scale: Scale in terms of community to support anyone interested in using machine learning. Scale in terms of business by providing the library under a commercially friendly, free software license. Scale in terms of computation to the size of data we manage today.

Built on top of the powerful map/reduce paradigm of the Apache Hadoop project, Mahout lets you solve popular machine learning problem settings like clustering, collaborative filtering and classification
over Terabytes of data over thousands of computers.

Implemented with scalability in mind the latest release brings many performance optimizations so that even in a single node setup the library performs well.

The complete changelist can be found here:

New Mahout 0.2 features include

  • Major performance enhancements in Collaborative Filtering, Classification and Clustering
  • New: Latent Dirichlet Allocation(LDA) implementation for topic modelling
  • New: Frequent Itemset Mining for mining top-k patterns from a list of transactions
  • New: Decision Forests implementation for Decision Tree classification (In Memory & Partial Data)
  • New: HBase storage support for Naive Bayes model building and classification
  • New: Generation of vectors from Text documents for use with Mahout Algorithms
  • Performance improvements in various Vector implementations
  • Tons of bug fixes and code cleanup

Getting started: New to Mahout?

For more information on Apache Mahout, see

Creating Indexes/Sorting on very large tables in Mysql

If your dataset is anywhere between 1 million to  180 million then you have come to the right blog.  Normally a procedure like this will require atleast 1 day or more depending on dataset. But I will teach you how to get results in as quick as 3 hrs. You will be able to execute queries on indexes fully utilizing them without going through the painful process of creating them on the the entire table. Recently at work I had to deal with a dataset of 180 million  and this entry gives details of the work flow that I adopted.  I will not effectively sort the complete table but will tell you how to achieve the same effect in less than 2-3 hrs for 180 million rows. I tried well known methods to achieve indexing. I tried to use Alter table statement after 2 hrs stopped it. I tried to dump the data in hope of inserting it in an indexed table but after 1 hr stopped that as well. Then I explored Mysql functionality a bit to discover 3 features which can really reduce time, those were Memory tables, Merge tables and Partitioning(not necessary). Fortunately sorting using merge sort is a recursive job, so in the process of sorting 180 million rows I also had to learn how to sort 2 million faster than normal index creation. So the first thing that you have to do is  find out the order of rows that you have.

If you have less than 1 million rows then you are better off using mysql’s inbuilt commands

Alter table <tblname> create index (<column name>);

If you have more than than then read on…..

The concepts I will be using are as follows:

Memory Tables:

These tables are hash maps that mysql stores in memory. These tables are temporary (in the sense they disappear after mysql server restart)  and have limited rows. However they are lighting fast on index creation or sorting.


Somebody in Mysql finally realized that to be fast database has to split the data across servers. Finally in version 5 they have introduced partitioning. If you have the luxery of more than 1 server you can use this. I did not have this luxury so will do indexing without it.

Merge tables:

So these are 1 server counterpart of Partitioning. If you want to split your dataset into multiple tables and run single query on all of them you can use Merge tables. They have their own share of problems, but for general functionality of insertion, deletion, searching and dumping, they work great. The only 2 major limitations one that they only work if base tables are MyISAM and searching on them is multiples of logn(see below if interested)

Irrelevant Mathematical jargon(feel free to skip)

So if you search  table on indexed columns then search takes Log(n) time (n = number of tuples). So imagine you have 100*n size of table and you create 100 sub tables of n each. Now searching on one table will take Log(n) time and total time is 100*log(n)=log(n to the power 100). If it were a single table it would have taken Log(100*n) =Log(100)+log(n). So in a single table it scale logarithmically but merge table scales exponentially.

I will explain the steps that I took below, you can customize them to your dataset.

There are several ways of creating index however mine will require you to have quite a powerful machine and a reliable power supply. Atleast 2gb of ram is required and more is better here. If you have around 2 million rows then skip to step 4.

Step 0

This step is about reconnaissance. It will determine the numerical data in the next steps. You need to know the approximate amount of space your row takes. This will determine the size of your memory table. You can determine the size of memory table by trial and error also but a numerical estimate will help.  In the steps below I will assume initial table as client_data and final table as client_data_sorted. All other tables are temp and generated on the spot.

Step 1: Increasing allowed size of Memory table to fit data

The first thing you need to do is extend the allowed memory limit of mysql. To sort faster we will need memory tables. Do so by adding the lines marked as red below under  [mysqld] in my.cnf file. Typically located in /etc/my.cnf or /etc/mysql/my.cnf depending on your distribution.

my.cnf file



Adding these lines will ensure that mysql can now take 4GB space for storing memory tables.Feel free to increase this number if you have higher memory but ensure that you dont encounter swapping.

Step 2: Dump data in memory table for indexing

Now you need to create a  memory table with same structure as your final table.

create table client_data_memory like client_data_sorted;
alter table client_data_memory engine=Memory;

The second line alters the engine to memory. The only thing to keep in mind is that memory engine keeps all the data in memory hence if your mysql server restarts or machine restarts then all the data is lost. Ensure that you never rely on them as a reliable storage.

Figure out  how many rows your memory table can contain. In a typical scenario this would be order of 10 million. This value will change depending on your system’s memory and values set in step 1.  You can test it out by inserting data from source table to memory table. After the limit the process will interrupt in the middle giving error like table is full.

Insert into client_data_memory select * from client_data limit <test limit value>;

Ex:  Insert into client_data_memory select * from client_data limit 0,10000000;    # 10 million rows from 0th row

Step 4: Storing data from Memory table to MyISAM table

If you have order of 10 million rows then the process stops for you here. Just use an alter table command to create index on memory table(3-4 minutes) then insert the memory table data into client_data_sorted table. You just saved yourself  hours of pain. If you have more than 10 million then  skip this step.

Alter table client_data_memory add index (<column name>);

(or sort the table)

Insert into client_data_sorted select * from client_data_memory;

If you can store 10million rows in your memory table and total tables are around 40 million then you are better off iteratively repeating the above steps. Merely sort 10 million then insert, then truncate memory table , then insert next 10 million , sort then insert. The process will take exponentially more time every time but still will finish far faster than normal.  If you have more than 40 million then read on. A world of pain awaits you….

Step 5: Create Merge table for accessing data

The above process will not work iteratively for you if you have more than 50 million rows as while inserting you have to merge too. Suppose your memory can store 10 million and you have 180 million rows then create 18 temporary tables of engine type MyISAM.

create table data_temp1 like client_data_sorted;
alter table data_temp1 engine=MyISAM;

Now use the above technique to insert data 10 million a piece into each of these tables. So data_temp0 will contain 0-10 million then data_temp1 will have 10 miliion to 20 million and so forth.

truncate table client_data_memory;insert into client_data_memory select * from client_data limit
0,15000000;insert into data_temp0 select * from client_data_memory;
truncate table client_data_memory;insert into client_data_memory select * from client_data limit
10000000,15000000;insert into data_temp1 select * from client_data_memory;


This will take quite a long time, for me it took 10 min for every 10 million so 180 min in total. Meanwhile consider some of the more worthwhile alternatives like  hive, its too late for it now but its worth a look. This is also useful.

Now create a merge table. A merge table is created on top of several MyISAM tables of same structure. Its important to have same structure and MyISAM tables. You can use this as  a single table however all the searching is performed on all tables and hence is slower than single table.

create table client_data_merge like client_data_sorted;
alter table client_data_merge engine=merge union=


9,data_temp10,data_temp11,data_temp12) INSERT_METHOD=LAST;

Step 5: Final Sorted table(if required)

For most temporary uses this merge table will work. You can use this table normally. Insert data into the table etc etc. However if the limitations of merge tables are not acceptable you have to create the final table  from this merge table. You can use merge tables to dump the data in a sorted fashion according to indexes or simply use

Insert into <tblname> Select * from <merge table> order by <index1>,<index2>…..

This way you get to Insert much faster as the data is already sorted and mysql simply has to insert the data and update the indexes. If this method helped.. do let me know in the comments.

Page Faults and Context Switches: Chrome, Flash and Techcrunch are killing my system

Update: Beta News reported a similar performance drop when running chrome on Windows 7 RC

I am a power user. When I surf, I have around 20-25 tabs open. I have many processes running simultaneously on my windows 7 RC.TaskManager When I code, I have a test Ubuntu installation running on VMware.  To do all these, I spent most of my earnings on a 17″, Core2duo 2.4Ghz, 4GB ram, NVidia 8600M GT Dell Inspiron Laptop 1720. I have been using Google Chrome since the day it was made available and i keep up with the latest builds of the same. I also love the fact that it was zipper than my earlier favorite(Firefox) in opening new tabs. But, the whole time Google Chrome has been stressing my laptop’s hardware.

To prove my point, take look at the screenshots from Process Explorer, a Task Manager replacement for Windows. The Laptop was restarted and the screenshots were taken after around 4-5 hours of casual browsing with around 25 tabs open. I also had a Hadoop installation running on a VMware Player. Simultaneously, I was also dabbling  around with some code in eclipse .

The following were my observations

  • The Browser Process of Chrome is doing over 10 Million Page Faults. The closest that comes to this process is the Microsoft AntiMalware service(Morro engine) at 2 Million. As you can see in screenshot 1(below), the whole top list is dominated by Chome processes.
  • The Flash process has over 70 Million Context Switches(see screenshot 2(below) which was taken 20 odd minutes after screenshot 1), @5000 Context Switches/sec which is quite appalling.  The nearest competitors are Interrupts, Chrome again, Google Talk (Another Windows Application from Google)
  • Lastly,, Mike Arrington’s “Bloat-ware” Blog , one tab of which is taking around 500MB of RAM taking the total RAM occupied by all  Chrome  processes at 1.05Gb (Note: Firefox was really good at this. I seldom get past 400-500 MB on that many number of tabs). Also funny thing is that,  if you compare Techcrunch to Gmail, the latter  just takes 64MB of ram(see Chrome’s Process Manager screenshot above)

Let me teach my readers a quick Computer Science 101. The following explanations are over simplified, so if you are not satisfied with the explanation, go read Wikipedia for a detailed discussion over the same.

Page Fault: When a processor tries to fetch some block of data from the memory, if doesn’t find the data in the memory it tries to get that data from the hard disk. Operating system manages the blocks of memory and moves some of them to the HDD based on Most Recently used criterion. Since you know the over head of a disk access is way larger than a memory access, too many page faults translate to more spinning of the hard disk. Read Wikipedia for a detailed description

Context Switch: In a multiprocess environment like windows. The cost of switching from one process to another process is quite high. If an application keeps switching very fast, that means it’s not getting enough cpu time to do its work, and it is wreaking havoc on the other applications by decreasing their effective cpu time also. Read Wikipedia for a detailed description

Few things come out of this little insight

  • Flash NPAPI plugin is a useless piece of crap. Even after 10 major versions, they still have an ill designed system. (Some people have told me this was specific to chrome as according to them Flash runs fine on Firefox)
  • Chrome even though is very fast in terms of HTML rendering and JavaScript execution. But, it kills the system by the large number of PageFaults and Context Switches it does.
  • Techcrunch seems to be doing more job than a complex web-application like Gmail. Or is it because the 20 odd flash ads they have on their Blog. They could be even doing mouse pointer/user tracking or even Javascript  code Instrumentation. Who knows!(Just Kidding). What ever it is, they need to find and fix the root cause of the problem.

If my readers believe they are also suffering from the same fate as I am, I would request them to put their stats up in the comments section.

I am using Chrome with Flash 10.0 on a Win7 RC ( Yeah I know that it’s a unstable combination. But, even when I was using Chrome 2.0 on Vista SP1 with Flash 9, I had the same problem)

Process ExplorerProcess Explorer2

Download videos from any sharing site(Youtube, Dailymotion, Metacafe, vimeo etc)

There are a lot of ways to download content from a video sharing sites. Most folks rely on applications to download videos. Applications are really unsuitable for 3 reasons

  • Mostly they are specific to one or two sites
  • In some cases they are shareware
  • They have hardcoded techniques for url retrieval which outdate easily.

A slightly better technique for downloading is relying on sites to decode the url for you. This removes the latter 2 limitations of applications. For most of you who have come here Ripzor will suffice .  However the top limitation still remains if you are looking for a site not listed in Also some times the url retrieval technique changes and sites do not update their scripts. I will give you the method to retrieve URLs in an easy manner using every day tools.

Tools required

  1. Firefox
  2. Firebug(or any other debugging tool)
  3. [Optional] Downloader ( like flashget or Downthemall)

Dont get scared from the word debugger. This technique is mostly a misuse of the word debugging 🙂 .  Now after you have installed the above “tools” . Lets start

Step 1: Select a video downloading site like and pick a video. For further screenshots I have picked up this youtube video and this dailymotion video. The same techniques will work on any video site.

Step2: Now open video in Firefox and enable firebug debugging tool from Tools ->Firebug-> Open Firebug in a new window

Open Firebug in Firefox

Open Firebug in Firefox

Step3: Click on the youtube tab and click on Activate Firebug for the firefox tab button in the firebug window

Activate Firebug for Firefox tab

Activate Firebug for Firefox tab

Step 4: Select the Net tab among Console/HTML/ CSS/Script/DOM and Net on the top . Then select the Flash tab. This tab will give you all the web requests that a .swf file will make. Most sites use flash players for playing the video so this tab will work. If the following steps dont work then select the All tab for seeing all requests

Tabs to be used are net and flash

Tabs to be used are net and flash

Step5. Play the video you will get a screen like this in firebug. Now look for requests that look like video requests. For ex. those ending with .flv, .mp4 or in youtube’s case get_video. Right click on that link and select Copy Location. If there is no such link then you need to go back to step3 and select All tab instead of flash.

Copy link location

Copy link location

Step6. Use your downloader to download that video or paste that link in firefox itself to use it for downloading the video. In some cases you will have to close the video site tab to download this video as only one streaming might be allowed.

So now you can download any video from any video sharing site as long as the video is not encrypted with the player. Let me know what all sites you tried below.

Swipeball for iPhone

SwipeBall is fun and addictive yet simple to learn and play! Light up bulbs by bouncing balls off of walls and lines that you draw by swiping your finger on the touchscreen! Balls bounce and change in color as they touch the colored walls.The objective of the game is to light up all bulbs – each with a ball of the same color before time runs out. Light up all bulbs and reach the next level – hit the light bulb with a ball of a different color and you lose. Game gets more exciting and challenging as you complete a level and encounter different wall layouts, colors, speed and number of balls!

SwipeBall is a must have for any user looking for a fun game that gives hours of entertainment and interaction and yet is quick to learn. Different levels of difficulty make it a perfect game to get started on and hone your skills and show off to your friends as you get better. Don’t tell us we didn’t warn you it’s addictive!

Download Now From Apple Appstore for 99 cents


– Multiple levels with different wall colors and layouts
– Touchscreen to create your own barriers while you play
– Sound effects to indicate you have lit a lamp and countdown as time runs out
– Increasing levels of difficulty as you get better
– Twenty-five levels for non-stop fun and action


SwipeBall - Level 20

SwipeBall - Level 20

SwipeBall - Level Selector

SwipeBall - Level Selector

SwipeBall - Level 15

SwipeBall - Level 20

For support please send an email to “swipeball <at> thinking <dot> me” or post a comment here

PS. It took us over 3 months to get accepted into the Apple Developer Program from India.
Download Now From Apple Appstore for 99 cents

Enable Multitouch(two-fingers) on Synaptics trackpad on Linux

This could be your Trackpad

This could be your Trackpad

Very few people know that every synaptics trackpad supports multitouch and as long as your OS supports it  you can always use multitouch to enhance your computers interface. Apple has patented several UI design elements ,hence has inhibited Windows and Linux to come out with similar UI elements like pinch and zoom, scroll etc.   

microsoft-windows-vista-logoIf you are using Windows xp, vista or Windows 7 then click here

signature_color_high_res_0 Install Synaptics driver  in ubuntu

apt-get install  xserver-xorg-input-synaptics 

Now  edit your xorg.conf file

sudo vi /etc/X11/xorg.conf

find the input section

Section “InputDevice” Identifier “Synaptics Touchpad”

Add these lines in it

Option “HorizEdgeScroll” “on”

Option “VertEdgeScroll” “on”
Option “VertTwoFingerScroll” “on
Option “HorizTwoFingerScroll” “on”
Option “CornerCoasting” “on”
Option “PalmDetect” “on”
Option “CircularScrolling” “on”
Option “CircScrollTrigger” “3”