How to: use "adb sideload" on your Android device

Probably every Android power-user at least once in his life used ADB - Android Debug Bridge. It is a versatile command line tool that lets you communicate with an emulator instance or connected Android-powered device. Using ADB shell commands gives you additional control over your device and sometimes it can save your device from being bricked (example: How to: copy ROM zip file to the freshly wiped device). You can find some more basic information about ADB here.

Since Android Jelly Bean there has been a new ADB mode available in the AOSP recovery, incorporated by the Android developer community into custom recoveries too. It is called "ADB sideload" and most of you probably have heard about it already. This is an alternate method to the one I wrote about here - How to: copy ROM zip file to the freshly wiped device. The main difference is that ADB sideload works only with recoveries based on Jelly Bean source or newer. I believe that ADB sideload was created to simplify the process of flashing/restoring Android update.zip packages.

Keep in mind that while using ADB sideload, the regular ADB shell won't work. To be able to use SIDELOAD mode make sure you're running latest ADB drivers from the Android SDK (Platform-tools). Here are the simple steps you need to follow to flash update.zip package using adb sideload mode (based on stock Android recovery):

1. Place the ZIP package you want to install in the same location where you keep ADB drivers - adb.exe, AdbWinApi.dll and AdbWinUsbApi.dll (usually it's SDK\platform-tools)
2. Make sure you have USB debugging enabled in Settings > Development on your device
3. Make sure your phone drivers are installed on the PC you're going to use
4. Boot your device in recovery mode (Android logo with a exclamation mark) and connect your device to PC
5. Hold down "power" button first, followed quickly by "volume up" button. You should now see the recovery menu
6. Use the volume up/down keys to select "apply update from ADB," then press power to select it
7. Open a command prompt on the PC (cmd.exe), type and confirm with ENTER:
8. cd /d <adb.exe location> (for example: cd /d c:\SDK\platform-tools) or you can open your SDK/platform-tools folder, then press SHIFT button and the right-click mouse button and choose “Open command prompt here”
9. adb sideload <filename>.zip (for example: adb sideload update.zip)
10. The zip package will begin installing. When it's done, select "reboot system now."

How is that different from the alternative method? You don't have to manually create the proper folders structure, push the file and later install if from inside the recovery menu. The result is basically the same, because ADB sideload is also transferring the zip file into the device internal memory and later it automatically begins the installation procedure. However, it works only with recoveries based on Android Jelly Bean source.

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!

How to: manually install an OTA update on your HTC device

Sometimes it turns out that your device for some reason can't find the most recent OTA (Over The Air) update. However, there is a solution to apply OTA update manually, without being forced to wait for the update being downloaded and installed by your device.

Facts you need to know before continuing:

1. This method has nothing to do with the "adb sideload" feature available on Android devices since Jelly Bean. It should work on every Android HTC device.
2. Read about different software versions for the HTC devices here. Keep in mind that the OTA update must match the software version on your device. Each original OTA update.zip package has 2 different software versions in its name - the version of the software that must be currently installed on your device and the version of the software that your device will run after the update. For example:

OTA_M7_UL_JB43_SENSE55_MR_HTC_Europe_3.62.401.1-2.24.401.8_release_338160gxmo5sd9337kadux.zip

As you can see, 2.24.401.8 is the software version you must currently have installed to be able to run the OTA and 3.62.401.1 is the version of the software expected after the update. You can't flash an OTA update on a different software version to that expected by the OTA update for two main reasons:

1. Formal reason - the OTA update.zip package checks if your system is running the expected software version before it starts to install the update:

   assert(file_getprop("/system/build.prop", "ro.build.fingerprint") == "htc/htc_europe/m7:4.2.2/JDQ39/235216.8:user/release-keys");

   This must match the following build fingerprint from the build.prop. If the versions don't match, the installation of the OTA package will be cancelled.
   

   ro.build.fingerprint=htc/htc_europe/m7:4.2.2/JDQ39/235216.8:user/release-keys

2. Technical reason - OTA update.zip packages usually contain patches for the target files. This way OTA update doesn't have to be 1GB size, even if it is supposed to update the whole OS to the newer version. Patch files have an *.apk.p extension (f.g. Camera.apk.p) and each patch file must match the target file (f.g. Camera.apk). To make it possible, OTA update.zip package checks the MD5 checksum of each file on the system partition on your device:

   assert(apply_patch_check("/system/app/Camera.apk",

   "b3b79e40bad0b1bec5fb949ac5f8662f91bee714",

   "5a5bf63e3d7c9de52a7d8d7a292e158263783844"));

   And finally, if the MD5 is correct it applies the patch. All these commands are stored in META-INF\com\google\android\updater-script of each OTA update.zip package. If the MD5 is incorrect, or at least one file is missing, the installation of the OTA package will be cancelled.
   

   assert(apply_patch("/system/app/Camera.apk", "-",
   b3b79e40bad0b1bec5fb949ac5f8662f91bee714, 4229394, 
   5a5bf63e3d7c9de52a7d8d7a292e158263783844,
   package_extract_file("patch/system/app/Camera.apk.p")));

Now you are ready to manually apply the OTA update on your device. The method you'll use is based on Android stock recovery.

1. Make sure you have the stock (original) recovery flashed on your device
2. Make sure you're running a completely unmodified operating system. Additional files like Superuser.apk or su binary are allowed, you just can't have any system files changed or removed (due to the MD5 check)
3. Make sure you have a reasonable amount of battery charge remaining
4. Make sure the OTA update you're about to install matches the software version on your device
5. Copy the OTA update.zip package to your device (internal storage preferred)
6. Turn OFF the device (make sure fastboot mode is disabled in settings)
7. Hold your volume down and power keys until the bootloader starts up
8. Using the volume keys, navigate down to RECOVERY and press power
9. You are now in stock recovery mode. You should see nothing more than a (sometimes spinning) Android with the red exclamation mark above it (an empty black screen is also possible). Wait for a few seconds
10. First hold volume up, then (with volume up held down) press power button to enter the main recovery menu. This keys combination may vary from device to device. Try different keys combination if the above one doesn't work (might be all three keys all together too)
11. Navigate down to "apply from phone storage" text and press power to confirm
12. Navigate to the location where you copied the OTA update.zip package and press power to confirm
13. Wait (sometimes you might need to follow further instructions on the screen) until the update process is finished
14. Once system is booted you should see a confirmation about the successfully flashed update.

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!

HTC ROM Update Utility (RUU) variants explanation

Owners of HTC devices might be a little confused when it comes to understanding the current method of numbering used by HTC to mark their software updates. Here you can find a short explanation of this matter.

First of all, we all know that HTC sells their devices all over the world. Every carrier and region of the world has some more or less different "local settings". By "local settings" I mean things like: emergency call numbers, languages, time-zone settings, network settings, carriers branding, GPS settings and more. Because of that each software for even the same device must be different.

Secondly, even the same device might use different hardware related to the region of the world. For example, HTC One is available in at least 6 different hardware configurations. One of the main difference is modem version. GSM version of the HTC One uses MDM8215 chip and LTE variant has MDM9215. Verizon (US) is known to have MDM9615. So that's already 3 different versions of the same device.

Every HTC update has the following numbering structure:

a.bb.ccc.d

a - the first number of the software. Normally it doesn't increase very often, but when it does, it usually means newer Android version.

bb - this number is also a software number. Increasing bb number means some more or less major update based on the same Android version, but sometimes it might even bring newer version of the HTC Sense.

ccc (also cc or cccc) - this is the number of the region/carrier. It has nothing to do with the software itself being newer or older. More details below.

d (sometimes dd) - the third part of the software version (the fourth one in the RUU name). Usually indicates some minor update with some hot fixes.

Example: 3.20.401.3

So basically, when you are interested in the software version, you should look at a.bb.d (3.20.ccc.3) because ccc (401) means only the region/carrier variant of the software.

Here is the list I was able to gather of possible RUU variants:

a.bb.61.d - Orange UK (United Kingdom)

a.bb.65.d - Orange CH (Switzerland)

a.bb.69.d - Orange PL (Poland)

a.bb.73.d - Orange FR (France)

a.bb.75.d - Orange ES (Spain)

a.bb.110.d - T-Mobile UK (United Kingdom)

a.bb.111.d - T-Mobile DE (Germany)

a.bb.112.d - T-Mobile AT (Austria)

a.bb.113.d - T-Mobile CZ (Czech Republic)

a.bb.118.d - T-Mobile PL (Poland)

a.bb.161.d - Vodafone UK (United Kingdom)

a.bb.163.d - Vodafone FR (France)

a.bb.165.d - Vodafone IT (Italy)

a.bb.166.d - Vodafone CH-DE (Switzerland - Germany)

a.bb.168.d - Vodafone GR (Greece)

a.bb.169.d - Vodafone AT (Austria)

a.bb.206.d - O2 UK (United Kingdom)

a.bb.207.d - O2 DE (Germany)

a.bb.401.d - World Wide English (WWE)

a.bb.415.d - Arabic (UAE/Middle East)

a.bb.468.d - Turkey

a.bb.497.d - Morocco

a.bb.498.d - WWE Brightpoint US (United States)

a.bb.502.d - Cingular US (United States)

a.bb.531.d - T-Mobile US (United States)

a.bb.631.d - Rogers Canada (WWE)

a.bb.651.d - Sprint US (United States)

a.bb.661.d - Telus WWE (Canada)

a.bb.666.d - Bell Canada

a.bb.707.d - Asia (WWE)

a.bb.708.d - Asia HK (Hong Kong)

a.bb.709.d - Asia TW (Taiwan)

a.bb.720.d - Asia (India)

a.bb.728.d - Virgin AU (Australia)

a.bb.751.d - Chunghwa Telecom (Taiwan)

a.bb.771.d - Hutchison 3G UK (United Kingdom)

a.bb.777.d - ?

a.bb.841.d - Telsra WWE (World Wide English)

a.bb.862.d - Voda-Hutch AU (Australia)

a.bb.901.d - TIM IT (Italy)

a.bb.911.d - SK Telecom (South Korea)

a.bb.980.d - Optus AU (Australia)

a.bb.999.d - HTC Test RUU

a.bb.1400.d - China

a.bb.1401.d - ?

a.bb.1403.d - HTC China CMCC (China Mobile Communications Corporation)

a.bb.1540.d - HTC One U.S. Developer Edition

a.bb.1600.d - ?

a.bb.1700.d - HTC One Google Edition

The list is quite long, and I'm sure there are still some some missing. If you know more variants, please let me know in the comments section.

The 401 and the 707 devices are usually the first ones that HTC prepare updates for. Carrier branded software is usually months after the WWE updates.

Finally, keep in mind that each software version is connected with the CID (Carrier ID) and MID (Model ID) of your device. That means you can't use whichever version you want, unless you have S-OFF on your device. Each ROM Update Utility.exe package will check your device details (bootloader version, MID and CID) before starting the update process.

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!

How to: copy ROM zip file to the freshly wiped device

Have you ever been in a situation where you mistakenly (or not) wiped the content of the internal storage (including system partition and virtual SD card) on your phone or tablet and unfortunately it doesn't support external storage? Now you can't flash desired custom ROM because you removed it and the device can't be booted due to wiped system partition. Don't worry - it's very easy to fix!

Here are the steps you need to follow to copy the correct rom.zip package to your internal storage:

1. Download this mini-sdk package and extract it to c:\mini-sdk
2. Rename the custom ROM package you want to flash to "rom.zip" and copy it to c:\mini-sdk
3. Connect your device to the PC
4. Boot your device in recovery mode
5. Open a command prompt on the PC (cmd.exe), type and confirm each command with ENTER:
6. cd /d c:\mini-sdk
7. adb shell
8. mount /data (error might mean that /data partition is already mounted)
9. on Android 4.2 and above: mkdir -p /data/media/0
10. on Android 4.1.2 and below: mkdir -p /data/media
11. exit
12. on Android 4.2 and above: adb push rom.zip /data/media/0
13. on Android 4.1.2 and below: adb push rom.zip /data/media
14. Wait for a few minutes until the file is copied. It will looks like command shell is not responding, but that means it's still copying. Wait patiently until you'll see blinking cursor again.
15. Depends on your custom recovery, use the navigation keys to flash the custom rom.zip package in a typical way you used to flash custom ROMs. It should be on your internal storage now. Done!

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!

Android terminology explanation and why ROOT isn't a crime

This Android terminology dictionary will be growing in time. If you have any suggestions for terms to be included here, please leave a proper comment below this article. The purpose of this dictionary is to explain basic and fundamental Android terminology to a regular "noob" Android user. It is supposed to be written in language that everyone can understand.

Related articles:

• Do we really need S-OFF?
• Virtual SD card on Android
• Unlocking bootloader or flashing custom ROM doesn't void your warranty

ROOT (Superuser) - special user account used for system administration with the highest possible access permissions. The root/superuser account on Linux/Android OS has the same purpose as the Administrator account on MS Windows. For some reason however the term "root" has a very pejorative meaning in the Android world. Many users believe that "root" (or "rooting") is a forbidden fruit that might brick their Android device, void its warranty, make it unstable, unsecured or even explode in some situations.

What is the truth? Very simple - just go to the Control Panel on your PC/notebook and in User Accounts check the type of your account. Is it "Administrator"? Ouch! You're rooted! I really hope you can live with that. Anyway, it seems that having account with an Administrator rights on the Windows powered machine is nothing unusual. What you need to realize, is that having Android powered device without root access is like using the "Guest" account on your Windows computer.

So what will root access give you? Well, it might give you everything, but might give you nothing too. Basically it all depends on what the user would like to do with his device. Root access by itself doesn't change anything. Just like Administrator account on Windows, you can use its capabilities and change almost everything on your device or you might just keep using the OS as it is.

To add root access to Android device you need 2 files - Superuser.apk and su binary. The su binaries people commonly use are modified to latch onto a companion app, which prompts the user when an app requires root access.

ROM (proper definition) - Read Only Memory. Actually it has nothing to do with what most people think. What does an internet definition say about it? "Read-Only memory is a class of storage medium used in computers and other electronic devices. Data stored in ROM cannot be modified, or can be modified only slowly or with difficulty, so it is mainly used to distribute firmware".

ROM (wrong definition) - most people call Android installation package a "ROM" or a "custom ROM". What we call a "ROM" is just a content of /data, /system and boot partitions. Their technical name is "update package". Flashing this package is nothing other than just replacing particular files on device partitions. Where do these files come from?

1. Developers can compile builds from Android Open Source Project (AOSP) or similar open source projects like Android Open Kang Project (AOKP) or CyanogenMod (CM).
2. They can modify closed source vendor images (HTC, Samsung, Sony, LG) by extracting factory system partitions, images (i.e. system.img) or dumping the system partition directly from the device.

What does a typical "custom ROM" looks like? It's a .zip package with files organized in a specific order - applications, framework, binaries, libraries and other files required for proper system work. Imagine a .zip package with "Windows, Users and Program Files" folders inside. That, if we massively simplify it, is what an Android "custom ROM" package is made of.

"Custom ROMs" usually contain root access, so flashing a "custom ROM" results in rooting the device, BUT rooting the device does not always mean flashing a "custom ROM" because you can add root access on a stock system too by adding the su (superuser) binary.

The best name for Android "custom ROM" would be a “modified system zip package”. Why .zip? Because update.zip packages were originally designed for over the air (OTA) updates. So both OTA update and custom "system zip packages" have more less the same construction - system files with the proper update-binary/updater-script (inside META-INF), compressed into ZIP archive. The only method to install (or "flash") ZIP package is via Android recovery.

Boot - The boot.img file is commonly mistaken for the Linux kernel that powers Android. It is actually the image containing the kernel and a ramdisk. The kernel is the fundamental layer of the operating system that controls the input and output of data in the system, it mounts the ramdisk during the boot sequence.

RUU - ROM Update Utility used by HTC to flash/update firmware on HTC device. Again - not very fortunate naming. In fact RUU does contain some "low level" firmware like bootloader, radio, touch panel drivers but it also contains entire EXT3/4 partitions like system or data, which surely can't be considered "Read Only Memory".

Firmware - a combination of persistent memory, program code and data stored in it. "Firmware" is actually a better term than "ROM", but many people are used to calling "Firmware" everything else that comes in the RUU except from system.img. The content of so called "firmware.zip" (called that by HTC itself) was described in this article.

S-OFF/S-ON - S-OFF stands for "Security OFF" and S-ON for "Security ON". It's a term specific to HTC devices (and refers to digital signature checking on the bootloader "hboot"). Retail devices always come with SHIP S-ON locked bootloaders. Hboot can also be found in an engineering version (ENG as opposed to SHIP), but it's not easy to get such a device.

Bootloader - a computer program that loads an operating system and has all the parameters for how to boot the OS. Bootloader is located on a separated partition on your device and is responsible for a low level security on your device. Extended bootloader commands (usually executed via fastboot protocol) can basically manage the whole device, including possibility to gather all device sensitive details (like IMEI, software version, serial number and more), flashing a new operating system or recovery, updating bootloader and other parts of the firmware or even formatting EXT partitions. This is why some vendors (like HTC) lock the bootloaders on their devices to avoid giving end-users too much control over the device.

Fastboot - diagnostic protocol used primarily to modify the flash filesystem via a USB connection from host computer. After enabling the protocol on the device itself (entering "fastboot" mode from inside the bootloader), it will accept a specific set of commands sent to it via USB using a command line, for example "fastboot flash boot boot.img" or "fastboot erase cache".

Android Recovery - a separate partition on your device responsible for installing original OTA updates (.zip) and performing some basic operations on partitions (wiping, formatting, mounting). In some way Android recovery is a sort of mini-OS because it has its own kernel (so it boots independently from the main OS) and on custom recoveries (like 4EXT Recovery, TWRP Recovery) you can run simple applications like AROMA installer or AROMA Manager. Custom recoveries also provide touch support and many advanced features like partition conversion (EXT3--->EXT4), Nandroid backup, possibility to use themes, different wipe and format options, permissions fix and more.

Android Recovery can be based on open source or closed source code. The Android kernel inside the recovery must be always based on open source code.

CID/MID - CID (Carrier ID) and MID (Model ID) numbers are specific tags used by HTC to match a particular device with proper software updates. Many Android devices need different settings, hardware drivers and applications depends on the region of the world or carrier. One device can be available in a few different variants, for example HTC One has 6 different models IDs (MIDs) - PN0711000 (Taiwan), PN0714000 (Austraila, Hong-Kong, Singapore), PN0710000 (Europe), PN0712000 (United States and Canada), PN0771000 (China) and PN0720000 (U.S. Sprint). To check your current MID/CID number you need to use fastboot protocol and type "fastboot getvar all" command or respectively "fastboot getvar mid" / "fastboot getvar cid" if you don't want to see all variables output.

NANDroid backup - an 1:1 copy of the /system, /data, /cache and boot partitions. NANDroid backup can be created and restored using custom Android recoveries like CWM, EXT4 or TWRP. The form of NANDroid backup vary from different custom recovery - could be .tar, .zip, .img or other. NANDroid backup is saved on your SD-card (either virtual or physical) and can be easily transferred to the PC. It's highly recommended to do a regular backup, but to be even more secured, remember about having backup of the backup too.

Article contributors:

• Shen Ye
• Want to became an article contributor? Write a comment with an explained term in the the comments section.

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!

How to: fix a corrupted DATA partition on the HTC One

Since the HTC One was released, I've had a few users asking me to help them fix their broken device. They couldn't boot their system or access anything on the DATA partition from inside custom recovery. Why this happens so often on the HTC One I have no idea...

Here is a short guide explaining how to fix this problem without sending the device for repair under warranty.

Short Explanation

This method can be used to fix any other partition too, just remember about changing the mount point number - from "mmcblk0p37" to the relevant one. For example, the SYSTEM on the HTC One is "mmcblk0p35". In some rare cases the mkfs.ext4 attributes (like dir_nlink or resize_inode) might change too! To get a list of all eMMC mount points on HTC devices use the command cat /proc/emmc. Never format RAW partitions! This method can be used only to format EXT4 partitions. Not following the guide properly might end in permanent damage to your device. I do not take any responsibility for that.

Some people might also find this helpful in case of need to remove the entire contents of the virtual SD-card. As long as the virtual SD-card is not a separate partition, (i.e. it is emulated space on the DATA partition [/data/media]) it can't be formatted - only wiped clean. I wrote an in-depth explanation of how the virtual SD-card works on Android here - Virtual SD card on Android.

How Do I Recognise The Problem?

When trying to mount the DATA partition from inside the custom recovery using the mount /data command you will most likely see this output: mount: mounting /dev/block/mmcblk0p37 on /data failed: Invalid argument

To be 100% sure you can check filesystem structure with the following command: /sbin/e2fsck -n -f /dev/block/mmcblk0p37

Repair Process

Note: the content of your DATA partition will be lost, including the contents of your virtual SD-card! I also assume you already have custom recovery flashed and you are a Windows user - LINUX people please amend as needed.

1. Download this mini-sdk package and extract it to c:\mini-sdk
2. Download this mkfs.ext4 binary and put it into c:\mini-sdk
3. Connect your device to the PC
4. Boot your device in recovery mode
5. Open a command prompt on the PC (cmd.exe), type and confirm each command with ENTER:
6. cd /d c:\mini-sdk
7. adb push mkfs.ext4 /tmp
8. adb shell
9. chmod 777 /tmp/mkfs.ext4
10. /tmp/mkfs.ext4 -b 4096 -O ^huge_file,^dir_nlink,^ext_attr,^resize_inode,^extra_isize -m 0 /dev/block/mmcblk0p37

Image of the whole command in one line to avoid mistakes:

Now your DATA partition will be formatted. It is not the same as "wiping", which is only removing all or just some files from particular partition. Formatting means that the entire partition will be re-created with the above attributes. You should see the following output:

mke2fs 1.41.12 (17-May-2010)
Filesystem label=  
OS type: Linux 
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1703936 inodes, 6815744 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
208 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768,98304,163840,229376,294912,819200,884736,1605632,2654208,4096000
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 25 mounts or 180 days, whichever comes first. 
Use tune2fs -c or -i to override.

Your DATA partition is now completely empty, so we need to copy our particular ROM zip file there:

1. Copy ROM zip file to c:\mini-sdk and rename it to "rom.zip"
2. In the same command prompt window type and confirm each command with ENTER:
3. mount /data
4. mkdir -p /data/media/0
5. exit
6. adb push rom.zip /data/media/0

Copying rom.zip to the device will take a few minutes. Wait until it's done. The output should be: xxxx KB/s (xxxxxxxxxx bytes in xxx.xxxs). Now you can flash rom.zip from inside custom recovery. Done!

Have any questions or comments? Feel free to share! Also, if you like this article, please use media sharing buttons (Twitter, G+, Facebook) below this post!