Best of Android 2015: Performance

Everyone always asks what’s the best phone to buy and we’re taking the guesswork out of that question. For this installment in Best of Android 2015, we’re taking a look at performance: which Android smartphone has the best performance? Let’s take a look.

As a reminder, the devices we are testing (in no particular order) are the Sony Xperia Z5 Premium, the LG V10, the Nexus 6P, the Moto X Force, the BlackBerry Priv, and the Samsung Galaxy Note 5.

AnTuTu

AnTuTu is one of the “standard” benchmarks for Android, it measures both CPU and GPU usage as well as a few other things like RAM bandwidth and I/O throughput. Although all of its workloads are completely artificial, meaning they don’t reflect real world usages, the benchmark is useful for establishing a baseline about the general performance of a device.

http://live.amcharts.com/yMzU5/embed/

As you can see the Galaxy Note 5 comes out on top due to the great performance of the Exynos 7420 SoC. However, close behind is the Moto X Force which uses the Snapdragon 810 from Qualcomm. In fact second, third and fourth place are all taken by phones using the Snapdragon 810. Fifth is the LG V10 which uses the hexa-core Snapdragon 808 and in last place comes the Blackberry Priv, which also uses the hexa-core Snapdragon 808. It is interesting to note that five of our six devices all use Qualcomm Snapdragon processors, either the octa-core 810 or the hexa-core 808. However the Exynos 7420 proves yet again that it is the top SoC of 2015.

GeekBench

GeekBench is another popular Android benchmarking tool, however this app only tests the CPU performance. The test is split into two parts, the single-core tests, which measures the speed of an individual core, regardless of how many cores there are on the SoC; and the multi-core tests, which exercises all the cores on the SoC simultaneously.

http://live.amcharts.com/MzIxN/embed/

As with the AnTuTu tests the Note 5 comes out top, however this time we see a strong performance from the Sony Z5 Premium. Its single-core result is only slightly less than the Note 5, as is its multi-core score. What is interesting is that the Moto X Force and the Nexus 6P didn’t match the pace of the Z5 Premium for this test. All things being equal they should have performed better as they are using the same System-on-a-Chip, clocked at the same frequency.

The two hexa-core phones come in fifth and sixth, mainly “hampered” by having only 6 cores rather than 8, something penalized by the multi-core tests.

Basemark OS II

Basemark OS II is an “All-In-One” benchmark that tests the overall performance of a device including system, memory, graphics, and web browsing. As well as these individual scores there is a overall rating calculated.

http://live.amcharts.com/QyNGQ/embed/

Until now a clear pattern was forming: The Note 5 first, followed by the three Snapdragon 810 devices and then the two Snapdragon 808 handsets. However the Basemark OS II test has shaken things up, just a little. The Note 5 is still top and second place goes to the Moto X Force. In third is the Nexus 6P, but in forth comes the BlackBerry Priv, a Snapdragon 808 device, and not the Z5 Premium as you would have expected. The Priv did well on the memory subsystem tests, in fact it got the highest score for that test, even beating the Note 5. At the same time the Z5 Premium did badly on the memory tests. The result is that the overall score for the Priv is higher. This unfortunately leaves the LG V10 in last place.

3D Mark Slingshot

3DMark is probably the world’s leading benchmark tool because of its cross-platform nature and because of its emphasis on 3D graphics. The Slingshot tests are designed specifically for devices running Android 5.0 or later that support OpenGL ES 3.1 or ES 3.0. For this test run we picked OpenGL ES 3.1 to test the latest and most advanced features of the GPU.
http://live.amcharts.com/jZjRh/embed/

And the results are quite surprising, this is the first test where the Note 5 didn’t win. Instead we have a great performance by the Nexus 6P. Although it has the same SoC as the Motorola and Sony, the Nexus 6P managed an incredible score of 1,565. Next comes the Note 5 with 1,252, closely followed by the other two Snapdragon 810 devices with scores of 1,146 and 1,125 for the Moto X Force and the Z5 Premium respectively. The LG V10 and the Blackberry Priv come in fifth and sixth with the low scores of 510 and 502, a third of the Nexus 6P’s scores.

GFX Highest Frames

Like 3D Mark, GFXBench is primarily concerned with GPU performance. The latest version of the app includes tests for OpenGL ES 3.1 and the Android Extension Pack. This means that it can test devices using game-like content that utilizes Android Extension Pack features such as hardware tessellation.

http://live.amcharts.com/jYzZk/embed/

Things return back to the norm with this test, well almost. The Note 5 comes in first, followed by the Z5 Premium and the Nexus 6P. In fourth place is the Moto X Force, but only just. It managed a score of 182.7 Frames, a much lower score than the 390.7 and 338.8 of the Z5 Premium and the Nexus 6P. The Motorola’s score was actually closer to the two Snapdragon 808 devices than to the other Snapdragon 810 handsets.

Lag Test

Benchmarks are one thing, but how long does it take to load your favorite game? This is where the rubber hits the road. The benchmarks mean nothing while you are twiddling your thumbs waiting for a game to load. To test this we started up 6 games in turn on each device and measured how long it took for the game to start. The games we picked were: Brave Frontier, Summoners War, Clash of Clans, Farmville 2, Walking Dead, and Game of Thrones. The results are quite surprising. For example Clash of Clans takes 27 seconds to open on the LG V10, but only 15 seconds on the Note 5.

Here is the average load time for all 6 games:

http://live.amcharts.com/zMjQ0/embed/

A strong performance from the Moto X Force puts it just ahead of the Note 5. Looking at the individual results it seems that the Note 5 struggled slightly with Farmville 2 which took 24 seconds to open, compared to the 19 seconds for the Moto X Force. Next comes the Sony followed by the Nexus and then the two Snapdragon 808 devices.

Over heating?

One of the things that is often talked about concerning smartphones is overheating. The harder the processor needs to work, the more heat it produces. This is true when playing 3D games and when recording video, especially 4K video. To see how much each phone heats up we conducted two tests, a 1080p video recording test and a 4K recording test. In both cases we noted the temperature rise.

Here are the results for the 1080p tests:

http://live.amcharts.com/k3MTN/embed/

The Moto X experiences the highest temperature rise, over 13C, while the Note 5 only heats up a few degrees. The other phones in our group perform well except the BlackBerry Priv, which heats up by 8C.

So what about 4K:

http://live.amcharts.com/2Y3MD/embed/

Recording 4K is more arduous for the processor and it causes most of the devices to heat up significantly. Again the Moto X force heats up the most while the other devices also see a significant temperature rise, with the exception of the Nexus 6P and the Z5 Premium.

As one final test we also looked at how much video, both 1080p and 4K each phone could record. There are three possible limits to these recordings. First, a built-in limit set by the manufacturer. Second, running out of internal storage while recording. Third, the phone overheats and the app shuts down. Here is what we found out:

http://live.amcharts.com/lNThl/embed/

When it comes to 1080p recording the Note 5 is the clear winner with a recording time of over two hours. Next comes the Sony, followed by the BlackBerry. The other three devices manage about 30 minutes each. For 4K recording the story is different. The Note 5, the LG V10 and the BlackBerry Priv all managed just 5 minutes. The Nexus 6P 8 minutes, and the Moto X Force, 10 minutes. However, the winner was the Z5 Premium which managed an impressive 40 minutes.

Wrap up

That is quite a lot of data, so here is my summary. The Note 5 with its octa-core Exynos 7420 processor is clearly the winner. It comes first in nearly every test and on the two occasions when it doesn’t, it came second. Close behind the Note 5 are the Snapdragon 810 based devices. Looking at the scores it is hard to say which one is better than the other two. If I was forced to name one, it would probably be the Moto X Force, but it would be a close call.

At the bottom of the performance table is the LG V10 and the Blackberry Priv. Both use the Snapdragon 808 hexa-core processor and both are fairly evenly matched. However don’t forget that we are comparing the crème de la crème, none of these phones are slow and you won’t be unhappy with the performance of any of them.

Also there is more to a phone than just its raw performance, there are lots of other factors including the display, the audio, the battery, plus unique features like the Priv’s keyboard or the Note 5’s pen.

The bottom line is this, these are all high performance phones and in that sense they are all equal, but as Orwell almost said, it is just that some phones are more equal than others!

Best of Android 2015

Don’t miss:

Best of Android: Display

Best of Android: Audio

Tomorrow we will be taking a look at the battery life of our six contenders, through benchmarks and real world tests. Come back on December 11 at 10AM EST for Best of Android 2015: Battery!

Credits

Post and video by: Gary Sims
Series Contributors: Rob Triggs, Gary Sims, Lanh Nguyen, Joe Hindy, Krystal Lora
Series Editors: Nirave Gondhia, Bogdan Petrovan, Andrew Grush

Best of Android 2015 – Audio

With the smartphone having firmly replaced the iPod as the portable music player of choice, audio playback is an increasingly important factor in today’s high end mobile. Furthermore, with increasingly bold claims of HiFi and even studio quality playback being touted on some smartphones – I’m looking at the LG V10’s ludicrous sounding 32-bit playback option here – we should check to see just how well these claims hold up to some testing.

For this test we’ve also thrown in the ZTE Axon, to see how its “HiFi quality” DAC performs in a smartphone environment and how it stands up to the competition.

Audio Benchmarking

To give us an overview of how each smartphone performs, we put each headphone jack through a Focusrite 2i2 interface and ran the popular RightMark Audio Analyser, checking for frequency response, noise and distortion characteristics.

We’ll start with noise and crosstalk, which should push well through the -96dB noise floor offered by 16-bit audio if any of our high quality DACs want to come close to realising their touted 24-bit or 32-bit performance (a practical impossibility I might add).

http://live.amcharts.com/JjNzY/embed/

Crosstalk isolation is excellent across all of the models and there certainly isn’t any sign of any stereo bleed issues. However, none of our smartphones make a notable move past the -96dB target. The Xperia Z5 Premium clocks in a decent -100dB, while the ZTE Axon falls some way short at just -92dB. It’s by no means noisy, but seems to defeat the objective of offering a 24-bit codec and LG’s 32-bit DAC boasting is just outright ridiculous.

To be fair to the smartphones though, at approaching -100dB we could be starting to push up against the noise limits of our testing environment as well. So to so avoid complicating the results, we’ll simply test with 16-bit audio from here on out.

http://live.amcharts.com/TJkNT/embed/

Moving onto dynamic range, the difference between the loudest and quietest output, we should see all of our smartphones hit the full 96dB dynamic range afforded by our 16-bit test signal. While some smartphones do, three of our models fall short. The Nexus 6P, Xperia Z5 Premium and the ZTE Axon cannot reach the target, meaning that additional noise is likely making its way to the audio signal path somewhere between the DAC and the jack. This isn’t an unexpected result from a compact device with lots of potential for interference, but circuit layout in the leading phones seems to be a little better thought out.

The total harmonic distortion results are excellent across the board, with all of the handsets putting in performance well below 0.004%. The ZTE Axon is the winner by a small margin, followed by the Priv and the V10. The real test for music though is intermodulation distortion, as this shows the amount of distortion generated when playing back multiple tones.

http://live.amcharts.com/xYzU5/embed/

There is a greater disparity in this test, with the LG V10, Nexus 6P and ZTE Axon all taking a lead coming in at or below 0.01%. Here the ZTE Axon is again the winner. The Galaxy Note 5 and Xperia Z5 Premium are very close by on 0.012%, while the Priv and Moto X Force are the worst performers, although by no means are they bad and such small percentage differences are going to be very tough to tell apart.

Frequency response is as flat as you like across all the tested smartphones, at least when connected up to our interface. Interestingly, the Moto X Force, Nexus 6P, BlackBerry Priv and LG V10 seem to have a low pass filter on their DAC output with a -3dB point at just over 18 kHz. This is likely designed to remove any high frequency interference from high speed signals, such as the codec clock, without interfering with the audible spectrum, or could be something in software.

Crunching through the data, it’s definitely a tight run thing in the portable audio space these days, will all the smartphones offering up specifications that are hard to complain about. It’s a bit disappointing to see that none of the higher resolution devices offer a notable bit-depth advantage, but this just highlights the difficulties in fitting low noise audio circuits into a mobile phone, if nothing else. With that said, the “32-bit” LG V10 performs consistently well across all of the tests, followed closely by the Galaxy Note 5 and the HiFi ZTE Axon.

Listening Test

Of course, no audio test would be complete without actually having a listen to the phones in the real world. So I stuck a couple of my favourite FLAC tracks on board and had a good listen through some AKG K550s, a reasonably priced pair of “reference” headphones. Honestly, all of the smartphones performed well, as the scientific testing suggested, but there were some subtle differences between a number of the phones.

The “worst” performers, to my ears, were the 6P and the X Force, as the balance didn’t quite seem to keep up with the other phones. Although tone and detail were good across both models, the X Force seems to throw in some heavy hitting bass that verged on boomy for my tastes. The 6P has the opposite problem, lacking any real punch to kick drums and both were missing that extra sizzle on cymbals that really flesh out a track.

The Priv sounded well balanced and delivered nice tone, but seemed to produce a narrower sound than all of the other handsets. This resulted in an experience that felt a bit more sterile. The ZTE Axon suffered from a similar lack of excitement despite sounding nice and wide, but I put this down some missing punch in the lows. That being said, the ZTE Axon offered excellent detail across the spectrum.

The Galaxy Note 5 stood out the most in terms of a unique tone, as the tracks seem a little warmer here than on other phones. After listening, I took a closer look at the intermodulation distortion results and found an interesting introduction of some second harmonic distortion that can’t be seen on other phones (at least not this noticeably). Although it only appears very slight, cumulatively this might be audible when listening to a full track. The panning and high-end detail are excellent with vocals cutting through nicely, but the extra smoothness might not sit well with everyone.

There appears to be some additional 2nd harmonic content introduced somewhere in the Galaxy Note 5’s signal path, which is more typical of a classic warm valve sound.

The Xperia Z5 is one that the bass junkies will love. It really thumps on kicks but isn’t overbearing and provides excellent details in the mids that holds up well in the highs too. I really couldn’t find much to complain about here but it’s tough to call it notably ahead of the Note 5 or Axon.

This leaves the LG V10 as the stand out smartphone for me, and it was the only one to bring a smile to my face as it managed to conjure up some new sounds from tracks that I thought I knew in and out. The V10 is well balanced across the frequency spectrum and really shines when it comes to detail and spacing. It’s by far the widest sounding phone, which was actually a little disorienting at first, but this really allows for all the small details to come through. Although a 32-bit DAC may be little more than a marketing gimmick from a technical standpoint, the detail and sense of space that comes through with this codec are simply fantastic.

Rear speaker volume test

If you’re planning to take your smartphone out on the road, then the maximum volume of the smartphone’s speaker may also be of interest. We took a peak volume reading right next to the phone’s speaker to see which is the loudest. The chart below shows the result and the typical dB values for common environments.

http://live.amcharts.com/VmZTF/embed/

Given that you’ll be holding the speaker away from you and the actual media volume will be more dynamic than the peak, you want to get notably past the markers to be sure that you can make out quieter details. So if you want to watch films in the car, the Note 5, X Force, Nexus 6P and LG V10 should definitely have you covered.

Remember, decibels are a logarithmic scale and every 10dB increase results in roughly a doubling of the perceived loudness of a sound. This makes the LG V10 the loudest of the bunch by some margin, offering up double the peak volume of BlackBerry Priv, Xperia Z5 Premium and the ZTE Axon. At a whopping 100dB I honestly don’t recommend sticking your ear right up next to the speaker.

Given the decent performance of the ZTE Axon in the headphone tests, the speakers were a major disappointment. We didn’t have any complaints about the other phones, but the Axon’s speaker was horrendously tinny.

Summing up

Taking the whole picture into account, the LG V10 is a pretty clear winner. It comes out as one of the best performers in both the jack and speaker tests, and sounds incredibly good. Second place is a very close run contest, with the Galaxy Note 5, Xperia Z5 Premium, and Nexus 6P all putting in good performances.

The Galaxy Note 5 just edges out ahead in the benchmark and speakers tests, but there’s excellent headphone sound available on the Z5 Premium too. Sadly, the Axon phone’s disappointing dynamic range and speaker results hurt what was otherwise a lovely sounding phone, proving that spec sheet point scoring doesn’t always translate in the real world. Unfortunately, the Moto X Force didn’t stand out in the listening test and had the worst distortion results, so you can certainly do better if you’re looking for a top notch audio experience.

Best of Android 2015

Don’t miss:

Best of Android: Display

Tomorrow we will be taking a look at the performance of our six contenders, through benchmarks and real world tests. Come back on December 10 at 10AM EST for Best of Android 2015: Performance!

Credits

Post by: Rob Triggs
Series Contributors: Rob Triggs, Gary Sims, Lanh Nguyen, Joe Hindy, Krystal Lora
Series Editors: Nirave Gondhia, Bogdan Petrovan, Andrew Grush

Best of Android 2015: Display

We’re always asked which is the best smartphone on the market and which is the one to buy right now. To kick off our Best of Android 2015 series we are going to delve right on into every smartphone’s big upfront piece of tech – its display.

Our test subjects are (in no particular order) the Nexus 6P, Moto X Force, BlackBerry Priv, Xperia Z5 Premium, LG V10 and the Galaxy Note 5. This not only gives us a good look at how Sony’s 4K display stacks up against the QHD resolution that has become the norm in this year’s flagships, but also how the latest and greatest AMOLED panels fare against LCD.

Before we delve into all of the results, a quick word about our test. We paired up an X-rite’s i1 Display Pro spectrophotometer up with with CalMAN’s ColorChecker software and the MobileForge companion app, which is used to wirelessly sync up the test images between the phone and our PC software. We set the phone to our desired brightness for each test, attached the i1 Display Pro to the middle of the display, made sure that it was flush to the screen, and then ran the software.

As well as giving us luminance levels and a gamut plot, the software checks the colors produced by the display against an idealised set and gives us an error value (where 0 is perfect). We looked at the basic pure colors used in the reproduction of all other shades; red, green, blue, cyan, magenta and yellow; along with white, to build a profile of how accurate each display is.

Display Luminance

To start with, we ramped up each phone to full brightness and measured a white light output with the i1 Display Pro. A higher peak brightness is very useful when trying to use your smartphone in bright sunlight, but this comes at the expense of additional battery drain.

The Xperia Z5 Premium is by far the brightest display with a whopping 629 nits, followed by the BlackBerry Priv at 527 nits and the Note 5 with 493 nits. The Moto X Force and the Nexus 6P are a fair bit darker at their peak settings, managing just 385 and 397 nits respectively.

http://live.amcharts.com/yMGM0/embed/

We also thought that it would be interesting to see how each display adjusts between its minimum and maximum settings, to see if the 50 percent phone setting is actually half as bright as the maximum option.

http://live.amcharts.com/MxOWU/embed/

What’s interesting about this result is that the AMOLED smartphones all exhibit pretty much straight lines, so your software brightness setting very closely follows the actual output. The LG V10 and Xperia Z5 Premium both opt for a curved setting, with very little change between 0 and 25 percent, but a much larger swing in the last 25 percent.

The Xperia Z5 Premium, the brightest phone, is actually one of the dimmest until you reach around the 85 percent setting on the handset. It’s the BlackBerry Priv that is actually by far the brightest display on average and the phone should make it easier to find the ideal manual brightness setting for your environment thanks to its linear adjustment.

250 nits calibrated color error

For our first color test we calibrated each of the smartphone displays to 250 nits, a rather typical brightness for viewing in well-lit environments. Here we are looking to see how closely each display can match the ideal shade, as well as the average and maximum error value across all the colors tested.

http://live.amcharts.com/M2ZkM/embed/

The Moto X Force is the clear winner here, with both lower average and maximum errors. This is followed by the LG V10 and the BlackBerry Priv, which both boasts very low average errors. The Xperia Z5, Galaxy Note 5 and the Nexus 6P are all similarly poorer performers here.

To grab a closer look at exactly where each display is performing well and poorly, we can take a look at the error values for each of the colors we tested.

http://live.amcharts.com/RjZmR/embed/

The Galaxy Note 5 shows some of the highest color errors across the field, which makes sense given the regularly discussed saturated blue and greens of Samsung’s display, which often divides consumers. The Nexus 6P also has a very similar profile. The Xperia Z5 Premium is quite different to the other phones which tend to offer accurate blues. Instead the Z5 Premium is the best at reproducing red and green, but has a higher Cyan error than any of the other phones.

The winner in this test is between the LG V10 and the Moto X Force, which both show lower color errors, particularly in the blues and greens. The V10 probably just edges it here though.

The one display part not tested above is whites. A trend common among LCD panels is to see a blue tint to the whites and this can be observed on both the LG V10 and most notable with the Xperia Z5 Premium. This is because LCD backlights are built from a filtered blue light, while AMOLED pixels provide their color independently. For a closer comparison, the graph below plots the average white error across varying display brightness’s.

http://live.amcharts.com/mE3MD/embed/

Interestingly, the LCD panels perform slightly better in this regard when the displays aren’t driven as hard, while the Nexus 6P sees a larger error when it’s display is darker. The BlackBerry Priv has the best whites, although the errors creep up a little when the display is at its darkest or brightest. The Galaxy Note 5 is the most consistent performer, but is a little worse on average than the other AMOLED displays.

The Z5 Premium and LG V10 (LCD) show a notable blue tint on whites compared with the Priv and Note 5 (AMOLED).

Color error range

Not every viewing condition lends itself to a single brightness. To see if the accuracy of any of these displays would shift if you’re out in the sun at full brightness or having a read before bed with the brightness dimmed, we redid the color test at each phone’s 10, 50 and 100 brightness levels.

http://live.amcharts.com/N2YzN/embed/

When it comes to maximum errors we again see a notably poor performance from the Xperia Z5 Premium, which is mostly due to its very poor whites. The Moto X Force again comes out ahead in this test, just beating the LG V10 because of its weaker performance when it comes to whites.

The Priv, Note 5 and Nexus 6P are all pretty close in this test. Also, a common theme across all of the phones is that the maximum color errors tend to improve slightly as you turn the display brightness down.

http://live.amcharts.com/hjNmV/embed/

Looking at an averaged picture, it’s a much closer content, with the Priv, Note 5, V10 and Nexus 6P all performing equally well. The Moto X Force is the most accurate display on average but also seems to vary in accuracy slightly when the display is nearer maximum and minimum brightness settings. By contrast, the Galaxy Note 5 and Nexus 6P vary very little as you adjust the brightness, while the V10 and Z5 again show improvements at lower settings.

Color Gamut

To explain some of the color error results in a little more detail we should have a look at how each phone’s color gamut compares to the test target.

Across all of the phones we see that colors are extended outwards from our test target, suggesting various levels of oversaturation depending on the manufacturer. As the earlier test results suggested, it’s the LG V10 sticks very close to the target around the blues and particularly the yellows, while the Moto X Force maintains quite a linear deviation in the blues and reds. The Priv is also quite a good fit, apart from the slightly overextended AMOLED greens.

The Galaxy Note 5 however is clearly overcompensating with the greens and blues and this extra saturation really takes the display away from our ideal test values, producing an unrealistic result and higher error count. The Nexus 6P also follows this same pattern very closely. That being said, some people do prefer this more saturated look, so this is a bit more of a subjective issue that the error results may have made it seem.

For a closer comparison, we can overlay the gamut results on top of one another, allowing us to see how each phone compares to one another.

Here we can see the familiar Samsung AMOLED oversaturation debate quite clearly. Both the Galaxy Note 5 and the Nexus 6P offer by far the deeper greens and blues when compared with the other smartphones. By extending saturation here, manufacturers are looking to add more pop to their pictures and it’s interesting that Samsung also choses these colors for its user interface.

The BlackBerry Priv and the Moto X Force are virtually identical across the spectrum, and sit right between the Note 5 and the tamer LCD displays when it comes to green saturation. All of the AMOLED panels are much closer to the target colours in the reds, while it’s the LCD based Xperia Z5 Premium and LG V10 that extend saturation out further here.

Conclusion

Overall, it’s quite a close run competition for the most part, although we have to say that, on paper at least, the Xperia Z5 Premium’s 4K display has rather underperformed. The whites carry a very noticeable blue tint and that has really had an impact on its performance across our tests. Although the phone has the brightest display, its profile is not really the most usable either. However, the Z5 Premium is actually a joy to use and to watch back content on, so like the Note 5, datasheet accuracy is not always the most important factor.

We haven’t really commented too much on the BlackBerry Priv, partly because it has been a solid performer in every test. It has vivid colors without veering into over-saturation, there’s no noticeable color tint and it’s brighter on average than the competition for viewing outdoors. Therefore, we have crowned the Priv as the winner in the display test.

It’s unbelievably close between the other handsets, with each offering up a selection of pros and cons. What the Moto X Force and Nexus 6P lack in brightness they make up for in color and white accuracy. The LG V10 may miss the mark on whites, but it has one of the most accurate displays otherwise. The Note 5’s heavy saturation continues to make it a contentious handset, but it was certainly a strong all-round performer as well.

Best of Android 2015

Tomorrow we will be listening to our six contenders to hear which one offers the best audio experience. Come back on December 9 at 10AM EST for Best of Android 2015: Audio!

Credits

Video / Post by: Lanh Nguyen & Rob Triggs
Series Contributors: Rob Triggs, Gary Sims, Lanh Nguyen, Joe Hindy, Krystal Lora
Series Editors: Nirave Gondhia, Bogdan Petrovan, Andrew Grush

Best of Android 2015: Display

We’re always asked which is the best smartphone on the market and which is the one to buy right now. To kick off our Best of Android 2015 series we are going to delve right on into every smartphone’s big upfront piece of tech – its display.

Our test subjects are (in no particular order) the Nexus 6P, Moto X Force, BlackBerry Priv, Xperia Z5 Premium, LG V10 and the Galaxy Note 5. This not only gives us a good look at how Sony’s 4K display stacks up against the QHD resolution that has become the norm in this year’s flagships, but also how the latest and greatest AMOLED panels fare against LCD.

Before we delve into all of the results, a quick word about our test. We paired up an X-rite’s i1 Display Pro spectrophotometer up with with CalMAN’s ColorChecker software and the MobileForge companion app, which is used to wirelessly sync up the test images between the phone and our PC software. We set the phone to our desired brightness for each test, attached the i1 Display Pro to the middle of the display, made sure that it was flush to the screen, and then ran the software.

As well as giving us luminance levels and a gamut plot, the software checks the colors produced by the display against an idealised set and gives us an error value (where 0 is perfect). We looked at the basic pure colors used in the reproduction of all other shades; red, green, blue, cyan, magenta and yellow; along with white, to build a profile of how accurate each display is.

Display Luminance

To start with, we ramped up each phone to full brightness and measured a white light output with the i1 Display Pro. A higher peak brightness is very useful when trying to use your smartphone in bright sunlight, but this comes at the expense of additional battery drain.

The Xperia Z5 Premium is by far the brightest display with a whopping 629 nits, followed by the BlackBerry Priv at 527 nits and the Note 5 with 493 nits. The Moto X Force and the Nexus 6P are a fair bit darker at their peak settings, managing just 385 and 397 nits respectively.

http://live.amcharts.com/yMGM0/embed/

We also thought that it would be interesting to see how each display adjusts between its minimum and maximum settings, to see if the 50 percent phone setting is actually half as bright as the maximum option.

http://live.amcharts.com/MxOWU/embed/

What’s interesting about this result is that the AMOLED smartphones all exhibit pretty much straight lines, so your software brightness setting very closely follows the actual output. The LG V10 and Xperia Z5 Premium both opt for a curved setting, with very little change between 0 and 25 percent, but a much larger swing in the last 25 percent.

The Xperia Z5 Premium, the brightest phone, is actually one of the dimmest until you reach around the 85 percent setting on the handset. It’s the BlackBerry Priv that is actually by far the brightest display on average and the phone should make it easier to find the ideal manual brightness setting for your environment thanks to its linear adjustment.

250 nits calibrated color error

For our first color test we calibrated each of the smartphone displays to 250 nits, a rather typical brightness for viewing in well-lit environments. Here we are looking to see how closely each display can match the ideal shade, as well as the average and maximum error value across all the colors tested.

http://live.amcharts.com/M2ZkM/embed/

The Moto X Force is the clear winner here, with both lower average and maximum errors. This is followed by the LG V10 and the BlackBerry Priv, which both boasts very low average errors. The Xperia Z5, Galaxy Note 5 and the Nexus 6P are all similarly poorer performers here.

To grab a closer look at exactly where each display is performing well and poorly, we can take a look at the error values for each of the colors we tested.

http://live.amcharts.com/RjZmR/embed/

The Galaxy Note 5 shows some of the highest color errors across the field, which makes sense given the regularly discussed saturated blue and greens of Samsung’s display, which often divides consumers. The Nexus 6P also has a very similar profile. The Xperia Z5 Premium is quite different to the other phones which tend to offer accurate blues. Instead the Z5 Premium is the best at reproducing red and green, but has a higher Cyan error than any of the other phones.

The winner in this test is between the LG V10 and the Moto X Force, which both show lower color errors, particularly in the blues and greens. The V10 probably just edges it here though.

The one display part not tested above is whites. A trend common among LCD panels is to see a blue tint to the whites and this can be observed on both the LG V10 and most notable with the Xperia Z5 Premium. This is because LCD backlights are built from a filtered blue light, while AMOLED pixels provide their color independently. For a closer comparison, the graph below plots the average white error across varying display brightness’s.

http://live.amcharts.com/mE3MD/embed/

Interestingly, the LCD panels perform slightly better in this regard when the displays aren’t driven as hard, while the Nexus 6P sees a larger error when it’s display is darker. The BlackBerry Priv has the best whites, although the errors creep up a little when the display is at its darkest or brightest. The Galaxy Note 5 is the most consistent performer, but is a little worse on average than the other AMOLED displays.

The Z5 Premium and LG V10 (LCD) show a notable blue tint on whites compared with the Priv and Note 5 (AMOLED).

Color error range

Not every viewing condition lends itself to a single brightness. To see if the accuracy of any of these displays would shift if you’re out in the sun at full brightness or having a read before bed with the brightness dimmed, we redid the color test at each phone’s 10, 50 and 100 brightness levels.

http://live.amcharts.com/N2YzN/embed/

When it comes to maximum errors we again see a notably poor performance from the Xperia Z5 Premium, which is mostly due to its very poor whites. The Moto X Force again comes out ahead in this test, just beating the LG V10 because of its weaker performance when it comes to whites.

The Priv, Note 5 and Nexus 6P are all pretty close in this test. Also, a common theme across all of the phones is that the maximum color errors tend to improve slightly as you turn the display brightness down.

http://live.amcharts.com/hjNmV/embed/

Looking at an averaged picture, it’s a much closer content, with the Priv, Note 5, V10 and Nexus 6P all performing equally well. The Moto X Force is the most accurate display on average but also seems to vary in accuracy slightly when the display is nearer maximum and minimum brightness settings. By contrast, the Galaxy Note 5 and Nexus 6P vary very little as you adjust the brightness, while the V10 and Z5 again show improvements at lower settings.

Color Gamut

To explain some of the color error results in a little more detail we should have a look at how each phone’s color gamut compares to the test target.

Across all of the phones we see that colors are extended outwards from our test target, suggesting various levels of oversaturation depending on the manufacturer. As the earlier test results suggested, it’s the LG V10 sticks very close to the target around the blues and particularly the yellows, while the Moto X Force maintains quite a linear deviation in the blues and reds. The Priv is also quite a good fit, apart from the slightly overextended AMOLED greens.

The Galaxy Note 5 however is clearly overcompensating with the greens and blues and this extra saturation really takes the display away from our ideal test values, producing an unrealistic result and higher error count. The Nexus 6P also follows this same pattern very closely. That being said, some people do prefer this more saturated look, so this is a bit more of a subjective issue that the error results may have made it seem.

For a closer comparison, we can overlay the gamut results on top of one another, allowing us to see how each phone compares to one another.

Here we can see the familiar Samsung AMOLED oversaturation debate quite clearly. Both the Galaxy Note 5 and the Nexus 6P offer by far the deeper greens and blues when compared with the other smartphones. By extending saturation here, manufacturers are looking to add more pop to their pictures and it’s interesting that Samsung also choses these colors for its user interface.

The BlackBerry Priv and the Moto X Force are virtually identical across the spectrum, and sit right between the Note 5 and the tamer LCD displays when it comes to green saturation. All of the AMOLED panels are much closer to the target colours in the reds, while it’s the LCD based Xperia Z5 Premium and LG V10 that extend saturation out further here.

Conclusion

Overall, it’s quite a close run competition for the most part, although we have to say that, on paper at least, the Xperia Z5 Premium’s 4K display has rather underperformed. The whites carry a very noticeable blue tint and that has really had an impact on its performance across our tests. Although the phone has the brightest display, its profile is not really the most usable either. However, the Z5 Premium is actually a joy to use and to watch back content on, so like the Note 5, datasheet accuracy is not always the most important factor.

We haven’t really commented too much on the BlackBerry Priv, partly because it has been a solid performer in every test. It has vivid colors without veering into over-saturation, there’s no noticeable color tint and it’s brighter on average than the competition for viewing outdoors. Therefore, we have crowned the Priv as the winner in the display test.

It’s unbelievably close between the other handsets, with each offering up a selection of pros and cons. What the Moto X Force and Nexus 6P lack in brightness they make up for in color and white accuracy. The LG V10 may miss the mark on whites, but it has one of the most accurate displays otherwise. The Note 5’s heavy saturation continues to make it a contentious handset, but it was certainly a strong all-round performer as well.

Best of Android 2015

Tomorrow we will be listening to our six contenders to hear which one offers the best audio experience. Come back on December 9 at 10AM EST for Best of Android 2015: Audio!

Credits

Video / Post by: Lanh Nguyen & Rob Triggs
Series Contributors: Rob Triggs, Gary Sims, Lanh Nguyen, Joe Hindy, Krystal Lora
Series Editors: Nirave Gondhia, Bogdan Petrovan, Andrew Grush