- A video playback performance comparison between Android 4.3 Jelly Bean and 4.4 KitKat
with Nexus 7
Video playback quality is an important measure in today’s mobile multimedia devices. These
devices are used for watching videos from various sources like Netflix, Viaplay, HBO or
YouTube. Consumers are very sensitive about video quality and do not accept poor
performance like skipped frames, pausing or jerking.
The frame rate of most videos is 24-30 fps, but a speed of 60 fps is a basic requirement for
tablets and is becoming more and more common. Today Samsung Galaxy S4 ja Apple iPhone
5S are already using max. 120fps speed. Watching of for example HFR (High Frame Rate)
content produced by GoPro and other action cameras as well as watching of videos captured
from games is getting more popular.
Video playback performance measurements with OptoFidelity VideoMultimeter
OptoFidelity measured video playback performance of the Nexus tablets by using
OptoFidelity VideoMultimeter measurement device. OptoFidelity VideoMultimeter is a
professional measurement solution for measuring the true and objective video playback
performance of mobile, tablet or any multimedia device. It is a perfect tool for R&D design
verification, test laboratory use and other R&D applications. You can, for example, define
the minimum, maximum, and average frame rate your display truly shows, or take a look at
the frame rate as a function of time. With this, you can also find faults in design and
problems in the interoperation between the different processes running in your device
under test.
OptoFidelity video performance measurement on YouTube:
http://www.youtube.com/watch?v=ennIq8FCod8
Image 1: OptoFidelity VideoMultimeter
Tests were done for Nexus 7 tablets version 2012 and version 2013 both with old Android
4.3 Jelly Bean and new KitKat 4.4 operating system. The operating system was updated to
both devices as an OTA (Over The Air) update by WiFi connection.
In this test we measured the playback of a HD video (24 fps) from YouTube and the playback
of a local video stored to the device with the following image qualities and speeds:
Video resolution 720 p, speed 30 fps
Video resolution 1080 p, speed 30 fps
Video resolution 720 p, speed 60 fps
Video resolution 1080 p, speed 60 fps
Test videos were generated with OptoFidelity Test Video Generator.
Every video playback measurement was repeated several times, and an average score was
taken to overall results. The results were generated by giving error points, so in overall
results a smaller score is better, as it means the product got less error points. If the video
playback couldn’t be done at all, the product got 500 error points.
Three qualities were measured in the test. First of all we measured deviation, which shows
to user as jerking. The target value for deviation is 0 msec. In this test deviation was rated
with multiplier 3 (e.g. ±15 msec results as 45 error points). If the value according to
measurements is ±10 msec or higher, the user can see this clearly as jerking of the video.
The other measured value was the speed over the whole video clip, that is AVG fps. The
desired value varies depending on the reported speed of video (24, 30 and 60 fps). Deviation
percentage from this target to any direction was taken into account as it was (for example
deviation of 10 % results in 10 error points).
The third measured quality was the amount of dropped frames. These results were taken
into account as they were, so e.g. 5 % of dropped frames caused 5 error points. Again if the
dropped frame value is bigger, the user can see this as jerking of the video.
An overall score was determined according to the amount of error points from the
mentioned measurements. The best score in this test is 0. Other categories according to
error points include
Score less than 10 error points very good
Score 10-20 good
Score 20-50 satisfactory
Score over 50 unsatisfactory
Results
Results were interesting. According to all measures the performance of the new operating
system was weaker than that of the old operating system. The best result in video playback
got Nexus 7 of year 2012 with the older operating system. The second best was Nexus 7 of
year 2013 with the old operating system. In practice the update of operating system has a
remarkable negative impact on results of both versions of Nexus. Dropped frames and
deviation appeared in the normal use of basic videos (video speed 24/30 fps).
Image 2: Overall results with frame rate 24/30 fps. The year 2012 Nexus 7 with the old
operating system got the best score.
An interesting result from the point of view of a demanding user requiring high frame rates
is that after KitKat 4.4 update the playback of a 60 fps video did not work anymore with
neither of the test videos (720 p or 1080 p) in Nexus 7 of year 2013. The older operating
system played the speed of 60 fps and resolution of 720 p satisfactory. The playback of this
speed in total is poor with both devices and versions.
Image 3: Overall results with all speeds. Nexus 7 Android 4.4 system wasn’t able to play at all
video resolution of 720 p or 1080 p with speed of 60 fps.
0,0
20,0
40,0
60,0
80,0
100,0
Nexus 7 (2012), 4.3 Nexus 7 (2012), 4.4 Nexus 7 (2013), 4.3 Nexus 7 (2013), 4.4
Overall Score, 24/30 fps. Smaller score is better
0,0
100,0
200,0
300,0
Nexus 7 (2012), 4.3 Nexus 7 (2012), 4.4 Nexus 7 (2013), 4.3 Nexus 7 (2013), 4.4
Overall Score, 24/30/60 fps. Smaller score is better
Image 4: The results of test winner Nexus 7 (2012) with old operating system
Image 5: Test results Nexus 7 (2013) with Android 4.3 operating system.
0,0
10,0
20,0
30,0
40,0
50,0
60,0
local
30fps
h264
1080p
local
60fps
h264
1080p
local
30fps
h264
720p
local
60fps
h264
720p
Youtube
24fps HD
Nexus 7 (2012), Android 4.3
Deviation (+/-msec)
AVG fps
Dropped frames %
0,0
10,0
20,0
30,0
40,0
50,0
60,0
local
30fps
h264
1080p
local
60fps
h264
1080p
local
30fps
h264
720p
local
60fps
h264
720p
Youtube
24fps HD
Nexus 7 (2013), Android 4.3
Deviation (+/-msec)
AVG fps
Dropped frames %
Image 6: Test results Nexus 7 (2012) with Android 4.4 operating system.
Image 7: The weakest performance in this test: Nexus 7 (2013) with Android 4.4 operating
system. A device with the new operating system wasn’t able to play the video resolution of
720 p or 1080 p with speed of 60 fps at all.
Conclusion
When evaluating video playback, the older KitKat operating system performs better than the
new one. The update has a clear effect on the quality perceived by user especially when
watching fast videos with more and more common 60 fps speeds. This is something that
user should know before loading a new version of operating system.
0,0
10,0
20,0
30,0
40,0
50,0
60,0
local
30fps
h264
1080p
local
60fps
h264
1080p
local
30fps
h264
720p
local
60fps
h264
720p
Youtube
24fps HD
Nexus 7 (2012), Android 4.4
Deviation (+/-msec)
AVG fps
Dropped frames %
0,0
10,0
20,0
30,0
40,0
50,0
60,0
local
30fps
h264
1080p
local
60fps
h264
1080p
local
30fps
h264
720p
local
60fps
h264
720p
Youtube
24fps HD
Nexus 7 (2013), Android 4.4
Deviation (+/-msec)
AVG fps
Dropped frames %
These results raise the question, how is it possible that updating to a new software version
has such a clear negative effect on the relatively measured results.
Company behind the results
OptoFidelity is a Finnish, nonaligned high technology company specializing in test and
measurement automation. Our focus is on non-intrusive testing and measuring devices from
the end-user point of view.
Our mission is to fight against poor manual testing and show that investment in right testing
automation can pay back in more qualified applications and devices as well as more satisfied
end-user user experience.
We have several test systems and testing tools which can be used separately or for building
a complete, fully automated test system. Most of our products are scalable and can also be
integrated to customers own testing automation system or our partners script test or
software testing tools.
Learn more about our test automation solutions on
http://www.optofidelity.com/products-and-services/test-automation
You can also watch our other test automation demo videos at
http://www.youtube.com/channel/UCxZMoLkyOc7rMwKuTrNp6HQ
Tidak ada komentar:
Posting Komentar