#15820 closed Bugs (Fixed)

Ass Subtitle memory leak

Reported by: GloW Owned by: elupus
Priority: 4 - Normal
Component: Subtitles Version: 17.0 "Krypton" Beta6
Severity: Normal Keywords: ass ssa memory leak
Cc: popcornmix, Milhouse, sraue Blocked By:
Blocking: Platform: All

Description

When playing a video with an ass subtitle, there is a memory leak, wich ultimately crashes kodi.

Playing any video with this subtitle : http://www.filedropper.com/homelands01e01hdtvxvid-asapen

will reproduce the bug :

Starting to play :

[[email protected] ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            369          63         169           0         136         291
Swap:             0           0           0

10 Minutes :

[[email protected] ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            369          93          16           0         258         261
Swap:             0           0           0

20 Minutes :

[[email protected] ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:            369         108           4           0         255         246
Swap:             0           0           0

Attachments (8)

Homeland.S01E01.HDTV.XviD-ASAP.en.ass (68.9 KB) - added by GloW at 2015-02-25T11:03:34Z.
ass subtitle
298-status.log (3.3 KB) - added by danieles at 2015-04-01T11:22:35+01:00.
mem_stats_process.sh (2.7 KB) - added by danieles at 2015-04-01T11:22:55+01:00.
Insane Test.ass (536.7 KB) - added by Milhouse at 2015-11-22T19:11:42Z.
Insane ASS test file - displays multiple subtitles every second
revo_openelec.txt (63.7 KB) - added by Milhouse at 2015-11-24T06:58:30Z.
Revo/OpenELEC looped playback raw data
revo_openelec_graph.jpg (597.4 KB) - added by Milhouse at 2015-11-24T06:59:37Z.
Revo/OpenELEC looped playback graph
ubuntu.txt (372.4 KB) - added by Milhouse at 2015-11-24T17:02:59Z.
Ubuntu looped playback raw data
ubuntu_graph.jpg (622.7 KB) - added by Milhouse at 2015-11-24T17:04:04Z.
Ubuntu looped playback graph

Change History (41)

Changed at 2015-02-25T11:03:34Z by GloW

ass subtitle

comment:1 Changed at 2015-02-25T11:18:30Z by GloW

may or may not be linked http://trac.kodi.tv/ticket/10896

comment:2 Changed at 2015-02-25T11:46:50Z by popcornmix

Forum thread: http://forum.kodi.tv/showthread.php?tid=219486

Confirmed to occur on OpenELEC for Pi and x86.

comment:3 Changed at 2015-02-25T11:58:58Z by GloW

Confirmed on ArchLinux ARM on Pi.

comment:4 Changed at 2015-02-25T13:32:14Z by GloW

Confirmed on ArchLinux x64 with freshly compiled kodi from git.

comment:5 follow-up: Changed at 2015-02-25T19:16:53Z by FernetMenta

  • Cc sraue added

note that free does not indicate available mem for applications. after 20 min available has not changed much. to me this looks like an OS issue.

comment:6 in reply to: ↑ 5 Changed at 2015-03-09T18:07:52Z by GloW

Replying to FernetMenta:

note that free does not indicate available mem for applications. after 20 min available has not changed much. to me this looks like an OS issue.

Actually, watching the exact same video, with the same subtitle in srt format does not show any change in used memory.

Also you can actually see in real time a jump ( 2/3 Mo ) in used memory each time a subtitle is displayed in screen.

comment:7 Changed at 2015-04-01T11:21:53+01:00 by danieles

No doubt, there is a leak of memory with ass subtitles. With a simple script I've analyzed memory consumption for 15 minutes first with the same subittle first srt and then ass.

The memory leak does useless the ass subtitles in kodi, especially in a computer like mine (raspberry pi with openelec) after about 30 minutes of playing the computer hangs and reboots

I attach a file with statistics of memory used by kodi process (298-status.log, open it like csv file tab-delimited) and the script used for get the statistics (mem_stats_process.sh):

MemFisica=physical memory consumption

MemVirtual=virtual memory consumption

DifTotMemFis=difference of physical memory consumption in relation with the first value obtained

DifTotMemVirt=difference of virtual memory consumption in relation with the first value obtained


15 minutes with srt subtitles: memory consumption does not increase (even decrease)

15 minutes with ass subtitles: memory consumption (physical and virtual) grows about 60MB

Besides, the memory leak with ass subtitles does not seem to recover when the playback stop

Changed at 2015-04-01T11:22:35+01:00 by danieles

Changed at 2015-04-01T11:22:55+01:00 by danieles

comment:8 Changed at 2015-05-25T10:50:00+01:00 by Martijn

can you retry with latest available nightly version (not the beta)? http://kodi.tv/download/#devbuilds

comment:9 Changed at 2015-05-30T13:07:54+01:00 by GloW

Can you be more precise about the version that i should test ? Is current git alrigth? after which commit ?

comment:10 Changed at 2015-05-30T13:19:46+01:00 by Martijn

newest as mentioned

comment:11 Changed at 2015-06-27T16:29:22+01:00 by danieles

I have tried it in xbian with Kodi 15.0-RC1 compiled June 27 2015, I also have needed to upgrade libass4 because the subtitle isn't displayed "Unable to resolve exports from dll libass.so.4" In the nightly version the bug seems solved, my raspberry has played a video with a crazy subtitle (several sentences per minute) for 45 minutes without problems, the memory has increased a little at the begining but later it has remained constant. I had tried before kodi 14 in the same conditions (same video, subtitle, machine, OS...) and the bug was reproduced.

I think that you can change the bug status to solved.

Last edited at 2015-06-27T19:28:00+01:00 by danieles (previous) (diff)

comment:12 Changed at 2015-11-22T18:16:50Z by danieles

After a some new tests with newer versions:

  • Xbian with Kodi 15.2 (git e8f1d16) compiled on 4 Nov. 2015: the problem isn't reproduce
  • Openelec with Kodi 15.2 (git 02e7013) compiled on 1 Nov. 2015: It reproduces the problem still

Maybe the memory leak is a combination of Kodi and some library, but it doesn't seem anymore a bug of Kodi from version 15.2

Changed at 2015-11-22T19:11:42Z by Milhouse

Insane ASS test file - displays multiple subtitles every second

comment:13 follow-up: Changed at 2015-11-22T19:19:36Z by Milhouse

This is still reproducible with latest Kodi 16/master when built on Ubuntu 15.10 - are you sure your xbian testing is accurate? Or have xbian included a fix that isn't upstream?

I'm testing with an insane ASS file (attached) and on a Raspberry Pi2 (latest OpenELEC Milhouse test build, ie. Kodi 16 with VideoPlayer), or with x86/Ubuntu build (ie. plain Kodi 16 with DVDPlayer), Kodi will leak 80MB in 5-6 minutes.

comment:14 in reply to: ↑ 13 Changed at 2015-11-23T06:49:48Z by danieles

Replying to Milhouse:

This is still reproducible with latest Kodi 16/master when built on Ubuntu 15.10 - are you sure your xbian testing is accurate? Or have xbian included a fix that isn't upstream?

I'm testing with an insane ASS file (attached) and on a Raspberry Pi2 (latest OpenELEC Milhouse test build, ie. Kodi 16 with VideoPlayer), or with x86/Ubuntu build (ie. plain Kodi 16 with DVDPlayer), Kodi will leak 80MB in 5-6 minutes.

All right if you can reproduce it with Kodi 16 on Ubuntu and Openelec then obviously it still be a existing bug on Kodi affecting many platforms and not an Openlec bug as I thought.

I tried it today again on Xbian latest with your 'Insane Test.ass' for 15 minutes without symptoms of leak memory, in the same machine with Openelec it hangs/crashes in a few minutes. I don't know if xbian has a fix that isn't upstream, although I could no longer reproduce it on Xbian and Kodi 15.0-RC1 five months ago (comment:11) with a massive .ass subtitle (similar to your 'Insane Test.ass'), I used the standard latest version (without modifications), I think it could be due to a some updated library, some library related to play with subtitles. Correct me if I'm wrong I believe that Xbian is based in debian testing (rolling release).

comment:15 Changed at 2015-11-23T17:46:48Z by danieles

I've seen in packages:

In libass changelog 0.12.3 - "* Fix some potential memory leaks and crashes" : https://github.com/libass/libass/blob/master/Changelog

Can you try upgrade libass version in your Ubuntu 15.10 and test again with your subtitle?, maybe there's luck seems to be a lot of changes in the latest version.

comment:16 follow-up: Changed at 2015-11-23T18:05:09Z by Milhouse

Ubuntu 15.10 with libass 0.12.3-2 would seem to include the memory leak fixes? And as far as I can tell that still leaks. Also, OpenELEC master is using libass 0.13.0 and still leaks, so whatever leaks have been fixed they're probably not the ones we're seeing here.

comment:17 in reply to: ↑ 16 Changed at 2015-11-23T18:08:41Z by danieles

Replying to Milhouse:

Ubuntu 15.10 with libass 0.12.3-2 would seem to include the memory leak fixes? And as far as I can tell that still leaks. Also, OpenELEC master is using libass 0.13.0 and still leaks, so whatever leaks have been fixed they're probably not the ones we're seeing here.

OK, then we must continue working

comment:18 Changed at 2015-11-23T18:38:45Z by FernetMenta

how do you measure and what makes you sure this is a mem leak?

comment:19 Changed at 2015-11-23T18:52:11Z by Milhouse

while [ : ]; do echo "$(date +%H:%M:%S): $(grep MemFree /proc/meminfo)" ; sleep 10; done

MemFree is reducing at a consistent rate when using my "insane test" file (other ASS files are naturally a little less predictable, but the trend is the same), and the lost memory is never freed until kodi.bin is terminated.

I realise we can debate the method of measuring this, as I'm sure there are pros and cons, but what can't be denied is that if you wait long enough while playing an ASS subtitle the OOM Killer will eventually make an appearance and it's goodbye kodi.bin.

comment:20 Changed at 2015-11-23T19:13:38Z by FernetMenta

Linux may consume memory for its own purposes but give it to applications on request. Hence "free" is not the actual free available memory. I am wondering if this is a OE system issue and not related to applications and its libs. Has anybody observed OOM killer on a non OE system?

comment:21 Changed at 2015-11-23T19:25:12Z by Milhouse

There's a forum thread discussing various non-OE systems that are (or were) crashing with ASS subs due to OOM: http://forum.kodi.tv/showthread.php?tid=219486

I've only got RPi/RPi2 and a 2GB x86 OE system here, which I can trigger OOM on in a reasonable amount of time, but waiting for an 8GB Ubuntu system to OOM could take a while but will give it a go - a movie on continuous repeat overnight might do the trick...

comment:22 Changed at 2015-11-23T22:56:19Z by popcornmix

Another data point is my own build of Kodi running on raspbian. This is a 512M Pi without swap. I log "free -h" every minute. I'm aware that you need to look at the "+/- buffers/cache" row to see memory usage excluding linux file cache.

It ran for just over 5 minutes before kodi seg faulted (I have also seen OOM in the past). We leak about 20MB per minute with Milhouse's subtitle file.

             total       used       free     shared    buffers     cached
Mem:          189M       175M        13M       4.5M        16K       112M
-/+ buffers/cache:        62M       126M
Swap:           0B         0B         0B
             total       used       free     shared    buffers     cached
Mem:          189M       178M        10M       4.5M        16K        91M
-/+ buffers/cache:        86M       102M
Swap:           0B         0B         0B
             total       used       free     shared    buffers     cached
Mem:          189M       175M        13M       4.5M        16K        67M
-/+ buffers/cache:       108M        80M
Swap:           0B         0B         0B
             total       used       free     shared    buffers     cached
Mem:          189M       177M        11M       4.5M        16K        51M
-/+ buffers/cache:       126M        62M
Swap:           0B         0B         0B
             total       used       free     shared    buffers     cached
Mem:          189M       176M        12M       4.5M        16K        31M
-/+ buffers/cache:       144M        44M
Swap:           0B         0B         0B
             total       used       free     shared    buffers     cached
Mem:          189M       176M        12M       4.5M        16K        11M
-/+ buffers/cache:       164M        24M
Swap:           0B         0B         0B

[1]+  Segmentation fault      PYTHONOPTIMIZE=1 LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libasound.so.2 LD_LIBRARY_PATH=/opt/xbmc-dbg2/arm-linux-gnueabihf/lib /opt/xbmc-dbg2/arm-linux-gnueabihf/lib/kodi/kodi.bin --standalone --fs

comment:23 Changed at 2015-11-24T01:50:08Z by Milhouse

I've had the Ubuntu machine (8GB RAM + 8GB swap) running for a few hours, playing a video on a loop with the "insane test" ASS.

MemFree started out at 884212 kB and had dropped to 141588 kB (ie. 740MB drop) before I kicked off tonight's RPi builds, which skewed this figure.

However according to top, the non-swapped physical memory being used by kodi.bin has reached 1.805GB, and continues to increase.

top - 01:28:27 up 7 days, 20:11,  7 users,  load average: 1.52, 7.14, 7.41
Tasks: 273 total,   1 running, 272 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.8 us,  0.7 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8072460 total,  6482724 used,  1589736 free,   789184 buffers
KiB Swap:  8283132 total,   346776 used,  7936356 free.  3112604 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
17749 neil      20   0 5050536 1.805g  54172 S   8.0 23.5  29:18.31 kodi.bin
 1235 root      20   0  432232  24056   8524 S   6.0  0.3  36:45.20 Xorg
 1742 neil      20   0 1306972  37188  12400 S   2.7  0.5  49:29.08 compiz

As an update to the poor-mans monitoring, I've added VmRSS which is the RES figure in top:

PIDKODI=$(pidof kodi.bin); while [ : ]; do echo "$(date +%H:%M:%S): $(grep MemFree /proc/meminfo)  /  $(grep VmRSS /proc/${PIDKODI}/status)" ; sleep 10; done

Let's see where we are in the morning... ;)

comment:24 Changed at 2015-11-24T02:24:15Z by Milhouse

48 minutes after I scraped the top output in the previous comment, and RES has now reached 2GB:

top - 02:20:11 up 7 days, 21:03,  7 users,  load average: 0.41, 0.33, 0.55
Tasks: 276 total,   1 running, 275 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.0 us,  0.7 sy,  0.0 ni, 97.3 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8072460 total,  6766988 used,  1305472 free,   823844 buffers
KiB Swap:  8283132 total,   342864 used,  7940268 free.  3144236 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
17749 neil      20   0 5230848 2.000g  54448 S   7.3 26.0  33:29.52 kodi.bin
 1235 root      20   0  432060  24416   8872 S   6.6  0.3  40:08.04 Xorg

It appears to be increasing at the rate of about 4MB/minute.

comment:25 Changed at 2015-11-24T06:11:56Z by GloW

FYI , ArchLinux ARM, wtih Kodi 16 and libass 0.13 does not have this bug.

Changed at 2015-11-24T06:58:30Z by Milhouse

Revo/OpenELEC looped playback raw data

Changed at 2015-11-24T06:59:37Z by Milhouse

Revo/OpenELEC looped playback graph

comment:26 Changed at 2015-11-24T07:10:40Z by Milhouse

Revo Testing (2GB RAM, no swap, OpenELEC Kodi master with VideoPlayer, build #1121).

Test video is an AVI, 49m50s, over smb:// with external "insane" ASS subtitle (5 subtitles per second, each subtitle has 3 lines of text, 49m50s of subtitle data).

The video was queued to a "repeat all" playlist, with playback commencing at 04:03:40, ending at 04:53:36, then immediately repeating:

04:03:40 153.148636 T:140438990833792  NOTICE: VideoPlayer: Opening: smb://NM-WIN7/Torrent/qTorrent/Psych.S05E12.Dual.Spires.HDTV.XviD-FQM.avi
...
04:53:36 3149.562256 T:140438990833792  NOTICE: CVideoPlayer::CloseFile()
...
04:53:36 3149.565430 T:140438990833792  NOTICE: VideoPlayer: Opening: smb://NM-WIN7/Torrent/qTorrent/Ps
...

etc.

Refer to attached graph of results (raw data also attached).

From the outset, RSS memory consumption increases steadily and consistently, gaining 484MB until the increase plateaus at 04:20:30 (16m50s after start of playback), with no further significant increase through to the end of playback. At end of playback, VmRSS is 763MB.

Playback commences for the second time at 04:53:41. Shortly afterwards, at 04:54:52, RSS memory begins to increase once again, gaining another 491MB until the next plateau at 05:10:43 (15m51s of playback). At the end of playback, VmRSS is 1270MB

Playback begins for the third time at 05:43:37. Again, RSS begins to increase steadily, gaining another 420MB, until plateauing at 06:00:07 (16m30s of playback). At the end of playback, VmRSS is 1702MB.

According to htop, memory in use use by all tasks is 1791MB, out of a total 1993MB available RAM.

Playback begins for the fourth and last time at 06:33:37. However almost immediately there is a problem with playback, which is now frozen:

06:33:37 9149.867188 T:140437092292352  NOTICE: CDVDVideoCodecFFmpeg::Open() Using codec: MPEG-4 part 2
06:33:38 9151.172852 T:140438990833792  NOTICE: GL: Selecting Single Pass YUV 2 RGB shader
06:33:38 9151.465820 T:140438990833792  NOTICE: GL: NPOT texture support detected
06:33:38 9151.465820 T:140438990833792  NOTICE: GL: Using GL_ARB_pixel_buffer_object
06:33:38 9151.465820 T:140438990833792  NOTICE: Using GL_TEXTURE_2D
06:33:39 9152.405273 T:140437092292352 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
06:33:39 9152.499023 T:140438990833792 WARNING: Previous line repeats 1 times.
06:33:39 9152.499023 T:140438990833792  NOTICE: GL: Selecting Single Pass YUV 2 RGB shader
06:33:39 9152.519531 T:140437092292352 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
06:33:39 9152.768555 T:140438990833792  NOTICE: GL: NPOT texture support detected
06:33:39 9152.768555 T:140438990833792  NOTICE: GL: Using GL_ARB_pixel_buffer_object
06:33:41 9153.954102 T:140437092292352 WARNING: CRenderManager::WaitForBuffer - timeout waiting for buffer
06:35:45 9278.231445 T:140438971639552 WARNING: Previous line repeats 121 times.
06:35:45 9278.694336 T:140438971639552   ERROR: CAESinkALSA - snd_pcm_writei(-32) Broken pipe - trying to recover
06:38:31 9443.916992 T:140438551901952   ERROR: Previous line repeats 5 times.
06:38:31 9443.930664 T:140438551901952  NOTICE: None

At this point Kodi is unresponsive to input, and needs to be killed. The point at which kodi.bin is killed should be obvious in the graph. :)

I'm still waiting for the Ubuntu test to complete, it's obviously taking a lot longer to reach any definitive conclusion, and VmRSS seems to be increasing at a slower rate (currently VmRSS is 3.662GB), but shows a similar increase-plateau-increase pattern as per the Revo.

Last edited at 2015-11-24T07:12:07Z by Milhouse (previous) (diff)

Changed at 2015-11-24T17:02:59Z by Milhouse

Ubuntu looped playback raw data

Changed at 2015-11-24T17:04:04Z by Milhouse

Ubuntu looped playback graph

comment:27 Changed at 2015-11-24T17:17:37Z by Milhouse

Ubuntu Testing (8GB RAM, 8GB swap, Kodi master with DVDPlayer, libass 0.12.3-2).

Same test video as with Revo, above: AVI, 49m50s, with "insane" ASS subtitle (5 subtitles per second, each subtitle has 3 lines of text).

The test started at 01:26:05 (data only from 01:32:38), with a single video queued to a "repeat all" playlist.

Refer to attached graph of results (raw data also attached - note that at 02:26:18, a short build of tvheadend add-ons for RPi started, which grabbed a chunk of MemFree memory).

Unlike the OpenELEC/Revo test machine, which would increase VmRSS for up to 17 minutes then stop increasing until the end of playback, Ubuntu typically shows no increase in VmRSS memory for up to 15-16 minutes, then increases VmRSS memory for 2-3 minutes, with no further increases until the end of playback.

After the first 3 plays, VmRSS has increased by 168MB, 178MB and 176MB, so much slower to leak than Revo, perhaps because it's only "leaking" for that 2-3 minute period.

At about 04:12, 2h46m into the test, the memory usage behaviour changes - a saw tooth pattern - with memory increasing then being returned. No idea what happened here, machine idle. Maybe it's nothing.

Then at 05:26, memory usage increases at a much faster rate. Again, no idea why.

At 07:58, VmRSS reduces significantly (and MemFree increases). Not entirely sure why, although it did coincide with cron.daily jobs being run. Prior to this, kodi.bin had 4.2GB of RAM allocated to it.

I ended the test at 17:00 with 7GB RAM allocated to kodi.bin - I need to use this machine for something else... there has been no OOM as I never actually ran out of memory, maybe I would have if I let the test run for another day or so, but clearly most of the physical memory is now being hogged by kodi.bin.

If kodi.bin using 7GB of RAM isn't a sign of a memory leak, I don't know what else to say on this matter.

comment:28 Changed at 2015-11-25T15:33:47Z by danieles

I think there's no doubt, there is a leak of memory with ass subtitles in kodi but... why? Another test, even stranger, two virtual machines on a W7 host:

  • Ubuntu 15.10 64bits desktop, fresh install, all packages updated: Play video with ass subtitle on Kodi 16 from ppa:team-xbmc/unstable and it loses memory.
  • Xubuntu 15.10 64bits desktop, fresh install, all packages updated: Play same video with same ass subtitle on Kodi 16 from ppa:team-xbmc/unstable and it DOES NOT lose memory.

Ubuntu 15.10 with libass5 updated to 0.13 is losing memory too.

comment:29 Changed at 2016-11-05T21:26:35Z by Martijn

We have update libass in v17. Can you recheck?

comment:30 Changed at 2016-11-05T22:38:33Z by popcornmix

Issue is still present (including in top of tree of libass)

I've done some investigating and can see the buffer that is not being release. I've created an issue here: https://github.com/libass/libass/issues/248

Not certain if it is a libass bug or a kodi usage problem.

comment:31 Changed at 2016-11-06T07:38:25Z by Martijn

  • Version changed from 15.0 "Isengard" Alpha1 to 17.0 "Krypton" Beta6

comment:32 Changed at 2016-11-07T11:15:04Z by popcornmix

I believe this is now fixed with https://github.com/xbmc/xbmc/pull/10873 Fix should be in last night's Milhouse pi build, or in future nightly builds.

comment:33 Changed at 2016-11-07T16:37:21Z by Martijn

  • Milestone changed from Pending / Future to 17.0 "Krypton"
  • Resolution set to Fixed
  • Status changed from new to closed

I'll close it as fixed.

Note: See TracTickets for help on using tickets.