deleting a playlist takes a very long time with version 0.19

Need help with MPD?
tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

deleting a playlist takes a very long time with version 0.19

Post by tinub »

so far, i used mpd version 0.17 under gentoo.

to add all tracks in the music folder (about 100k tracks), i do a
mpc ls | mpc add

to get rid of these tracks, i do a
mpc clear

with the setup mentioned above, the mpc clear command takes about 2s to complete.

now i switched to raspbian jessie, with the mpd version provided there (version 0.19).
unfortunately, something seems to go wrong there.

the
mpc ls | mpc add
command takes about the same time as with the former setup (about 5s), while the
mpc clear
takes about 15 minutes.

during these 15 minutes, cpu usage of mpd is at 100% and mpd does not answer, i.e. mpc gets nothing but a timeout.

what am i doing wrong?

i tried this on a raspberry pi B and a 2B with different sd cards, but behaviour is always the same.

i have no idea on how to proceen ;-(

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

On my Pi, clearing a playlist with 40k items takes 2.5s (MPD 0.19.13). That is not unexpected. A lot of memory has to be freed.

15 minutes is too long, of course. But from here, I cannot possibly tell why. You could run "perf" to capture some data about MPD's CPU usage.

But what you should do first is upgrade to the latest MPD version. Your MPD version is 13 releases behind. Note that MPD packages shipped by Debian Stable are unsupported, because Debian refuses to adopt MPD bug fixes!

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

Thanx for the help!
How can i update to the latest version for the raspi when this can't be found using apt-get install?
I'm quite new to linux. Sorry for this question!

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

By compiling from source.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

hi max
ok, i have mpd 0.19.13 running now, but unfortunately, the problem persists ;-(
while mpc ls | mpc add takes about 5s, mpc clear takes about 15 minutes.
i really can't believe that deleting entries of a database takes such a lot more time than adding.
so there's obviously something going wrong, as this hasn't been a problem with my old Installation (0.17.6 under gentoo).

btw: i recognized that mpc reports wrong version number.
while mpd --version reports 0.19.13, mpc version reports 0.19.0, i.e. i already had a newer version than 0.19.0 before.
(for my old Installation which is 0.17.6, mpc version reports 0.17.0)

so what's the next thing i should try?

thanx for your help!!

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

I told you already what to do.

btw. mpc does not show the MPD version number - it shows the protocol version, which usually doesn't change with MPD minor releases.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

hm. sorry, i don't understand what you mean.
you told me to go to the latest version, and that's what i did now.
unfortunately, this did not help.

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

Maybe you should read more than just the last sentence in my post.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

if you mean the 'perf' thing, yes, i read, but haven't been really successful with it so far.

btw this does not seem to be officially supported by jessie, but in the mean time, i found a version that can be executed there...
but i have never used perf before and i wasn't able to do anything with it that seems to make any sense.

can you please let me know what exactly i have to do to get the results you need?
thanx!

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

I'm glad you now read my post. I hope you're aware that there's a big difference between "I did what you told me to do" and "I did not do what you told me, because I couldn't figure out how". This cycle was completely unnecessary, and wasted both of our times! That is annoying.

What you need to do is fill the MPD playlist as usual, and open one root terminal and type "perf record -p `pidof mpd`"; in another terminal, type "mpc clear".

As soon as the "mpc clear" finishes, press Ctrl-C in the "perf" terminal. Type "perf report >report.txt" and upload that text file.

Before you do this (READ THIS): make sure your MPD binary and your C library has debug symbols. If you upload a report without debug symbols, the file is useless, and you need to repeat everything. I can't assist you with this part. Just figure it out.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

thanx. now i think this is an instruction i understand!
being new to linux, sometimes it's not so easy to understand what experts mean.
from my experience on old unix machines, perf does about the same top does, so i reported that mpd is at 100% for 15 minutes.

question about the debug symbols:

when getting the results below, i think this indicates that debug symbols are available and i can do the test you mentioned. right?
(that's what i found about checking if there are debug symbols)

objdump -h mpd | grep debug
26 .debug_aranges 00007398 00000000 00000000 000897da 2**0
27 .debug_info 00b118b0 00000000 00000000 00090b72 2**0
28 .debug_abbrev 000dfd1e 00000000 00000000 00ba2422 2**0
29 .debug_line 000ab76b 00000000 00000000 00c82140 2**0
30 .debug_frame 000153ec 00000000 00000000 00d2d8ac 2**2
31 .debug_str 00125938 00000000 00000000 00d42c98 2**0
32 .debug_loc 00104528 00000000 00000000 00e685d0 2**0
33 .debug_ranges 00041f40 00000000 00000000 00f6caf8 2**0

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

ok, i think i did what you wrote ;-)
and please let me know i you think i did not!

surprisingly, this time, mpc clear took about 1 hour to complete.
i have no idea why. does perf have any Impact to this?

i uploaded the Report anyway and try to repeat the whole Thing.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

hm. shouldn't the uploaded file be visible somewhere?
did you get it or shall i try again?

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

ok, this time it only took a few minutes...

btw, when writing the Report, it says:
Failed to open [vectors], continuing without symbols
but i think according to the message i posted this afternoon, symbols should be available?!?

i think the file upload worked this time...

please let me know if you can see something out of These files...
Attachments
longest_ever.rar
(6.2 KiB) Downloaded 87 times
much_quicker.rar
(1.87 KiB) Downloaded 115 times

max
Forum team
Posts: 984
Joined: January 15th, 2013, 3:43 pm

Re: deleting a playlist takes a very long time with version

Post by max »

No. Your MPD build is stripped, i.e. no debug symbols. Same for your C library. Useless.

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

hm.
now, according to Google, i replaced all gcc within the Makefile with gcc -g, but the result is still about the same.
so i tried to do a make -B, but then it does not compile. right at the beginning, i get a

ld/missing aclocal-1.15 -I m4
/home/pi/tmp/mpd/mpd-0.19.13/build/missing: Zeile 81: aclocal-1.15: Kommando nicht gefunden.
WARNING: 'aclocal-1.15' is missing on your system.
You should only need it if you modified 'acinclude.m4' or
'configure.ac' or m4 files included by 'configure.ac'.
The 'aclocal' program is part of the GNU Automake package:
<http://www.gnu.org/software/automake>
It also requires GNU Autoconf, GNU m4 and Perl in order to run:
<http://www.gnu.org/software/autoconf>
<http://www.gnu.org/software/m4/>
<http://www.perl.org/>
Makefile:4332: recipe for target 'aclocal.m4' failed
make: *** [aclocal.m4] Error 127

so how do you compile to get a debug Version (i think you are the developer of mpd, so you probably compile with debug Option, right?)

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

after a few more hours of googling and testing...

with some perf Options i got a better view, showing me that it's mpd itself using most of the cpu time, but no Details on mpd.
so i deleted the Main.o file, hoping that doing a make (with -g added to all gcc commands) should generate debug info.
but the resulting mpg file seems to still have about the same size.

i then found that perf record needs either a -g dwarf or a --call-graph dwarf to extract debug info from gcc -g, but both do not work. perf tells me that dwarf is unknown argument.

so now, i have no further ideas, i think...

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

although i couldn't get any usefull info from the profiler - many sleepless nights, but no results - i think i found something quite interresting:

the time it takes to clear the current playlist depends on the playlist size, but not in a linear manner. i measured the time it takes to do an mpc clear with different sizes of playlist:
size of current playlist / time for mpc clear:
8192 traks / < 1s
16k tracks / 10s
32k tracks / 83s
64k tracks / 342s
97k tracks / 750s

this is the time it takes for an mpd 0.19.x version running on jessie.

with the 0.17.6 version, running on gentoo, the 97k tracks take about 5s to do an mpc clear.

have you done any changes to the way the current playlist is saved since 0.17.6?
i think this is hold in memory, isn't it?

the fact, that the time, mpc clear takes, increases much more than linear (based on the 8k= <1s measurement, the 97k shouldn't take more than 10s) seems to be a clue that mpc clear starts some kind of recursive process, or clears the tracks not at once, but instead does something like "clear a track, rearrange list, clear a track, rearrange list,..." or something like that.

i really would have liked to do a testdrive of a 0.17.6. version under jessie to check if this really is an mpd issue or a jessie issue, but unfortunately, i was not able to compile 0.17.6 because of the ffmpeg problem.
is this ffmpeg required to play mp3 files? and if not, is there a way to exclude this from compilation?
or do you have a precompiled 0.17.6 (or older) version which is portable and can just be copied on a raspi for test?

regards
tinub

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

installed perf from another source, but to me, this doesn't look much more useful. just different ;-(
Attachments
test.rar
think this isn't much better... is it?
(1.94 KiB) Downloaded 101 times

tinub
Posts: 21
Joined: March 8th, 2016, 11:34 am

Re: deleting a playlist takes a very long time with version

Post by tinub »

i think i finally got the right data out of perf.
hope these are meaningfull to you.

please let me know if these files are ok now and if this problem can be fixed!
thanx!!
Attachments
tests_shortened.rar
script file, shortened
(176.64 KiB) Downloaded 110 times
testa.rar
annotation file
(26.44 KiB) Downloaded 91 times
test.rar
report file
(3.35 KiB) Downloaded 92 times

Post Reply