UI freeze due to large xml config file writes

[Tin Ching]

BitComet UI not responding

A lot of users has already brought this up, but I think so far no one has located the root cause of the issue. I personally experience this issue and have dig deep into the root cause of the issue. Some user may not have experienced this and it really depends on how "popular" your torrent file is.

The torrent file I download has 15,000+ users in the network. Every 5 minutes exactly, BitComet writes xml configuration files for the tasks into the AppData folder. Two of my files in my download list has 15,000+ users and this is where BitComet UI freeze up during the time it starts writing these gigantic xml files (each 150MB+ in size). This causes the UI freeze for 30+ seconds.

Can these config file writes to be put into a background thread so that it doesn't freeze the UI?  Same applies to program startup, BitComet reads all these xml files which makes the program unresponsive.

I can also confirm that the freeze is not coming from a disk bottleneck, as my hard disk is an SSD drive and disk activity is very low, I suspect the UI freeze happens when the program writes the XML data into memory OR it's busy gathering all the info for the task (e.g. peer IPs etc)

I hope this helps the community, and the info I have provided should allow the issue to be easily reproduced.

Edited May 8, 2020 by Tin Ching (see edit history)

[Tin Ching]

More details:

- A lot of users in the past has mentioned the UI freeze every 5 minutes, this coincides the every 5 minute writes for the task xml config files

- Users also mentioned they get the freeze if you have enabled DHT, this is because you will find a lot more peers if you enabled DHT, and this will bloat up the xml config files

In my opinion, this is the most critical performance issue for BitComet.

Why?

BitComet writes these files every 5 minutes, but it doesn't write them all at the same time, it writes one by one. As time goes on, the time gap of each write of file increases and spread to different times. Imaging you have 5 torrent files like this and evenly spread across, on average you will get a GUI freeze every minute. If you have 10 files like this, you will get a freeze every 30 seconds on average. This will make the program hardly usable.

During the time it freezes, all uploads/downloads will be jammed/halted (goes to 0)

Possible solutions:

I do not think writing these xml files is time critical.

- Write these xml files in it's own thread separate from the GUI, or

- Maybe have an advanced option where we can adjust the auto save frequency? or even have the option to only save these files upon closing the program.

Edited May 9, 2020 by Tin Ching
More useful info (see edit history)

[TripleThreat]

When my UI is with 1000+ active torrents, interface comes more unresponsive as I start my torrents.  

At 2000+, the UI needs a lot of loading time to scroll up and down the window of torrents, yet it only uses 6% of my CPU power and I still have plenty of ram available with the version 1.66.

More critical issue of Bitcomet that remains for many BitComet versions, which I am planning to start a post in the future, is that at 2100~2500 active tasks, just starting more torrents will come to a point that all the  uploading stream and the downloading stream will halt to 20KB/s and stay here for hours until I force to brutal-force quite BitComet. At that point I have to restart the program and slow restart all torrents at patches back up again with tons of hash rechecking to do before running tasks.

[Tin Ching]

Yeah, even we have a very powerful machine, but the current architecture of BitComet is not scalable. The xml config file writes is killing it.

Edited May 9, 2020 by Tin Ching
More accuracy in wording (see edit history)

[Tin Ching]

Attached below is screenshot of 2 large xml files causing UI to freeze at regular intervals.

[TripleThreat]

I wonder if the entire BitComet program is put on RamDisk and run it there, the XML write performance might improve a bit.

[Tin Ching]

Nah, I think this is a software issue, not hardware, particularly relates to the algorithm/architecture of how these XML files get read / written. The UI freeze is when the program started triggering the xml writes (but hasnt actually started writing to disk yet).  During this time its busy doing something else which I'm not sure what it is, could be busy gathering all the peer IPs and writing to memory.

I would say if this issue gets fixed, most of the UI hang bug tickets in the past would be solved.

[TripleThreat]

v1.67 is out. There are couple of fixes to long-term seeding lagging the UI. I am going to give that a try.

[Tin Ching]

Just tried v1.67, still have freeze when it writes the current tasks into the xml files. I saw the changelog some other torrent history xml file got migrated to sqlite, I wonder if they can do the same for the current active tasks as it may help to remove the freeze.

[TripleThreat]

Yeah same for me . UI freezes, and the download stream and upload stream would freeze beyond 2000 torrent tasks... Disappointing, but at least the BitComet team is trying. I hope they recognize the download/upload streams issue as well. I don't know if the dev team knows that already.