Skip to main content


I deleted about 90 users, which gave me 180000 tasks in the queue. Didn't think about that before, but that will take a few hours now.



!Friendica Support #Friendica

2 people reshared this

The number of items in the queue is increasing and did now reach 207.000 items.
from the log:
Load: 3.55/8 - processes: 1422/5/217062 - jpm: 78/43/56 (0:1, 20:2/36, 30:0/378, 40:1/157, 50:1/217913)


220.000
This entry was edited (2 months ago)
Just amazing to watch, not we're at 227.934 items in the queue.
This entry was edited (2 months ago)
now at 248.574
only down 55.000 items by now, so 266.000 to go.
Still fun to watch, now it is 223.000, so by this rate it will be finished tomorrow noon, taking into account that the server usually is less busy during nighttime, on the other hand there might kick in the backup which slows down the DB for as long as it dumps or so.
Only 15.000 to go, but now 45.000 in deferred queue.
If there are a lot of deffered posts, then you can decrease that with the configuration value "worker_defer_limit". I had set it to 6 on squeet.me.

Which kind of jobs are there the most?
@Michael Vogel I think it is fine, it is just prio 50.

Load: 3.3/8 - processes: 1520/6/240430 - jpm: 45/49/59 (0:0, 20:3/33, 30:1/144, 40:1/323, 50:1/241450)


Right?
This entry was edited (2 months ago)
@Michael Vogel let me ask about this, is this "worker_defer_limit" about the first number in /admin (above 1284 in the picture)?

What I would like is to make the server more usable for the users on it, so the "every day" things should work faster, like delivery of posts and fetching of posts.
Well, your system should have no problems with the "every day" work, since the masses of tasks are having priority 50.
Diese Erfahrung machte ich auch schon... Das löst eine ziemliche Flut von Tasks aus 😉
Please execute: select command,count(*) from workerqueue where not done group by command;
@Michael Vogel hmmm I think I see where this is going to.
+-------------------------+----------+
| command                 | count(*) |
+-------------------------+----------+
| APDelivery              |   294421 |
| CleanWorkerQueue        |        1 |
| ClearCache              |        1 |
| ContactDiscovery        |        9 |
| Delivery                |      749 |
| ForkHook                |        3 |
| Notifier                |       18 |
| PostUpdate              |        1 |
| PullDirectory           |        1 |
| RemoveUser              |       86 |
| SpoolPost               |        1 |
| UpdateContact           |      527 |
| UpdateContacts          |        1 |
| UpdateGServers          |        1 |
| UpdateServerDirectories |        1 |
+-------------------------+----------+
15 rows in set (0.756 sec)
Then execute this one:
select command,substr(parameter, 3, locate('"', parameter, 3)-3) as para, priority, count(*) from workerqueue where command = 'apdelivery' and not done group by para,priority;
@Michael Vogel
select command,substr(parameter, 3, locate('"', parameter, 3)-3) as para, priority, count(*) from workerqueue where command = 'apdelivery' and not done group by para,priority;
+------------+----------+----------+----------+
| command    | para     | priority | count(*) |
+------------+----------+----------+----------+
| APDelivery | removeme |       50 |   319020 |
| APDelivery | wall-new |       20 |       10 |
| APDelivery | wall-new |       40 |        3 |
| APDelivery | wall-new |       50 |       38 |
+------------+----------+----------+----------+
4 rows in set (1.486 sec)
Now the JPM is increasing and the DB is using less CPU, the systems has a lower load.
Maybe the creation of new items has finished now?

Load: 2.32/8 - processes: 1739/9/318725 - jpm: 283/195/70 (0:1, 20:0/9, 30:0/2, 40:6/809, 50:2/319644)
Okay. Some user removed their account. This will trigger this "removeme". But it shouldn't create that much jobs. I have to check. I will try to have a look at it in the evening.
@Michael Vogel I think now the queue item number is decreasing and the system load is very low again, so the JPM is high as there is no load that reduces the JPM.
@utzer @Michael Vogel Well, that's happening in the beginning... when the cleanup script runs for longer there won't be that many users anymore that'll get deleted. In my first run it deleted approx. 200 users... took my system a while as well to work through the queue... 😀
@Ingo Jürgensmann @Michael Vogel yes, no prob, it is fine now, it will finish the queue after some time, the 300.000 remaining items are not a problem.
@Michael Vogel the JPM could be higher I think, maybe it is some bottleneck, but if so it is not memory, CPU or io.
2021-01-11T11:59:04Z worker [NOTICE]: Load: 2.97/8 - processes: 5134/7/296439 - jpm: 259/326/384 (0:1, 30:0/12, 50:6/301561) - maximum: 7/25 [] - {"file":"Worker.php","line":772,"function":"tooMuchWorkers","uid":"f1f78c","process_id":3588982}
By the way, it was on 0 by about 10 o'clock the morning, so about 12 hours ago. Not the queue has 43000 defered items and 0 items that are currently worked on.
It was also interesting yesterday that the maximum number of workers was never used, while system load went to 3.x max and is permitted to go to 8. The JPM was up to 6xx, the highest I saw at least.

So I think it could have been faster, but something kept it back. There was also no bottleneck io wise, the friendica VM only had 5MB/s, so that seems not too much.

This website uses cookies to recognize revisiting and logged in users. You accept the usage of these cookies by continue browsing this website.