Queue Offset

[Midnight Modding]

Isn't it decently risky using queued tasks, since offsets could be incorrect over time?

For instance, let's say we want to send a notification to every member where column abc's value is 7. Well, in the time it takes to finish the queued task, members may have been deleted or their value in that column changed, but the offset is going to be based on the original set of data.

Or would queued tasks be run quickly? Seems to me I have seen some sit around for a while. I know you can force them to finish, but many site admins may not know that or do it. And if the data set does change between the first notification and the last set of them, then some people could be skipped receiving the notifications.

[newbie LAC]

Read 2nd part of the post

 

[Midnight Modding]

2 hours ago, newbie LAC said:

Read 2nd part of the post

 

Yes, good point there... although I was simplifying my situation. In my real situation, the problem is some of the members with lower member ids may have originally not been set to receive notifications and over the time of the task suddenly those lower member ids need the notification also. In other words, the whole set of who should get them could change and even knowing the most recent member id wouldn't solve that issue in my current situation.

Somehow I end up with so many feature requests which end up putting me in these odd situations. I think I know what I am going to do to avoid the issue, though, since this isn't a 100% custom app, so I can slightly change the feature and have it more dieal for this setup.

edit: well honestly... it shouldn't matter if the original set was different, now that I think about it. The whole point of a task is that it's sent based on THAT INSTANT'S statuses. So if they change over time for those earlier member ids, I shouldn't care. So doing the query like you showed is probably going to be fine.

[bfarber]

For the offset, you can take two approaches.

1) You can return an offset and use that in a LIMIT clause on the next loop (e.g. you process 250 records, return 250, and on the next cycle you do LIMIT 250, 250 to get the next batch). This is useful when you will absolutely be looping through every result or if the things you're restricting against in a WHERE clause aren't likely to change.

2) You can return the last ID that you processed, and then on the next loop you do WHERE id > {$offset}. This avoids problems with rows changing (or being deleted) from earlier cycles, and also performs better particularly on very large tables.

Which approach you take is up to you - each has its purposes. It sounds like if you are concerned the earlier results may change while the task is running, you could go with method #2 there.

[Midnight Modding]

2 hours ago, bfarber said:

For the offset, you can take two approaches.

1) You can return an offset and use that in a LIMIT clause on the next loop (e.g. you process 250 records, return 250, and on the next cycle you do LIMIT 250, 250 to get the next batch). This is useful when you will absolutely be looping through every result or if the things you're restricting against in a WHERE clause aren't likely to change.

2) You can return the last ID that you processed, and then on the next loop you do WHERE id > {$offset}. This avoids problems with rows changing (or being deleted) from earlier cycles, and also performs better particularly on very large tables.

Which approach you take is up to you - each has its purposes. It sounds like if you are concerned the earlier results may change while the task is running, you could go with method #2 there.

Thanks, yeah that second one is basically what was mentioned in the newbie post. I always did like #1 on my past ones (I did a lot of queued tasks on another app), but I can think of situations for the second way, as well. And if I somehow end up in a really odd situation, there's always $data['extra'] to store something in that may help me. In this case, I can easily do it either like #1 or #2 depending on my final decision on how I want the actual feature, though. 🙂