Scheduled script run interval

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Scheduled script run interval

Steve Cassidy-3
It's quite a while since I posted here. And a while since I've read any posts – there were almost 2000 unread emails in my FMExperts folder! (Which is very unusual; until last year I had been fully on top of Experts emails every since the list started up – and I don't really want to think about how long ago that was…)

Anyhow, a question for you about scheduled scripts on the server. Basically, I'd like to know what kind of load they are putting on the server and how much I can reduce the interval between runs without causing problems.

The background is this:  for one of my clients, I've moved a lot of inventory transaction processing to a scheduled script run by the server (a Mac Mini; I don't have the specs to hand). The server handles at most 20 Filemaker clients (and nothing else).

A server-side script runs once every 30 minutes, at which time it processes records held in a queue table. Generally, records are added to the queue in small batches – perhaps 20 stock movements within a few minutes, that kind of thing. Very often there are no records to process when the 30-minute script runs – in which case the script simply does a find that comes up with no found records and then exits. Twice a day (at night), major shipping routines run and these can generate 500 or more queue records. On a client, 500 queue records take anything up to 45 minutes to process. On the server about 30 seconds. (Each queued transaction involves multiple steps – such as determining the lot to be shipped, breaking open bulk cases, creating low inventory notifications, etc.)

The point is that some inventory transactions are more time-critical that others; if a user moves inventory from one warehouse to another, they like to be able to see that the transaction has happened – and also to use the moved inventory for some further purpose. A 30-minute interval is not ideal.

So, would there be any concerns with running this server-side script every five minutes, say? Or even every two minutes? As explained above, for the most part the processing is minimal (and is completed in less than two seconds). Once or twice a night (when no users are generally connected), there are these heavier batches of records that take the server 30 seconds to process. I'm thinking about slowdowns noticeable by users, or perhaps what might happen if the scheduled script runs very slowly for some reason and bumps into the next run time...

Thanks for any hints, particularly if anyone actually runs a scheduled script on a very short interval.

Steve
_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scheduled script run interval

Stephen Wonfor-3
Steve

I have a client running FMS on a MacMini that does a search, export then ftp every 10 minutes business hours Monday to Friday. More than 4 years so far without issue.

Stephen

------------------
Sent using a device built by a popular California technology cult.

> On Jul 29, 2017, at 09:46, Steve Cassidy <[hidden email]> wrote:
>
> It's quite a while since I posted here. And a while since I've read any posts – there were almost 2000 unread emails in my FMExperts folder! (Which is very unusual; until last year I had been fully on top of Experts emails every since the list started up – and I don't really want to think about how long ago that was…)
>
> Anyhow, a question for you about scheduled scripts on the server. Basically, I'd like to know what kind of load they are putting on the server and how much I can reduce the interval between runs without causing problems.
>
> The background is this:  for one of my clients, I've moved a lot of inventory transaction processing to a scheduled script run by the server (a Mac Mini; I don't have the specs to hand). The server handles at most 20 Filemaker clients (and nothing else).
>
> A server-side script runs once every 30 minutes, at which time it processes records held in a queue table. Generally, records are added to the queue in small batches – perhaps 20 stock movements within a few minutes, that kind of thing. Very often there are no records to process when the 30-minute script runs – in which case the script simply does a find that comes up with no found records and then exits. Twice a day (at night), major shipping routines run and these can generate 500 or more queue records. On a client, 500 queue records take anything up to 45 minutes to process. On the server about 30 seconds. (Each queued transaction involves multiple steps – such as determining the lot to be shipped, breaking open bulk cases, creating low inventory notifications, etc.)
>
> The point is that some inventory transactions are more time-critical that others; if a user moves inventory from one warehouse to another, they like to be able to see that the transaction has happened – and also to use the moved inventory for some further purpose. A 30-minute interval is not ideal.
>
> So, would there be any concerns with running this server-side script every five minutes, say? Or even every two minutes? As explained above, for the most part the processing is minimal (and is completed in less than two seconds). Once or twice a night (when no users are generally connected), there are these heavier batches of records that take the server 30 seconds to process. I'm thinking about slowdowns noticeable by users, or perhaps what might happen if the scheduled script runs very slowly for some reason and bumps into the next run time...
>
> Thanks for any hints, particularly if anyone actually runs a scheduled script on a very short interval.
>
> Steve
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scheduled script run interval

JClose
Steve,
    Another approach could be to add a 'priority' value to each queue record.  You mentioned that some things were more time sensitive - those records would get a high priority.  Then when your scheduled script runs it would search for records with a certain priority based on how long it had been since the last time it ran.  The scheduled script could run every two minutes - it could search for highest priority items each time; but would only look for mid priority records if, say the minutes of the system clock = 32 or some such (the schedule would need to be set so it would run at the 32 minute); really low priority things would only get searched for if the hour = 1am or some such.  (You have to be a bit careful because you don't have 100% control over when the script actually runs - other processes can delay the starting time a bit.)

You can, of course, search for multiple groups of priorities at each run based on the time ( or whatever calculation you come up with).  For example, if its the 30 minute run you can search for the 2 minute records, the 5 minute records, and the 30 minute records, to process them all in one go.

The short description is that you search for specific records based on the time of day or minute, and based on a priority you define.

--  Justin

> On Jul 29, 2017, at 5:31 AM, Stephen Wonfor <[hidden email]> wrote:
>
> Steve
>
> I have a client running FMS on a MacMini that does a search, export then ftp every 10 minutes business hours Monday to Friday. More than 4 years so far without issue.
>
> Stephen
>
> ------------------
> Sent using a device built by a popular California technology cult.
>
>> On Jul 29, 2017, at 09:46, Steve Cassidy <[hidden email]> wrote:
>>
>> It's quite a while since I posted here. And a while since I've read any posts – there were almost 2000 unread emails in my FMExperts folder! (Which is very unusual; until last year I had been fully on top of Experts emails every since the list started up – and I don't really want to think about how long ago that was…)
>>
>> Anyhow, a question for you about scheduled scripts on the server. Basically, I'd like to know what kind of load they are putting on the server and how much I can reduce the interval between runs without causing problems.
>>
>> The background is this:  for one of my clients, I've moved a lot of inventory transaction processing to a scheduled script run by the server (a Mac Mini; I don't have the specs to hand). The server handles at most 20 Filemaker clients (and nothing else).
>>
>> A server-side script runs once every 30 minutes, at which time it processes records held in a queue table. Generally, records are added to the queue in small batches – perhaps 20 stock movements within a few minutes, that kind of thing. Very often there are no records to process when the 30-minute script runs – in which case the script simply does a find that comes up with no found records and then exits. Twice a day (at night), major shipping routines run and these can generate 500 or more queue records. On a client, 500 queue records take anything up to 45 minutes to process. On the server about 30 seconds. (Each queued transaction involves multiple steps – such as determining the lot to be shipped, breaking open bulk cases, creating low inventory notifications, etc.)
>>
>> The point is that some inventory transactions are more time-critical that others; if a user moves inventory from one warehouse to another, they like to be able to see that the transaction has happened – and also to use the moved inventory for some further purpose. A 30-minute interval is not ideal.
>>
>> So, would there be any concerns with running this server-side script every five minutes, say? Or even every two minutes? As explained above, for the most part the processing is minimal (and is completed in less than two seconds). Once or twice a night (when no users are generally connected), there are these heavier batches of records that take the server 30 seconds to process. I'm thinking about slowdowns noticeable by users, or perhaps what might happen if the scheduled script runs very slowly for some reason and bumps into the next run time...
>>
>> Thanks for any hints, particularly if anyone actually runs a scheduled script on a very short interval.
>>
>> Steve
>> _______________________________________________
>> FMPexperts mailing list
>> [hidden email]
>> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scheduled script run interval

Steve Cassidy-3

On 29 Jul 2017, at 16:45, Justin <[hidden email]> wrote:

> Steve,
>    Another approach could be to add a 'priority' value to each queue record.  You mentioned that some things were more time sensitive - those records would get a high priority.  Then when your scheduled script runs it would search for records with a certain priority based on how long it had been since the last time it ran.  The scheduled script could run every two minutes - it could search for highest priority items each time; but would only look for mid priority records if, say the minutes of the system clock = 32 or some such (the schedule would need to be set so it would run at the 32 minute); really low priority things would only get searched for if the hour = 1am or some such.  (You have to be a bit careful because you don't have 100% control over when the script actually runs - other processes can delay the starting time a bit.)
>
> You can, of course, search for multiple groups of priorities at each run based on the time ( or whatever calculation you come up with).  For example, if its the 30 minute run you can search for the 2 minute records, the 5 minute records, and the 30 minute records, to process them all in one go.
>
> The short description is that you search for specific records based on the time of day or minute, and based on a priority you define.
>
> --  Justin

Ah… That's an interesting approach.

I already have a specified 'earliest transaction time' in queue records – because we have certain types of transactions during the day that have to be completed before the day's main shipment. That's how the majority of the queue requests get processed middle of the night – I can set them to be processed no earlier than 3am. It also means that, during the workday, there are either no records to process at each script run, or just a few handfuls. Using this idea I could finesse still further…

But whatever I do, if I want a minimum five-minute lag time, then the scheduled script has to run every five minutes. Even with no queue records to process, it has to do something – a find and a bit of housekeeping. Is the overhead on that so minimal for a server that I don't need to worry about it?

Steve




_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scheduled script run interval

JClose
I would say yes - but you have to be a bit careful with Psos and scheduled script calls - there's work you can do to make them as minimally invasive as possible.  When you run a script as a scheduled process, the system will still open the file the same as any user session.  So everything that happens when the file opens will happen. You can short circuit any startup scripts by checking for if it's running on the server, and be sure to use layouts that are minimalist.  It helps lessen the load.

It could also depend on you serve config - something with small amounts of RAM or processor specs will be more taxed.

But in general , yes, it's ok to run a small something very frequently.

-- Justin

> On Jul 29, 2017, at 09:06, Steve Cassidy <[hidden email]> wrote:
>
>
>> On 29 Jul 2017, at 16:45, Justin <[hidden email]> wrote:
>>
>> Steve,
>>   Another approach could be to add a 'priority' value to each queue record.  You mentioned that some things were more time sensitive - those records would get a high priority.  Then when your scheduled script runs it would search for records with a certain priority based on how long it had been since the last time it ran.  The scheduled script could run every two minutes - it could search for highest priority items each time; but would only look for mid priority records if, say the minutes of the system clock = 32 or some such (the schedule would need to be set so it would run at the 32 minute); really low priority things would only get searched for if the hour = 1am or some such.  (You have to be a bit careful because you don't have 100% control over when the script actually runs - other processes can delay the starting time a bit.)
>>
>> You can, of course, search for multiple groups of priorities at each run based on the time ( or whatever calculation you come up with).  For example, if its the 30 minute run you can search for the 2 minute records, the 5 minute records, and the 30 minute records, to process them all in one go.
>>
>> The short description is that you search for specific records based on the time of day or minute, and based on a priority you define.
>>
>> --  Justin
>
> Ah… That's an interesting approach.
>
> I already have a specified 'earliest transaction time' in queue records – because we have certain types of transactions during the day that have to be completed before the day's main shipment. That's how the majority of the queue requests get processed middle of the night – I can set them to be processed no earlier than 3am. It also means that, during the workday, there are either no records to process at each script run, or just a few handfuls. Using this idea I could finesse still further…
>
> But whatever I do, if I want a minimum five-minute lag time, then the scheduled script has to run every five minutes. Even with no queue records to process, it has to do something – a find and a bit of housekeeping. Is the overhead on that so minimal for a server that I don't need to worry about it?
>
> Steve
>
>
>
>
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scheduled script run interval

Steve Cassidy-3

On 29 Jul 2017, at 17:47, Comcast IMAP <[hidden email]> wrote:

> I would say yes - but you have to be a bit careful with Psos and scheduled script calls - there's work you can do to make them as minimally invasive as possible.  When you run a script as a scheduled process, the system will still open the file the same as any user session.  So everything that happens when the file opens will happen. You can short circuit any startup scripts by checking for if it's running on the server, and be sure to use layouts that are minimalist.  It helps lessen the load.
>
> It could also depend on you serve config - something with small amounts of RAM or processor specs will be more taxed.
>
> But in general , yes, it's ok to run a small something very frequently.

Thanks Justin

I'm thinking of running the script every 5 minutes on a trial basis.

I learned that I have to short-circuit startup scripts. And I do use minimal (blank) layouts. Given that in most cases the server-side script runs in less than a second, I can't imagine serious adverse effects. But if any show up I think I'll try prioritising the queue and throttling the number of transactions that are handled per run.

Thanks to all for contributing.

Steve
_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Scheduled script run interval

Richard DeShong
Hi Steve,
Now that you mention it, it has been a long while (at least in internet
time).

How about having both a "robot" client and the server scheduled scripts?

You can setup a client to run in a loop looking for queued records to
process.  Inside the loop, the client is looking for one queued record that
it can lock.  When it finds one, process it, and end of loop.  So this is a
loop-within-a-loop:
LOOP
   Find all queued rcds
   Loop
      Can I lock this rcd?
   EndLoop
   If Got-A-Lock
      Process Rcd
   EndIf
ENDLOOP

And then also have a scheduled script run on the server.  This script will
"pick up the slack" if there is suddenly a large number of queued rcds.
While the robot client will typically process rcds almost immediately.

It is important for both scripts (or they could be the same script) to not
assume that it is okay to process a record just because it is in the queue.

Finally, if you know that there are typical times of the day or week when
the queue might get slammed, then you can have "normal" schedule (meaning a
casual timing) for the off-peak times, and another schedule for the
possibly-peak times that has tighter timing.  The two should not run at the
same time.
--
Richard



From: Steve Cassidy Sent: Saturday, July 29, 2017 12:25 PM

On 29 Jul 2017, at 17:47, Comcast IMAP <[hidden email]> wrote:

> I would say yes - but you have to be a bit careful with Psos and scheduled
script calls - there's work you can do to make them as minimally invasive as
possible.  When you run a script as a scheduled process, the system will
still open the file the same as any user session.  So everything that
happens when the file opens will happen. You can short circuit any startup
scripts by checking for if it's running on the server, and be sure to use
layouts that are minimalist.  It helps lessen the load.
>
> It could also depend on you serve config - something with small amounts of
RAM or processor specs will be more taxed.
>
> But in general , yes, it's ok to run a small something very frequently.

Thanks Justin

I'm thinking of running the script every 5 minutes on a trial basis.

I learned that I have to short-circuit startup scripts. And I do use minimal
(blank) layouts. Given that in most cases the server-side script runs in
less than a second, I can't imagine serious adverse effects. But if any show
up I think I'll try prioritising the queue and throttling the number of
transactions that are handled per run.

Thanks to all for contributing.

Steve
_______________________________________________
It's quite a while since I posted here. And a while since I've read any
posts - there were almost 2000 unread emails in my FMExperts folder! (Which
is very unusual; until last year I had been fully on top of Experts emails
every since the list started up - and I don't really want to think about how
long ago that was.)

Anyhow, a question for you about scheduled scripts on the server. Basically,
I'd like to know what kind of load they are putting on the server and how
much I can reduce the interval between runs without causing problems.

The background is this:  for one of my clients, I've moved a lot of
inventory transaction processing to a scheduled script run by the server (a
Mac Mini; I don't have the specs to hand). The server handles at most 20
Filemaker clients (and nothing else).

A server-side script runs once every 30 minutes, at which time it processes
records held in a queue table. Generally, records are added to the queue in
small batches - perhaps 20 stock movements within a few minutes, that kind
of thing. Very often there are no records to process when the 30-minute
script runs - in which case the script simply does a find that comes up with
no found records and then exits. Twice a day (at night), major shipping
routines run and these can generate 500 or more queue records. On a client,
500 queue records take anything up to 45 minutes to process. On the server
about 30 seconds. (Each queued transaction involves multiple steps - such as
determining the lot to be shipped, breaking open bulk cases, creating low
inventory notifications, etc.)

The point is that some inventory transactions are more time-critical that
others; if a user moves inventory from one warehouse to another, they like
to be able to see that the transaction has happened - and also to use the
moved inventory for some further purpose. A 30-minute interval is not ideal.

So, would there be any concerns with running this server-side script every
five minutes, say? Or even every two minutes? As explained above, for the
most part the processing is minimal (and is completed in less than two
seconds). Once or twice a night (when no users are generally connected),
there are these heavier batches of records that take the server 30 seconds
to process. I'm thinking about slowdowns noticeable by users, or perhaps
what might happen if the scheduled script runs very slowly for some reason
and bumps into the next run time...

Thanks for any hints, particularly if anyone actually runs a scheduled
script on a very short interval.

Steve

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Loading...