FMProAdv 16.05 importing issue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

FMProAdv 16.05 importing issue

Keith M. Rettig
Anyone else having this issue?

I am importing records from one copy of the database on the hard drive to the copy open on Filemaker Pro Advanced 16.05 (the most recent update) and it is taking forever.  120,000 records are only two-thirds done after 13 hours!  It has never taken me this long to move records between various copies of this database (I am trying to put them in their own db by year).  I am worried that it is something with 16.05; but if not, I would like some ideas why something would change like this?  Yes there are a bunch of fields that are indexed but that hasn’t been that much of an issue before; it seems like it is adding to the index one record at a time now rather than the whole imported set at once.

I have to move a couple million records next.  I am going to turn off all of the indexing as a result of this current experience.  But maybe I will wait until the next update if anyone supports blaming the update for the slow performance.

Thanks for any ideas.

Keith.
_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|

Re: FMProAdv 16.05 importing issue

Patrick Neame
Hello Keith,

I did an import with the latest update, only about 3000 records, but had the distinct impression that it was taking longer than I expected, long enough for me to stop watching it and go and do something else, something more time consuming than make a cuppa.

Patrick Neame

> On 12 Mar 2018, at 00:01, Keith M. Rettig <[hidden email]> wrote:
>
> Anyone else having this issue?
>
> I am importing records from one copy of the database on the hard drive to the copy open on Filemaker Pro Advanced 16.05 (the most recent update) and it is taking forever.  120,000 records are only two-thirds done after 13 hours!  It has never taken me this long to move records between various copies of this database (I am trying to put them in their own db by year).  I am worried that it is something with 16.05; but if not, I would like some ideas why something would change like this?  Yes there are a bunch of fields that are indexed but that hasn’t been that much of an issue before; it seems like it is adding to the index one record at a time now rather than the whole imported set at once.
>
> I have to move a couple million records next.  I am going to turn off all of the indexing as a result of this current experience.  But maybe I will wait until the next update if anyone supports blaming the update for the slow performance.
>
> Thanks for any ideas.
>
> Keith.
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|

Re: FMProAdv 16.05 importing issue

Bob Gossom
Keith,

I don’t have any idea if the most recent update affects imports or not. But past versions of FMP (10? 11? awhile ago) would slow down as large imports progressed. Some things that helped included chunking the imports; bring in several smaller sets. Some developers have reported faster times with doing an export to a merge file, and importing from that. Note that if you do export first, it will be a LOT faster if you sort the fields by type first, noting where the calculations start and only exporting the data fields.

Alternatively, v16 got much faster at deleting records. You might do better by duplicating the entire file, and then deleting the years you don’t want.

Bob Gossom
Absolute Advantage, Inc.

> On Mar 12, 2018, at 5:03 AM, Patrick Neame <[hidden email]> wrote:
>
> Hello Keith,
>
> I did an import with the latest update, only about 3000 records, but had the distinct impression that it was taking longer than I expected, long enough for me to stop watching it and go and do something else, something more time consuming than make a cuppa.
>
> Patrick Neame
>
>> On 12 Mar 2018, at 00:01, Keith M. Rettig <[hidden email]> wrote:
>>
>> Anyone else having this issue?
>>
>> I am importing records from one copy of the database on the hard drive to the copy open on Filemaker Pro Advanced 16.05 (the most recent update) and it is taking forever.  120,000 records are only two-thirds done after 13 hours!  It has never taken me this long to move records between various copies of this database (I am trying to put them in their own db by year).  I am worried that it is something with 16.05; but if not, I would like some ideas why something would change like this?  Yes there are a bunch of fields that are indexed but that hasn’t been that much of an issue before; it seems like it is adding to the index one record at a time now rather than the whole imported set at once.
>>
>> I have to move a couple million records next.  I am going to turn off all of the indexing as a result of this current experience.  But maybe I will wait until the next update if anyone supports blaming the update for the slow performance.
>>
>> Thanks for any ideas.
>>
>> Keith.
>> _______________________________________________
>> FMPexperts mailing list
>> [hidden email]
>> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
>
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|

Re: FMProAdv 16.05 importing issue

Keith M. Rettig
In reply to this post by Patrick Neame
So I couldn’t take any longer, the most recent 100,000 record import took so long I don’t remember when I started the import.  I moved the two databases (source and destination) to a machine that is half as powerful and one-third the RAM but still running 16.04 (both are running MacOS 13.3.3 with supplemental).  Well...I just finished importing the remaining 457,000 records in 20 minutes.

Unfortunately this doesn’t prove my theory.  The problem could still just be on my machine.  But it does add to the circumstantial evidence.
I am going to reboot my machine to exclude that explanation next.

The new data was enough for me to push out this message again to see if others are experiencing a slowdown in imports and deletes with 16.05 on 13.3.3+SU?
If you have both upgrades and you are NOT experiencing such slowdowns, it would be helpful for you to call that out as well.  It would help me search for the problem.

Thanks,
Keith.

> On Mar 12, 2018, at 5:03 AM, Patrick Neame <[hidden email]> wrote:
>
> Hello Keith,
>
> I did an import with the latest update, only about 3000 records, but had the distinct impression that it was taking longer than I expected, long enough for me to stop watching it and go and do something else, something more time consuming than make a cuppa.
>
> Patrick Neame
>
>> On 12 Mar 2018, at 00:01, Keith M. Rettig <[hidden email]> wrote:
>>
>> Anyone else having this issue?
>>
>> I am importing records from one copy of the database on the hard drive to the copy open on Filemaker Pro Advanced 16.05 (the most recent update) and it is taking forever.  120,000 records are only two-thirds done after 13 hours!  It has never taken me this long to move records between various copies of this database (I am trying to put them in their own db by year).  I am worried that it is something with 16.05; but if not, I would like some ideas why something would change like this?  Yes there are a bunch of fields that are indexed but that hasn’t been that much of an issue before; it seems like it is adding to the index one record at a time now rather than the whole imported set at once.
>>
>> I have to move a couple million records next.  I am going to turn off all of the indexing as a result of this current experience.  But maybe I will wait until the next update if anyone supports blaming the update for the slow performance.
>>
>> Thanks for any ideas.
>>
>> Keith.
>> _______________________________________________
>> FMPexperts mailing list
>> [hidden email]
>> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
>
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
Reply | Threaded
Open this post in threaded view
|

Re: FMProAdv 16.05 importing issue

Patrick Neame
Just to give you a bit more information,

3500 Records, Mac Pro Late 2013 running 10.13.3 with FM Advanced 16.0.5.500 imported from a file hosted off site at a data centre.


Patrick Neame

> On 14 Mar 2018, at 08:00, Keith M. Rettig <[hidden email]> wrote:
>
> So I couldn’t take any longer, the most recent 100,000 record import took so long I don’t remember when I started the import.  I moved the two databases (source and destination) to a machine that is half as powerful and one-third the RAM but still running 16.04 (both are running MacOS 13.3.3 with supplemental).  Well...I just finished importing the remaining 457,000 records in 20 minutes.
>
> Unfortunately this doesn’t prove my theory.  The problem could still just be on my machine.  But it does add to the circumstantial evidence.
> I am going to reboot my machine to exclude that explanation next.
>
> The new data was enough for me to push out this message again to see if others are experiencing a slowdown in imports and deletes with 16.05 on 13.3.3+SU?
> If you have both upgrades and you are NOT experiencing such slowdowns, it would be helpful for you to call that out as well.  It would help me search for the problem.
>
> Thanks,
> Keith.
>
>> On Mar 12, 2018, at 5:03 AM, Patrick Neame <[hidden email]> wrote:
>>
>> Hello Keith,
>>
>> I did an import with the latest update, only about 3000 records, but had the distinct impression that it was taking longer than I expected, long enough for me to stop watching it and go and do something else, something more time consuming than make a cuppa.
>>
>> Patrick Neame
>>
>>> On 12 Mar 2018, at 00:01, Keith M. Rettig <[hidden email]> wrote:
>>>
>>> Anyone else having this issue?
>>>
>>> I am importing records from one copy of the database on the hard drive to the copy open on Filemaker Pro Advanced 16.05 (the most recent update) and it is taking forever.  120,000 records are only two-thirds done after 13 hours!  It has never taken me this long to move records between various copies of this database (I am trying to put them in their own db by year).  I am worried that it is something with 16.05; but if not, I would like some ideas why something would change like this?  Yes there are a bunch of fields that are indexed but that hasn’t been that much of an issue before; it seems like it is adding to the index one record at a time now rather than the whole imported set at once.
>>>
>>> I have to move a couple million records next.  I am going to turn off all of the indexing as a result of this current experience.  But maybe I will wait until the next update if anyone supports blaming the update for the slow performance.
>>>
>>> Thanks for any ideas.
>>>
>>> Keith.
>>> _______________________________________________
>>> FMPexperts mailing list
>>> [hidden email]
>>> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
>>
>> _______________________________________________
>> FMPexperts mailing list
>> [hidden email]
>> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au
>
> _______________________________________________
> FMPexperts mailing list
> [hidden email]
> http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au

_______________________________________________
FMPexperts mailing list
[hidden email]
http://lists.ironclad.net.au/listinfo.cgi/fmpexperts-ironclad.net.au