Out of memory error on Calibre import

12 posts / 0 new
Last post
Mr Reese
Mr Reese's picture
Out of memory error on Calibre import

I have tried twice to import a Calibre database into AEM Pro. The database includes 16,431 books (37 MB file). On the first attempt (which took several hours), AEM flagged one book due to name length over a certain number of characters, but continued the scan. Once complete, AEM would NOT save the import.

I attempted a second scan overnight (unattended operation). When I checked in the morning, AEM failed to perform the import and displayed an out of memory error.

I am using Win10 Pro 64-bit on an i7 w/16 GB RAM.

Please advise.

admin
admin's picture

please send your calibre database to support@alfaebooks.com

Mr Reese
Mr Reese's picture

Email sent!

Mr Reese
Mr Reese's picture

I received the corrected database from Yuriy. It opened fine. However, all the file links are now incorrect. They point to here:

C:\Users\Yuriy\Desktop\metadata\...

How do I correct this?

admin
admin's picture

please try to do the import on your computer

Mr Reese
Mr Reese's picture

Using Alfa eBooks 6.0.0.2, the import is incredibly slow (as in about a book every 30 seconds)

The program is only using 181MB RAM, and about 35% CPU

I used File > Import from > Alfa eBooks Manager Database...

My PC specifications:

i7-6500U @ 2.50GHz
16 GB RAM
Win 10 Pro 64-bit

Given my PC specs and a database including 16,431 books, approximately how long should a typical import of this size take?

admin
admin's picture

10 minutes

Mr Reese
Mr Reese's picture

Even on a fresh install with nothing running in the background, it's only loading about one book every 10 seconds. At that rate, it would take two days to simply import the database you sent me.

I scanned my PC for books, to include creating thumbnails and covers. All 12,711 books are in a consolidated directory on my main SSD. The scanning process took about 5 hours. That works out to about 1.4 seconds per book. Not sure if that's good or bad.

It seemed like everything was fine until I hit save, then the system crashed with yet another out of memory error. AEM did NOT save anything. A monumental waste of time. After over a week trying to make AEM work in any fashion, I have nothing to show for it.

I can't properly explain how frustrating this has been.

I will try, yet again, to scan for books and go from there. Maybe I can simply reduce the size scanned. At this point I just don't know.

admin
admin's picture

Please try to open the database that we sent to you and use Tools -> FileFinder to re-attach files to the books in the database

Mr Reese
Mr Reese's picture

So that process took two hours... It updated the file links but failed to update the covers.

I ran the process again. I made sure the "Include Book Cover" option was checked. Same time and result.

I was able to use the scan function to simply import the books themselves into a new database. No metadata, no covers. That took about 15 minutes. Of course, all I have in that database is the barest of info now.

I then tried to run the same scan, this time including metadata. That took about 30 minutes... And simply stopped. It didn't import anything.

To recap:

I now have one working database of almost 14,000 books and nearly 13,000 genres that should really be tags... With no covers.

I have another working database with no metadata. I attempted to run a web update without downloading covers. It froze five minutes and 59 books later.

One would think this process shouldn't be so difficult.

admin
admin's picture

Hi, I just checked: File Finder finds and attaches covers.
Please make a small experiment with a new database if it works on your computer

You can try to scan your library with all options unchecked to quickly import all books and then use Books -> File Parser to retrieve metadata part by part.

We'll make the Calibre genre import option, but for now you also can use a free software CompactView to edit the database file (.SDF) - for example to quickly delete all genres

Mr Reese
Mr Reese's picture

I created another database and scanned with all options unchecked. That process went smoothly. I then used File Parser, using title letters as the "part." I had no problem running File Parser with a group of 500 books, 1,000 books, and 3,200 books. This doesn't identify the problem, but gives me a workaround.

As for CompactView, I can open the .sdf file and edit individual cells in any table, but can't delete an entire row or group of rows.