ebook reader

onemorechance

I am wondering about data persistence in Android.

I am looking to make an app which will allow users to read, download and store books.

I am wondering what are the best options for data storage and retrieval.

I guess my options are:

sqllite, xml or json resource files.

I think that sqllite would be the best option as it's easier to get only the current data needed doesn't have the overhead of xml or json parsing.

Also, I think I should be able to offer new books through simply the download of a DB / table dump of the new book.

Anyone have any comments or suggestions?

I would rather have only one, e.g. not an xml resource file which gets converted to sqllite at runtime. I would rather just provide the data in one format only, to save space.

The Corinthian

I would have thought you might want to let the user store their ebooks as files (epub or whatever format) on the phone, rather than import them into a database.

onemorechance

Thanks for the reply.

I did not think of epub and I don't know much about it. I will look into it.

I guess that epub gives the user the opportunity to use the book on more devices; but I wanted to move away from XML as I worry about performance issues with XML parsing.

Also, in my app, the books are to be interactive and this is to be the main use of the books.

Storing in a db suits my needs more as it will more easily allow the user to interact with and even modify the book.

I figure that that db querying will be more efficient than xml parsing too.

My idea is that I can offer new books through SQL table dumps and that the books are stored on the user's device in db tables only.

If transfer of the books to other devices is not so important or being able to view in other ebook readers / apps, I think that db storage is the best option in terms of ease of access and modification.

It might be an option to offer the book in epub also, as convenience to the user.

Other than that (mostly portability), do ye think that the db is better than xml, json, xhtml (epub)?

The Corinthian

onemorechance wrote: »

I guess that epub gives the user the opportunity to use the book on more devices; but I wanted to move away from XML as I worry about performance issues with XML parsing.

To begin with, I suspect you'll find more performance issues with storing large text or binary items in a database than parsing them.

Secondly, you're writing an ebook app, which means you probably should know what an ebook is and what formats are used. This is because if you don't support them, your app isn't going to be very successful.

Storing in a db suits my needs more as it will more easily allow the user to interact with and even modify the book.

I think you need to explain the business logic of your reader here, because according to this, it's not a reader and they're not actually ebooks.

My idea is that I can offer new books through SQL table dumps and that the books are stored on the user's device in db tables only.

Images too? What if they want to import ebook files? Export them back to files? I presume you're going to support conversion?

If transfer of the books to other devices is not so important or being able to view in other ebook readers / apps, I think that db storage is the best option in terms of ease of access and modification.

I disagree. It may make searching through the text of the ebook faster at best. Modification-wise, it really doesn't make a huge difference.

It might be an option to offer the book in epub also, as convenience to the user.

EPUB is only one format, and not even one of the most popular. I'd look more at Amazon's and Apple's formats.

Other than that (mostly portability), do ye think that the db is better than xml, json, xhtml (epub)?

A DB actually makes the ebooks less portable. Honestly, from what you've said to date, you're better off not going down the DB route.

onemorechance

The Corinthian wrote: »

To begin with, I suspect you'll find more performance issues with storing large text or binary items in a database than parsing them.

Secondly, you're writing an ebook app, which means you probably should know what an ebook is and what formats are used. This is because if you don't support them, your app isn't going to be very successful.

I think you need to explain the business logic of your reader here, because according to this, it's not a reader and they're not actually ebooks.

Images too? What if they want to import ebook files? Export them back to files? I presume you're going to support conversion?

It's not meant to be a general ebook reader; it's only for the books I will provide. It's not important that the books can be copied elsewhere or that it can read other ebooks. I guess the title is wrong, but it's an app to allow the user read and interact with books that I provide only.

The Corinthian wrote: »

I disagree. It may make searching through the text of the ebook faster at best. Modification-wise, it really doesn't make a huge difference.

EPUB is only one format, and not even one of the most popular. I'd look more at Amazon's and Apple's formats.

A DB actually makes the ebooks less portable. Honestly, from what you've said to date, you're better off not going down the DB route.

Fast access to the data is most important. Portability is only important once; allowing the user to download a new book. I think I will look at an epub format, or simply xml or json for this.

My main question really is what is the best format for quick access to data and db jumps out. Once the book is on the device, what is the best way and most efficient to access the data, especially non-sequencially; I just don't like the idea of parsing xml each time, especially as the books can become huge.

Splitting them into smaller parts will create lots of files; but this may be an option.

Thanks again for your reply, it has a lot of useful info.

The Corinthian

onemorechance wrote: »

It's not meant to be a general ebook reader; it's only for the books I will provide. It's not important that the books can be copied elsewhere or that it can read other ebooks. I guess the title is wrong, but it's an app to allow the user read and interact with books that I provide only.

OK, I see. Calling it an ebook reader confused the issue. Essentially you want a custom reader, with an emphasis on full text searchability, and interoperability with ebook formats is a low priority as, publishing-wise, you're going to be operating a closed garden policy. Is this essentially correct?

If this is the case, I'd forget about EPUB (unless you want to allow an export format, in which case it's one of the easiest to implement), although still use something like xhtml to store your text in the DB - you're still going to want to format and lay out your content.

I'd stick to using something like XML or Json for downloading the ebooks. If you're worried about bandwidth, look at compression related techniques, such as WBXML.

Fast access to the data is most important. Portability is only important once; allowing the user to download a new book. I think I will look at an epub format, or simply xml or json for this.

Just bare in mind, if you want to support images, you'll need to do that too. You can do so by simply referencing the image URL for download, or could also encode them into your XML/Json - but I'd council against that.

Storing images is another thing you'll need to deal with; in the DB as a blob, or keep them outside as separate files with only a reference to them in the DB.

My main question really is what is the best format for quick access to data and db jumps out. Once the book is on the device, what is the best way and most efficient to access the data, especially non-sequencially; I just don't like the idea of parsing xml each time, especially as the books can become huge.

If you're looking to do keyword searches within large slabs of text, then you might want to look at information retrieval libraries, rather than RDB's - such as Lucene. I've not used them myself, but believe they're very fast at searches, but a lot slower on indexing.

onemorechance

The Corinthian wrote: »

OK, I see. Calling it an ebook reader confused the issue. Essentially you want a custom reader, with an emphasis on full text searchability, and interoperability with ebook formats is a low priority as, publishing-wise, you're going to be operating a closed garden policy. Is this essentially correct?

If this is the case, I'd forget about EPUB (unless you want to allow an export format, in which case it's one of the easiest to implement), although still use something like xhtml to store your text in the DB - you're still going to want to format and lay out your content.

A custom reader, exactly, but it's the navigation of the book which has the emphasis. I want to provide multiple ways for the user to navigate (quickly) and read the book. While parsing an xml file can be quick, I like the idea of the db where I can access the bit I want only and directly. The main issue with this may be the offering of new books. I was hoping that I could provide an SQL dump file which will contain the book, rather than an xml file which will then get converted to a db table. I don't like the idea of having two versions of the same book on the users phone, especially if the xml file is only used once. But this may also be used as a feature, where the user can export the book and maybe re-import it later.

The Corinthian wrote: »

I'd stick to using something like XML or Json for downloading the ebooks. If you're worried about bandwidth, look at compression related techniques, such as WBXML.

Just bare in mind, if you want to support images, you'll need to do that too. You can do so by simply referencing the image URL for download, or could also encode them into your XML/Json - but I'd council against that.

Storing images is another thing you'll need to deal with; in the DB as a blob, or keep them outside as separate files with only a reference to them in the DB.

If you're looking to do keyword searches within large slabs of text, then you might want to look at information retrieval libraries, rather than RDB's - such as Lucene. I've not used them myself, but believe they're very fast at searches, but a lot slower on indexing.

You have given me lots of good advice and pointers! I hope that I will be able to return the favour someday!

The Corinthian

onemorechance wrote: »

I don't like the idea of having two versions of the same book on the users phone, especially if the xml file is only used once. But this may also be used as a feature, where the user can export the book and maybe re-import it later.

An xml feed would only be the means of transport; uploading can be done at runtime, without the user ever having direct access to the feed.

Of course, you can do this upload also as a DB, rather than an xml feed, but then there are a few considerations you'll need to take into account, with regards to copy-write protection and publishing.

Finally, you're going to have to consider DB format backward compatibility. Say over time, your DB format changes and you end up with three versions; will you make your reader compatible with all three? Or will you convert the older version to the latest version? And if the latter, you do know that once you have more than two versions published, you'll have to be able to support upgrade from any previous version, not just the last one?

onemorechance

The Corinthian wrote: »

An xml feed would only be the means of transport; uploading can be done at runtime, without the user ever having direct access to the feed.

Of course, you can do this upload also as a DB, rather than an xml feed, but then there are a few considerations you'll need to take into account, with regards to copy-write protection and publishing.

Finally, you're going to have to consider DB format backward compatibility. Say over time, your DB format changes and you end up with three versions; will you make your reader compatible with all three? Or will you convert the older version to the latest version? And if the latter, you do know that once you have more than two versions published, you'll have to be able to support upgrade from any previous version, not just the last one?

You mean if I change the structure of the tables in db? I don't see this happening; I would hope to design a book structure and leave it at that.

Even if this did happen, it would be the same issue with any other format; if I changed the xml or json structure, then the code would need to adapt to that.

The Corinthian

Fair enough. Backward compatibility that way makes far more sense.