Thousands of notes with the same tag slows down search

< Next Topic | Back to topic list | Previous Topic >

Posted by Simon Bolivar
Mar 17, 2013 at 02:00 PM

I am looking at an outliner that offers tagging where I can for example create a database called quotations with the idea that each single quote would belong in a single note/page. I would then tag this note eg Twain Mark, 19th C, USA, *then add the topic(s) as relevant”. Each note then may have approx 5 tags. After thousands of such notes entered the index of tags will grow large, and some tags eg ‘War’ may have hundreds or thousands of notes associated with it. Can you suggest a software that firstly can have a database of many thousands of tags without slowing down the performance of the software, and secondly that can have many thousands of entries associated with a given tag eg ‘War’ that when I open that tag to see the entries the software can manage that without being unresponsive for minutes at a time, and preferably respond in a few seconds?

Thanks, Simon

Posted by Dr Andus
Mar 17, 2013 at 02:26 PM

Have you tried or considered ConnectedText? You can keep thousands of such entries in one project (although certain fancy queries and transclusions can slow it down eventually, but those are optional). Check Steve’s blog post on this type of use:
http://welcometosherwood.wordpress.com/2012/04/08/connectedtext-does-the-work-for-you-or-a-lot-of-it-anyway/

Please note that CT is a free-form database, so there are many different ways of solving this kind of task. You can freely decide whether something is going to be a document (topic) name, or a category (tag, label), or an attribute or property, depending on what you want to get out of it.

Posted by Alexander Deliyannis
Mar 17, 2013 at 04:14 PM

Simon, I’ve been tempted since I first saw your name in this forum to ask you if your surname is really Bolivar, and if you happen to be from Chile, but I can leave it for another time… :-)

On your subject, I’ve been using Evernote for several years and have more than 10,000 notes in my notebooks, many of them including rich content (mostly images and sometimes audio files). The database is about 1.4 Gbytes. I have a couple of hundred tags, organised in a hierarchy. As far as I see, none of my tags happen to have more than a couple of hundred items, but switching from a tag to the other takes less than a second, and this time seems independent of the number of items under the various tags I switch between.

Evernote uses SQLite as its infrastructure; I would say that both SQLite and Evernote have a tremendous user base and that if there were issues of scaleability they would have turned up long ago.

Depending on what you will want to do with your notes, Connected Text might offer more useful features, e.g. for organising them around specific scientific papers. In fact, there is little you can do with your notes after they are entered in Evernote, other than search them, tag them and edit them—you can of course select some and export them as HTML for use in other programs, but that’s it. But I believe that Evernote has the advantage in ease of collection of rich content, plus ubiquitous availability.

Posted by Simon Bolivar
Mar 19, 2013 at 10:02 PM

Dr Andus wrote:
Have you tried or considered ConnectedText? You can keep thousands of
>such entries in one project (although certain fancy queries and
>transclusions can slow it down eventually, but those are optional).
>Check Steve’s blog post on this type of use:
>http://welcometosherwood.wordpress.com/2012/04/08/connectedtext-does-the-work-for-you-or-a-lot-of-it-anyway/
> >Please note that CT is a free-form database, so there are many
>different ways of solving this kind of task. You can freely decide
>whether something is going to be a document (topic) name, or a category
>(tag, label), or an attribute or property, depending on what you want to
>get out of it.

Thanks for the feedback Dr Andus!

Of all of the outliners that I’ve seen and read about, the two that most interest me for their power are Infoqube and ConnectedText. The latter, which I know from reading your posts, you use extensively and are very impressed with. I read the link you posted for CT, thanks very much for that. I had a look around on the blog and this is a software I’d very much like to roadtest. However at this time I am progressing with data inputting diffuse and extensive information from EssentialPIM and Mempad into Rightnote, which is more advanced as a notetaker than the other two (which are very limited in comparison with the many fascinating softwares mentioned on this great forum) and when I have fine tuned my information storage I can then look at the next step of putting 1 or more specific topics into the likes of CT and then trial and error it and spend time learning how to get the most from this software.

I’ve purposely bookmarked your blog for when i do try CT. I will get back to you, (and ask for advice I expect) when I do try it out. Thanks again!!

Posted by Simon Bolivar
Mar 19, 2013 at 10:24 PM

Alexander Deliyannis wrote:
Simon, I’ve been tempted since I first saw your name in this forum to
>ask you if your surname is really Bolivar, and if you happen to be from
>Chile, but I can leave it for another time… :-)
> >On your subject, I’ve been using Evernote for several years and have
>more than 10,000 notes in my notebooks, many of them including rich
>content (mostly images and sometimes audio files). The database is about
>1.4 Gbytes. I have a couple of hundred tags, organised in a hierarchy.
>As far as I see, none of my tags happen to have more than a couple of
>hundred items, but switching from a tag to the other takes less than a
>second, and this time seems independent of the number of items under the
>various tags I switch between.
> >Evernote uses SQLite as its infrastructure; I would say that both SQLite
>and Evernote have a tremendous user base and that if there were issues
>of scaleability they would have turned up long ago.
> >Depending on what you will want to do with your notes, Connected Text
>might offer more useful features, e.g. for organising them around
>specific scientific papers. In fact, there is little you can do with
>your notes after they are entered in Evernote, other than search them,
>tag them and edit them—you can of course select some and export them as
>HTML for use in other programs, but that’s it. But I believe that
>Evernote has the advantage in ease of collection of rich content, plus
>ubiquitous availability.

Hi Alexander, I replied to Dr Andus above reagrding ConnectedText and it is a programme which I am very interested in but which I will try at a later stage, as I am currently consolidating all of my notes into Rightnote. I have an Evernote account but haven’t used it very much, I actually used onenote a lot more but haven’t moved forward with the latter product. RN was that it was offered at 50% discount on bits du jour and at that time (Nov 2012) I was torn between UltraRecall, myinfo and WhizFolders; I was trialling them all for most of 2012, but I took the plunge for RN, as as enjoyable as it is to learn different softwares, I really need to pool my data into one software, even if there are very good cases for the ones that I have tried, and for many more besides. Funnily enough, I saw recently that UR was reduced on bits du jour so it was a matter of timing! Que sera sera.

Anyway back to Evernote, one aspect of RN that is helpful, is it’s integration with EN, so I am minded to do a simple experiment entering quotations into EN and seeing how that works with RN.

I don’t care to leave you with a mystery, I am English, living in the UK, but I did travel in South America; Peru and Colombia, (I do have friends in Chile too), and I visited Santa Marta where Simon Bolivar died. I took a shine to his exploits and hence my nom de plume on this forum.

Thanks again.

Back to topic list