Hello all,

The next stage in my attempts at organising myself is to find a tool to allow easy searching and access of my text and documents archive.

The situation is that, using PaperPort, I have scanned thousands of documents and images which are now on my hard disk, mainly as pdf files, and organised into folders. I also have (Word) document files of older articles and books, and various image scans.

I'd like to get this data into a program so that I could access relevant material using keywords. And extra would be if that program could also be used to sort and output that information, but that's not essential. Basically, I spend too much time searching files for old pearls of wisdom or having to think them up again. I want to avoid the "now, where did I see that screendump from .....?" and then having to spend hours searching.

I have UltraRecall, which is disappointing to me in this respect as it has some irritating limitations on the size of directories it imports and simply doesn't always do what it says on the box (and, of course, being UR, without error message). I could use WhizFolders for the text snippets, but it can't handle the pdf or image files easily. I bought IdeaMason when it was recently on Bitsdujour, but I haven't tried it yet and don't want to invest the time at the moment if it's not the solution.

Does anybody have any suggestions for me? By the way, it needs to be very robust - more so than UR, for example, because of the thousands of documents involved.

Thanks in advance!

Graham

Hello Graham

If you have a large number of documents in different formats already organised into folders, you could just use a good indexer such as dtSearch or X1 to find what you want when you need it. X1 is now free.

Or am I missing something crucial?

Jack

Hello Jack,

I have X1 and, after posting, I also realised that I can easily use PaperPort to search as well. I suppose what I'm missing is depth and quality of search, and ease of use of returned information.

Using PaperPort, for example, if I look for "Spanish given names", first of all it gives me false positives (because it cannot properly OCR non-English data); and the data is not highly keyworded, so I may have to look through 20 documents to find the relevant details, and the search text isn't highlighted. Also, it doesn't (always) index as it scans, so I need to re-index after adding new documents, and that is agonisingly slow.

I think I'd like to be able to view the returned texts easily and be able to give each document a powerful set of keywords. Also I'd like to create a parallel snippets data system whereby I can snip my articles into paragraphs and store those, so that I can find these snippets and quotes without having to dredge through articles.

The dreaded ADM and IdeaMason come to mind when I formulate these ideas. I'm wondering if IM (I won't touch ADM again) is a good way to go, or whether this is just too much wishful thinking ....

Graham


Jack Crawford wrote:

Hello Graham

If you have a large number of documents in different formats already
organised into folders, you could just use a good indexer such as dtSearch or X1 to find
what you want when you need it. X1 is now free.

Or am I missing something
crucial?

Jack

Graham Rhind wrote:

I think I'd like to be able to
view the returned texts easily and be able to give each document a powerful set of
keywords. Also I'd like to create a parallel snippets data system whereby I can snip my
articles into paragraphs and store those, so that I can find these snippets and quotes
without having to dredge through articles.

Graham,

My gut feeling is that you are looking for something more elegant but I do have a rough solution for you. In Windows XP, if you right click on the icon for your document and go to the "Summary" tab you will see fields for "Categories", "Keywords", and "Comments". I have been known to keep text files on my desktop with categories and keywords listed for easy copying and pasting. The "Comments" field could be a place to put your snippets. These fields are searchable, at least with Windows Desktop Search. The other indexing search programs like X1 might do it as well.

Again, not terribly elegant but it is (almost disappointingly) simple.

Tom S.

Tom S. wrote:

Graham,

My gut feeling is that you are looking for something
more elegant but I do have a rough solution for you. In Windows XP, if you right click on
the icon for your document and go to the "Summary" tab you will see fields for
"Categories", "Keywords", and "Comments".

Ummm. Sorry. I meant right click, choose "Properties", then go to the "Summary" tab.

Tom S,

Thanks Tom - you're right, I crave elegance (speed and power in adding keywords, apart from anothing else) - but I didn't know that those properties were there! Useful and simple ...

Addendum about PaperPort - PaperPort will only work within its "own"directory, so all files created outside it can't be indexed or searched. Another reason to look for an alternative ...

Tom S. wrote:

My gut feeling is that you are looking for something
more elegant but I do have a rough solution for you. In Windows XP, if you right click on
the icon for your document and go to the "Summary" tab you will see fields for
"Categories", "Keywords", and "Comments". I have been known to keep text files on my
desktop with categories and keywords listed for easy copying and pasting. The
"Comments" field could be a place to put your snippets. These fields are searchable,
at least with Windows Desktop Search. The other indexing search programs like X1
might do it as well.

Again, not terribly elegant but it is (almost disappointingly)
simple.

Tom S.

At least in more recent versions of PaperPort (I have version 11), you can add any folder to show up and be managed in PaperPort - go to Tools -> Folder Manager.

One disappointing thing about PaperPort is that you can't scan to standard searchable PDFs with it - it will OCR and index scans for its own internal search engine using the basic product, but to make a standard text PDF (not image), you also have to have OmniPage Professional - which is $499. Ouch. As far as I can tell, this limits the usefulness of PaperPort in conjunction with external desktop search engines and other software that searches PDFs. Someone please correct me if I'm wrong.

I need to look for cheaper alternative scanning software that would scan and OCR to a normal text PDF. I don't deal with a huge amount of paper, so it's kind of on the back burner, but there has to be something out there. I'm pretty satisfied with Ultra Recall, but I would love to be able to drop scanned pdfs into it and actually be able to search the text content.

Check out this thread on the UR forum for a possible approach for UR users (and probably for desktop search engine users as well) : http://tinyurl.com/2pay7f - Microsoft Office Document Imaging creates tiffs that have their text info from OCR stored with them.I haven't done a lot of testing with this, but it seems to work okay.

Chris

I've been using IdeaMason heavily for 2 months now, and I'm pretty disappointed.

It won't work for what you want to do, because as far as I can tell, it doesn't really handle external documents well at all. There is a special "documents" tab that holds external documents, but those are not integrated into the rest of the program at all. In other words, that tab is nothing more than a link list. You can't associate any comments or even a citation with the external doc.
And, most disappointingly for your purposes, those documents can't be searched.

One work-around is that you can create a "library" item, which has a "source on the web" tab, and view your external document there -- it allows you to enter a local file:: url. Library items can have comments and citations associated with them, so that solves that problem. But you can't easily view your document outside of their viewer, and, again, IM search does not search within the external document.

IM has a lot of other flaws for what it's supposed to do. I used it to research and write a major paper for my final year of law school, which is due tomorrow. I was really excited about it, and thought that it would help structure my writing process. But it just slowed me down, and continues to do so, because of all the work-arounds I have to use -- and because it is so painfully slow. However, I do think it's a nice concept, and someday I'll write the IM folks with my wish-list and see if they're responsive.

One suggestion for Graham: what about Google Desktop Search? It gives you a preview of the search results inside the document, which is awesome. For Outlook emails it even caches the entire email so you can view it in the browser instead of launching outlook. And I find it is about 1000 times faster than Windows Search - plus Windows search doesn't preview the results and when I turn Windows indexing on it really slows down my machine.

Chris Murtland wrote:

At least in more recent versions of PaperPort (I have version 11), you can add any
folder to show up and be managed in PaperPort - go to Tools -> Folder Manager.

Thanks Chris - I had found that one it seems - and then forgot about it again :-)

One
disappointing thing about PaperPort is that you can't scan to standard searchable
PDFs with it - it will OCR and index scans for its own internal search engine using the
basic product, but to make a standard text PDF (not image), you also have to have
OmniPage Professional - which is $499. Ouch. As far as I can tell, this limits the
usefulness of PaperPort in conjunction with external desktop search engines and
other software that searches PDFs. Someone please correct me if I'm wrong.

I'm wondering whether standard text pdfs would work for me, as my documents contain many languages - though the pdfs that PaperPort produce are not ideal, at least I can be sure to see what was on the paper. Oh, and I should say, re-scanning isn't an option - apart from the time limitations, most of the original stuff has been recycled :-)

Graham

Thanks for this Dan. I have always been a little wary about starting working with IdeaMason - I always had a feeling that it would take me longer to learn how to work with the program than it would save in the writing process. You seem to be confirming my suspicions. I was wondering how well external documents were supported, so thanks for the information on that!

Graham

dan7000 wrote:

I've been using IdeaMason heavily for 2 months now, and I'm pretty disappointed.

It
won't work for what you want to do, because as far as I can tell, it doesn't really handle
external documents well at all. There is a special "documents" tab that holds
external documents, but those are not integrated into the rest of the program at all.
In other words, that tab is nothing more than a link list. You can't associate any
comments or even a citation with the external doc.
And, most disappointingly for
your purposes, those documents can't be searched.

One work-around is that you can
create a "library" item, which has a "source on the web" tab, and view your external
document there -- it allows you to enter a local file:: url. Library items can have
comments and citations associated with them, so that solves that problem. But you
can't easily view your document outside of their viewer, and, again, IM search does
not search within the external document.

IM has a lot of other flaws for what it's
supposed to do. I used it to research and write a major paper for my final year of law
school, which is due tomorrow. I was really excited about it, and thought that it would
help structure my writing process. But it just slowed me down, and continues to do so,
because of all the work-arounds I have to use -- and because it is so painfully slow.
However, I do think it's a nice concept, and someday I'll write the IM folks with my
wish-list and see if they're responsive.

One suggestion for Graham: what about
Google Desktop Search? It gives you a preview of the search results inside the
document, which is awesome. For Outlook emails it even caches the entire email so you
can view it in the browser instead of launching outlook. And I find it is about 1000
times faster than Windows Search - plus Windows search doesn't preview the results
and when I turn Windows indexing on it really slows down my machine.

Tom S. wrote:

My gut feeling is that you are looking for something
more elegant but I do have a rough solution for you. In Windows XP, if you right click on
the icon for your document and go to the "Summary" tab you will see fields for
"Categories", "Keywords", and "Comments".

Just running some quick tests on the Properties | Summary:

From Windows Explorer select multiple files, then from the Summary Tab select Advanced, chose a field to modify and add the desired value (text string), this will be applied to all selected files.

Could be a method to mass update. UR is supposed to be able to recognize these fields and import the values during an import.

Just sayin' there might be something here,
KenA

I'm not sure if this is a solution to your problem or not, but I think it is worth considering. With OneNote 2007 you can import PDFs (or any almost any files) as printouts -- so you can view the document as an image as well as having a link to the original. You can then ask OneNote to make the text in each image searchable -- ON performs an OCR and can handle English, French or Spanish.

The potential drawbacks are these:
1. You can import all the files in any one hard drive folder at one time, but they all go into the same page. That may or may not be a drawback... though it would be nice if MS offered the ability to have new pages created for each file.
2. As far as I can tell, you can choose one of these languages as the default language for the OCR/Search. You will need to manually change the default on those images that use a different language.

I'm sure there are other issues as well, but ON is worth considering.

Steve Z.

Thanks Stephen. Do you know of a way/URL to try OneNote 2007 alone? On the Microsoft site I can only find a URL to download a test version of the whole Office 2007 suite, and as I already have Office 2007 (without OneNote) that's not useful.

I've never liked the OneNote paradign, with the tabs along the top and down the side of the workspace, but I suspect it has the power that I need.

Graham

Stephen Zeoli wrote:

I'm not sure if this is a solution to your problem or not, but I think it is worth
considering. With OneNote 2007 you can import PDFs (or any almost any files) as
printouts -- so you can view the document as an image as well as having a link to the
original. You can then ask OneNote to make the text in each image searchable -- ON
performs an OCR and can handle English, French or Spanish.

The potential drawbacks
are these:
1. You can import all the files in any one hard drive folder at one time, but
they all go into the same page. That may or may not be a drawback... though it would be
nice if MS offered the ability to have new pages created for each file.
2. As far as I can
tell, you can choose one of these languages as the default language for the
OCR/Search. You will need to manually change the default on those images that use a
different language.

I'm sure there are other issues as well, but ON is worth
considering.

Steve Z.

Apologies, cancel that - I've found a download page.

Graham Rhind wrote:

Thanks Stephen. Do you know of a way/URL to try OneNote 2007 alone? On the Microsoft
site I can only find a URL to download a test version of the whole Office 2007 suite, and
as I already have Office 2007 (without OneNote) that's not useful.

Graham Rhind wrote:

Thanks Stephen. Do you know of a way/URL to try OneNote 2007 alone? On the Microsoft
site I can only find a URL to download a test version of the whole Office 2007 suite, and
as I already have Office 2007 (without OneNote) that's not useful.

I've never liked
the OneNote paradign, with the tabs along the top and down the side of the workspace,
but I suspect it has the power that I need.

Graham,

I can see how the ON paradigm would not be for everyone. Personally, I like it better than most PIMs, it feels more manageable. ON 2007 adds some additional navigational aids, including a notebook/section/folder menu that rests on the left side of the screen. This makes the top row of tabs unnecessary, but for some reason, MS didn't include the ability to turn these tabs off.

I'll be interested to read what you think of ON once you give it a try.

Steve Z.

Graham,

Have you heard of / tried eDoc Plus? From what I've read it should be similar to Paperport but quite focused on the document management side.

alx

I felt it was time to summarise some of my findings on this search and the related one for a task manager in case they prove useful to the group members.

I looked at masses of programs and short-listed a few for both document management and task management.

On the task management side:

- The release date for the new beta of TaskPilot keeps slipping, so a review of that one is on hold.
- Priorganizer is very buggy, and though the developer is very quick to answer support requests, the waiting time for a new version is too long in my opinion. I found the little pop-up window for notes and to show "next tasks" useful, but as I couldn't use it as my main task manager with its current bugs, I've sidelined it until a better version is released.
- FusionDesk: ditto. It's a great concept, but given its bugs (some of which cause data deletion, such as when syncing to Outlook), if I were the developer I'd be producing a new version for release every day, but the developer has taken the "wait until everything is resolved" path, which is not to my taste. I also find the support patchy (I am of the opinion that if you host a forum on your site, you should answer all of the questions posted there).
- UltraRecall: I have given up on UR except to keep a client database in it. I think it suffers terribly because its developers are too technical and don't really (in my mind) understand how many end-users, except technical ones, would use such a product. It is unnecessarily complicated - it could be so much better - and, although technical myself, I don't want to waste my life fighting with software. It has power, but that power needs to be more easily harnessed ...
- ITSD: this is the software I went with, though it is clearly no longer being developed and it contains some really annoying bugs (such as reminding of tasks which have already been marked as complete). However, it's fairly stable and useful in its current form, so it will do until something better somes along :-)
- ListPro: marvellous. Simple, useful, mobile (on the Palm) - just what I needed for my lists (but not for my todos).

On the document management side:

- PaperPort and eDoc plus both give errors when reading/indexing some documents, but neither give the name of the document giving the problem so they tend to force the user away from them in terms of document management when one has as many documents to manage as I do. I continue to use PaperPort as a scanning manager.
- IdeaMason: just gives me the eebie jeebies each time I look at it, so if I can't bring myself to test it, I think I'd probably never get the most out of it if I were to use it for resource management!
- Whizfolders Deluxe: my main notation tool at the moment, but it's lack of power when it comes to things like inserting graphics are limiting to workflow.
- OneNote2007: listen up good people at Kinook: THIS is what you should be producing. And, given UR's base and its additional features, I don't see why it wouldn't be possible. OneNote is hardly more expensive than UR, it's simplicity itself in use and it hides great power. I'm no Microsoft basher, and this is a good reason why. Make no mistake - I do NOT like the Notebook/Sections/Tabs paradigm - tree organisers are common for a good reason! Microsoft seem to have realized this to some extent by allowing multiple notebooks/groups of sections in this version, but it still tends to hide information more than I'd like, and it smacks of digging themselves into the paradigm that they're not too keen on any more either. Getting data into it from existing documents is very time consuming, and I miss being able to set an automatic directory sync like one can in UR (though it has never worked properly in UR for me, and if it can't be trusted it's useless for me).

However, by keeping the contents of each tab as a separate file on the disk, OneNote doesn't get as sluggish as programs that store everything in a single database (such as UR and Surfulator). Printing a document to the notebook is an inspirational idea. Basically, though I'm a long way from giving up on Whizfolders, the power of what can be thrown into OneNote means I shall be using it more and more. I am almost through reading all my documents into OneNote, and the proof will then be how easy data is to find in it - but it's useful to see the contents of the document when searching. Oh yes, that reminds me: OneNote does slip up sometimes over importing documents, but closing and restarting it tends to resolve any problems.

So, there we are. My ideas. Doubtless by next month I'll be on to my next set of programs, but this is it for the moment ....

Graham

Graham, as usual for me, I am still sifting through possibilities -- CRIMPing as usual.

I agree with your assessment of UltraRecall. I feel it is a program that should be the program I most often use for info -- but I do not.

I find it is pesky at times. For example, the annoying file messages that pop up (recent discussion of these on the forum), and the inability for me to use Gmail fully within the embedded browser.

Also, it is has a very technical feel about it. That wouldn't necessarily be bad -- but the help files badly need to talk more about how to do things, and not just what the program will do.

Also, bcause of the way UR is set up, it very much needs to have many more examples of how it can actually be used.

OneNote -- you and others have me thinking about it.

For ease of web clipping and using that information, nothing comes close to Surfulater in my opinion.

Evernote, which I have and sometimes look at is a very distant secondor third, and IMHO is over-rated.

I use WhizFolders for writing because of its ability to have multiple windows open. For writing I like it.

I am primarily using MyInfo for a lot of my info management -- hoping that in its new version it will have a hoist capability, and ability to have multiple windows. That would knock WhizFolders and UR out of my life completely.

For detailed academic stuff where I want lots of my own keywords and the ability to finely select according to them, I use MDE InfoHandler. A beta version is out -- I wish it had a web capture capability such as Surfulater's, and the easy ability to have multiple windows open.

I am longing to see Zoot 32 -- potentially, depending what all the Admiral has been up to, Zoot could be the main program in its new form. I am hoping for that. It would make life easier.

I am sentimental. I remember the good old days with ADM and Eric, and a time when Eric was truly open to all the ideas his trusted testers supplied him with.

Now that it looks like ADM is a as dead as a door nail, and that Arne and Eric have about the same amount of credibility when it comes to announcing and following through with information and/or new versions, ADM is out of the mix.

I say sentimental because ADM's metadata ability, its multiple windows, and even its web capture (if Eric could have just tweaked it a bit) for me were great. ADM metadata development path was for a time very exciting, progressive, and full of creative energy. Then, I think the Asian market became too much of a temptation and the rest of us could go to hell in a handbasket.

I hope Arne and Eric can finally get their stuff together, but I won't use a program where a) there is no reason to believe it is even still being developed, and (b) where the developers treat their most loyal followers like cow cakes.

So it looks like Surfulater, MyInfo, InfoHandler, and Zoot.

Daly

Graham Rhind wrote:

I felt it was time to summarise some of my findings on this search and the related one for
a task manager in case they prove useful to the group members.


- UltraRecall: I have given up on UR except to keep a client
database in it. I think it suffers terribly because its developers are too technical
and don't really (in my mind) understand how many end-users, except technical ones,
would use such a product. It is unnecessarily complicated - it could be so much better -
and, although technical myself, I don't want to waste my life fighting with software.
It has power, but that power needs to be more easily harnessed ...
- ITSD: this is the
software I went with, though it is clearly no longer being developed and it contains
some really annoying bugs (such as reminding of tasks which have already been marked
as complete). However, it's fairly stable and useful in its current form, so it will do
until something better somes along :-)
- ListPro: marvellous. Simple, useful,
mobile (on the Palm) - just what I needed for my lists (but not for my todos).

........
- OneNote2007: listen up good people at Kinook:
THIS is what you should be producing. And, given UR's base and its additional
features, I don't see why it wouldn't be possible. OneNote is hardly more expensive
than UR, it's simplicity itself in use and it hides great power. I'm no Microsoft
basher, and this is a good reason why. Make no mistake - I do NOT like the
Notebook/Sections/Tabs paradigm - tree organisers are common for a good reason!
Microsoft seem to have realized this to some extent by allowing multiple
notebooks/groups of sections in this version, but it still tends to hide information
more than I'd like, and it smacks of digging themselves into the paradigm that they're
not too keen on any more either. Getting data into it from existing documents is very
time consuming, and I miss being able to set an automatic directory sync like one can in
UR (though it has never worked properly in UR for me, and if it can't be trusted it's
useless for me).

However, by keeping the contents of each tab as a separate file on
the disk, OneNote doesn't get as sluggish as programs that store everything in a
single database (such as UR and Surfulator). Printing a document to the notebook is an
inspirational idea. Basically, though I'm a long way from giving up on Whizfolders,
the power of what can be thrown into OneNote means I shall be using it more and more. I am
almost through reading all my documents into OneNote, and the proof will then be how
easy data is to find in it - but it's useful to see the contents of the document when
searching. Oh yes, that reminds me: OneNote does slip up sometimes over importing
documents, but closing and restarting it tends to resolve any problems.

So, there
we are. My ideas. Doubtless by next month I'll be on to my next set of programs, but this
is it for the moment ....

Graham

Daly,

I don't see MyBase on your application's review, did you give it a try? It was the PIM of my choice before moving to UR, ... very nice soft, but UR beats it cause of the attributes feature ...

I have used MyBase long ago -- but for some reason it is no appealing.

I like columns and metadata capability. Thus MyInfo, UR, or ADM (RIP).

Daly

quant wrote:

Daly,

I don't see MyBase on your application's review, did you give it a try? It was
the PIM of my choice before moving to UR, ... very nice soft, but UR beats it cause of the
attributes feature ...