This topic has been on my mind for the past few months as our office has been trying to increase utilization of MS Sharepoint for collaborative workspaces and storage of documents and communication. My apologies if this post is a bit long or seems slightly OT, but I think that many regulars here will have some thoughts on this issue. I work in a large municipal government that is runs on Microsoft Office products. Our email system pre-dated Outlook when it ran on Novel's GroupWise, so I have a fairly old archive of messages that has been migrated over the years. Messages are retained for legal and informational purposes, and were often considered a departmental responsibility. When a person left, their email account was to be preserved, and if their replacement was reasonably sharp, they might have asked for access to that account so they had some history to reference if need be. In short, the messages were stored with the person. If you wanted find out something, you had to know who might have worked on a project. Files were stored in a somewhat similar fashion, but at least MS File Manager offered shared folder prior to the existence of SharePoint.

Today, we still store email and person files by person, but given the more recent rate of turnover in the work environment, this has made less and less sense. And, with Teams being rolled out on top of SharePoint, it is quite easy to store files and chats by project rather than by individual persons (or in addition to individual persons). This makes a lot of sense for an organization since questions most often arise about specific projects rather than specific people. But, the issue I struggle with is how to easily store large number of email messages (with internal and external parties) with a project rather than a person's account. I have seen examples of where IT staff have dissembled messages into metadata and then tried to put them in some type of SP database that is searchable, but that seems like a Frankenstein approach that is not at all user friendly.

I realize that I am constrained by our departmental policies, but I am still curious as to how others have handled this issue. One thought I had was to ask if we could create email accounts for projects, but I suspect that will get shot down. Any thoughts or words of wisdom would be greatly appreciated.

Thanks,

--Ken

Ken wrote:

My apologies if
this post is a bit long or seems slightly OT, but I think that many
regulars here will have some thoughts on this issue.

///


...Excellent post, Ken.
Thank you for taking your time to prepare it.

... I view this forum as a source of tools for people who organize ideas.
Outliners certainly are one of the prime tools for that, but not the only tool.

When a person
left, their email account was to be preserved, and if their replacement
was reasonably sharp, they might have asked for access to that account
so they had some history to reference if need be. In short, the
messages were stored with the person.

... Have you considered a system of email aliases?

Jim@project-A.engineering.city.org
Mike@project-A.engineering.city.org
Betty@project-A.accounting.city.org

Jim@project-B.engineering.city.org
Alex@project-B.legal.city.org
Charlie@project-B.personnel.city.org
Bill@project-B.sanitation.city.org

Jim@engineering.city.org
Mike@engineering.city.org
Betty@accounting.city.org
Alex@legal.city.org
Charlie@personnel.city.org
Bill@sanitation.city.org


... No new software.
No programming.
No cost.
Negligible training time.
1/2 page handout of instructions.
And a heavy wood ruler for Ken to walk around and slap the knuckles of those who don't follow instructions.

@Ken, I have worked at implementing very large M365 (nee O365) installations in government -- I assume that is what your agency is running on. The short story is that the underlying infrastructure to do what you want to do is there. M365 has very robust tools and metadata that can be used to mark the data (emails, documents, messages, etc.) in multiple ways. You might want to look into, for example, the advanced e-discovery features of M365.

Specifically on the creation and management of project mailboxes -- that is a very common approach that is accomplished with group mailboxes. Many agencies settle on a naming convention for these mailboxes, and they need administrator(s), but you should be able to find plenty of use cases describing that in the public domain. We considered group mailboxes -- for projects, or deparments, or other non-personal uses -- as a way to avoid storing email with institutional importance in personal mailboxes that later get abandoned when the inevitable retirement or resignation occurs.

If you can get your technical staff involved with users in a conversation with your agency's Microsoft technical account team, you will probably be able to work toward a solution. One interesting thing is that a lot of the security and data administration underpinnings of O/M365, from a user perspective, were designed with a heavy hand from Microsoft's legal departments. If anyone knows the value and challenges of lost data, it is the legal team.

Larry_in_Bangkok wrote:

...Excellent post, Ken.
Thank you for taking your time to prepare it.

... I view this forum as a source of tools for people who organize ideas.
Outliners certainly are one of the prime tools for that, but not the only tool.




... Have you considered a system of email aliases?

Jim@project-A.engineering.city.org
Mike@project-A.engineering.city.org
Betty@project-A.accounting.city.org

Jim@project-B.engineering.city.org
Alex@project-B.legal.city.org
Charlie@project-B.personnel.city.org
Bill@project-B.sanitation.city.org

Jim@engineering.city.org
Mike@engineering.city.org
Betty@accounting.city.org
Alex@legal.city.org
Charlie@personnel.city.org
Bill@sanitation.city.org


... No new software.
No programming.
No cost.
Negligible training time.
1/2 page handout of instructions.
And a heavy wood ruler for Ken to walk around and slap the knuckles of
those who don't follow instructions.

Thank you for indulging this question. I would agree that this is a great place to tools to help us organize data, ideas and information. And it is a great archive for me as well since I can search back to many a thread.

I agree that alias mailboxes would be a great option, but I am far down in the pecking order, so I can suggest and encourage, but would probably need to carry around a box of chocolates instead of a stick!

--Ken

Paul Korm wrote:

@Ken, I have worked at implementing very large M365 (nee O365)
installations in government -- I assume that is what your agency is
running on. The short story is that the underlying infrastructure to do
what you want to do is there. M365 has very robust tools and metadata
that can be used to mark the data (emails, documents, messages, etc.) in
multiple ways. You might want to look into, for example, the advanced
e-discovery features of M365.

Specifically on the creation and management of project mailboxes -- that
is a very common approach that is accomplished with group mailboxes.
Many agencies settle on a naming convention for these mailboxes, and
they need administrator(s), but you should be able to find plenty of use
cases describing that in the public domain. We considered group
mailboxes -- for projects, or deparments, or other non-personal uses --
as a way to avoid storing email with institutional importance in
personal mailboxes that later get abandoned when the inevitable
retirement or resignation occurs.

If you can get your technical staff involved with users in a
conversation with your agency's Microsoft technical account team, you
will probably be able to work toward a solution. One interesting thing
is that a lot of the security and data administration underpinnings of
O/M365, from a user perspective, were designed with a heavy hand from
Microsoft's legal departments. If anyone knows the value and
challenges of lost data, it is the legal team.

Yes, you have described our setup pretty well. I do sit on a working group for technology implementation, so I do have some access to mid-level staff who can help me shape up a possible proposal. Ironically, we had originally outsourced our archive software when we moved to Outlook to meet our legal retention requirements, but that is a whole other story. So, I believe that MS is handling some of our current retention features. I just need to find a user-friendly way of getting project files to a common project repository, most likely in SP with a Teams front end that meets approval. It is not going to be an easy sell, but with some background research here, at least I will hopefully not be barking up the wrong tree.

Thanks,

--Ken

@Ken, when you meet with staff, keep in mind it is not a technical gap in the software your agency bought, it is about altering business rules and roles and perhaps asking administrators and managers to do something a bit different. The agency owns the capabilities -- but willingness to change how the technology is configured and administrated is always the challenge. Good luck!

Paul Korm wrote:

@Ken, when you meet with staff, keep in mind it is not a technical gap
in the software your agency bought, it is about altering business rules
and roles and perhaps asking administrators and managers to do something
a bit different. The agency owns the capabilities -- but willingness
to change how the technology is configured and administrated is always
the challenge. Good luck!

Yes, the wheels of IT change often move slowly even though the tools are there. After thinking about this a bit today, I do wonder why MS has not really offered a way to effectively save an email message with all of its data intact as a separate file? You can kind of save out a message, but the attempts I have seen do not look easy or user friendly. If I could just run a script to save out flagged messages as discrete files, then they could be easily uploaded like other files (and would hopefully be searchable as well) into a commonly accessed site or folder location.

--Ken

There are document management systems that will let you upload emails as well as stand-alone files to what is, in effect, a giant database. Some of them might have some kind of tagging system that would let you assign specific emails to specific projects. The tagging system would have to permit combinations of tags. One of the problems I've noticed with using email as an informal "data dump" is that often people write about multiple projects or topics in a single emails. That's why MS-Outlook's foldering system isn't very useful for email - the same email might belong in more than one folder, but it won't let you do that.

After thinking about this a bit today, I do wonder why MS has

not really offered a way to effectively save an email message with all
of its data intact as a separate file? You can kind of save out a
message, but the attempts I have seen do not look easy or user friendly.
If I could just run a script to save out flagged messages as discrete
files, then they could be easily uploaded like other files (and would
hopefully be searchable as well) into a commonly accessed site or folder
location.

--Ken

Not fully sure that I understood Ken correctly, but there is a very simple way to save email messages with all data intact, including attachments, formatted text etc.: just drag the message out of a normal email client (Outlook or TB) to the desktop and then rename it if desired; you can also save multiple emails this way. (I do this all the time, albeit in a home environment = for my academic research and the like, rather than a business environment.)

Leib Moscovitz wrote:

Not fully sure that I understood Ken correctly, but there is a very
simple way to save email messages with all data intact, including
attachments, formatted text etc.: just drag the message out of a normal
email client (Outlook or TB) to the desktop and then rename it if
desired; you can also save multiple emails this way. (I do this all the
time, albeit in a home environment = for my academic research and the
like, rather than a business environment.)

That's true, Leib. But you get a .msg file that you can only open successfully using Outlook (other e-mail clients either do not open the msg file at all or at least not correctly). You'd need a DMS or a database tool or a search tool (such as Lookeen) that can handle msg files.

It's a fascinating problem, especially for small businesses without dedicated IT departments.

I looked into this at great length back in the early 2000s, and concluded that Novell GroupWise would indeed be the best all-round solution for this kind of requirement (a fundamental business requirement, in fact). Nothing else came close. But implementing GroupWise would have cost us an arm and a leg, so we used a combination of solutions in the end, including a powerful piece of Australian groupware that is now, alas, defunct. Now, of course, GroupWise is also defunct.

I have been amazed by the failure of IT developers to get a really user-friendly handle on this. There are solutions around for small businesses, but most of them focus on specific aspects (e.g. bookkeeping/accounting, document management, e-mail backups) with very limited cross-functional support. Larger businesses must turn to expensive consultants, first, to sell them the systems (and good luck with finding an "independent" consultant for that!) and second, to configure them so they work properly. This, based on chats with colleagues involved in IT, is by no means guaranteed to work; external consultants rarely understand internal processes as well as they pretend to (and in any case, are liable to select off-the-shelf configurations just to make their own lives easier).

User-friendly solutions that can be implemented by an SME are very thin on the ground. I can think of two that have impressed me recently: MarketCircle's DayLite suite (https://www.marketcircle.com/ and the German solution Revolver (https://revolver.info/ – not available in English). Both of them, interestingly, are Mac-based, and in both cases, the pricing is pretty robust.

Of course there is a rapidly growing number of web apps that claim to be all-singing, all-dancing business management solutions – many of them fulfilling this promise by offering "integrations" with other (online) software. But many – I dare say most – SMEs are not delighted by the idea of running their entire business management operation through a third party, let alone multiple third parties. My Financial Director was appalled when it became apparent that to keep using QuickBooks, we were going to have to abandon the desktop app and use the (strikingly inferior) online version.

We've not found anything that suits us perfectly, which means we rely on shared group inboxes (something that IMAP servers are perfectly capable of, without any sophisticated additions – but you do have to trust everybody with access to a shared inbox) and a rigorously defined file system using set formulae for storing files; key project folders are synced using Datto Workspace (a very powerful Dropbox equivalent for businesses).

I haven't found a perfect way to integrate e-mails – project-critical ones are generally printed out to the relevant project folders as PDFs, but it's far from ideal. E-mails are automatically backed up to a centralised server, however, using MailStore (another outstanding German product with a fantastic search engine). This particular solution doesn't run on Mac (more details at https://www.mailstore.com/ for a good Mac-based backup solution, you could do worse than Mail Backup X (http://inventpure.com/ but it's not as good as MailStore.

Microsoft SharePoint I've experimented with, but like most MS "enterprise" products, it's not SME-friendly (despite many avowals to the contrary); to use the online version, you have to sell your soul to Microsoft. Fair enough, you will say, but it's not something I'm happy to do. I live in hope that a fully integrated SME-friendly solution will appear in the near future, and meanwhile keep an eye on DayLite, just in case they decide to moderate their pricing...

Cheers,
Bill

That's true, Leib. But you get a .msg file that you can only open
successfully using Outlook (other e-mail clients either do not open the
msg file at all or at least not correctly). You'd need a DMS or a
database tool or a search tool (such as Lookeen) that can handle msg
files.

Again, don't know whether this would help, but you might also use a regular file manager which allows for internal display of .msg or .eml files - both XYplorer and Directory Opus handle this well.

Ken wrote:

... I do wonder why MS has
not really offered a way to effectively save an email message with all
of its data intact as a separate file? You can kind of save out a
message, but the attempts I have seen do not look easy or user friendly.

We're stuck with all-Microsoft at work, too. My Outlook has a button in the "Move" section of the ribbon to copy a message to OneNote. You can share notebooks, so maybe that would be the way to go?

Sharepoint, at least for us, turned into a data swamp. The only way to effectively navigate it is to get a link from someone who knows where the good stuff is, then be religious about saving bookmarks.

MadaboutDana wrote:

for a good Mac-based backup solution, you could do worse than
Mail Backup X (http://inventpure.com/

The specs look good for it but I've been wary because it's always on sale somewhere, at differing prices (AppSumo and StackSocial right now, for $49 and $59) - and even on its own site it's $29 down form $79 but there is also an unknown discount available if you fill out a questionnaire on the site and wait for an email from them.

Also, when searching I never find reviews, just hits for sales.

Again, it looks good, but I've found that using EagleFiler ( https://c-command.com/eaglefiler/ $50, or $40 with coupon code if you're a Tidbits.com member) is an easy way to backup mail messages (.eml format) or entire mailboxes (.mbox format) - and can import from webmail or all major Mac mail apps - that can easily be searched.

https://c-command.com/eaglefiler/help/importing-mail

More generally, I also use it as a replacement for Devonthink.

Excellent overview, many thanks!

MadaboutDana wrote:

It's a fascinating problem, especially for small businesses without dedicated IT departments.

James Salla wrote:

One of the problems I've noticed with
using email as an informal "data dump" is that often people write about
multiple projects or topics in a single emails. That's why MS-Outlook's
foldering system isn't very useful for email - the same email might
belong in more than one folder, but it won't let you do that.

Agreed, and that is why I try to keep topics on subject in my messages. And while I agree that tagging is much more flexible, I would be happy with multiple copies if need be, but I am sitting very low on the totem pole and do not have much say as to how things get done that will effect thousands of my colleagues. Welcome to the bureaucracy.

--Ken

MadaboutDana wrote:

I looked into this at great length back in the early 2000s, and
concluded that Novell GroupWise would indeed be the best all-round
solution for this kind of requirement (a fundamental business
requirement, in fact). Nothing else came close. But implementing
GroupWise would have cost us an arm and a leg, so we used a combination
of solutions in the end, including a powerful piece of Australian
groupware that is now, alas, defunct. Now, of course, GroupWise is also
defunct.

We started out with Novell GroupWise as well. I probably still have files that were migrated from GW to Outlook. I am not sure which I prefer more as an end user, but I can say that I truly enjoyed using Ecco (for myself) to keep a personal calendar separate from GW. I am not even sure we initially used the calendar feature when we were using GW for mail.

--Ken

Larry Kollar wrote:

We're stuck with all-Microsoft at work, too. My Outlook has a button in
the "Move" section of the ribbon to copy a message to OneNote. You can
share notebooks, so maybe that would be the way to go?

Sharepoint, at least for us, turned into a data swamp. The only way to
effectively navigate it is to get a link from someone who knows where
the good stuff is, then be religious about saving bookmarks.

You can shout this from the mountaintops and the diehards would never believe you about SP becoming a data swamp. I have a OneNote page that has all of the invitations that I receive so at least they are in one place, although I believe that MS now shows you all of the files that have been shared with you in certain views. But, yes, if everybody has a SP site, and the project has a SP site, pretty soon it is impossible to find anything. My wife's nephew used to work for MS and told me that nobody he worked with used it because they switched teams so often that it did not make sense. Perhaps things have changed, but I wish MS would overhaul the UI for SP. It is a kludge at best IMHO, and it is the underpinning of Teams, so behind the scenes it is creating untold numbers of folds and paths.

--Ken

I use it as my regular backup system now, running on a Mac Mini; it’s very reliable and picks up e-mail at least as well as any of the (many) e-mail clients I’ve experimented with over the years.

Unlike MailStore, however, its built-in search engine is quite slow. You can export e-mails as PDF files, but if you’ve got a lot of e-mails (and yes, we do have many thousands), that takes a ridiculously long time. But the search function works fairly well, as long as you’re not in a tearing hurry.

I may have to go back to MailStore at some point – because let’s face it, if you’re searching through mail archives, then normally you ARE in a tearing hurry, and MailStore’s search function is extremely fast – but alas, that will mean running a dedicated Windows machine, which seems a bit ridiculous (although I do have a couple of those tiny Lenovo micro-PCs sitting around not doing a lot).

Effective e-mail management is another business niche just waiting to be filled…




satis wrote:

MadaboutDana wrote:

> for a good Mac-based backup solution, you could do worse than
> Mail Backup X (http://inventpure.com/

The specs look good for it but I've been wary because it's always on
sale somewhere, at differing prices (AppSumo and StackSocial right now,
for $49 and $59) - and even on its own site it's $29 down form $79 but
there is also an unknown discount available if you fill out a
questionnaire on the site and wait for an email from them.

Also, when searching I never find reviews, just hits for sales.

Again, it looks good, but I've found that using EagleFiler (
https://c-command.com/eaglefiler/ $50, or $40 with coupon code if you're
a Tidbits.com member) is an easy way to backup mail messages (.eml
format) or entire mailboxes (.mbox format) - and can import from webmail
or all major Mac mail apps - that can easily be searched.

https://c-command.com/eaglefiler/help/importing-mail

More generally, I also use it as a replacement for Devonthink.

Ha, yes, everything you say is spot on. We experimented with SharePoint a few years ago, and after desperately trying to control the efflorescence, I finally gave up and sacked it.

We even had a special extranet running on SharePoint for one of our major clients at the time (part of Sony); it had lots of useful info on it, including archives of advertising/marketing material and all sorts. Did anybody ever use it? Nope. Members of the client’s marketing department would visit it, admire it… and then leave it alone.

There’s something basically confusing about SharePoint; it’s what I would call a “fuzzy” app, something that appears to be friendly but isn’t really. Craft, a wonderful, gorgeous app for macOS/iOS, is similar – the structures are fuzzy, making it quite difficult to organise things in logical ways. But at least Craft has a really great search function (now that they’ve upgraded it), whereas the SharePoint search function almost feels bolted on as an afterthought. Or it did – I must confess I haven’t used SP in anger for several years.

And I shouldn’t really criticise Craft; it’s brand new, and its developers produce new, improved features at a stonking pace, whereas SP? Just seems to get fuzzier and fuzzier…

Cheers!
Bill

Ken wrote:

Larry Kollar wrote:


>
>We're stuck with all-Microsoft at work, too. My Outlook has a button in
>the "Move" section of the ribbon to copy a message to OneNote. You can
>share notebooks, so maybe that would be the way to go?
>
>Sharepoint, at least for us, turned into a data swamp. The only way to
>effectively navigate it is to get a link from someone who knows where
>the good stuff is, then be religious about saving bookmarks.

You can shout this from the mountaintops and the diehards would never
believe you about SP becoming a data swamp. I have a OneNote page that
has all of the invitations that I receive so at least they are in one
place, although I believe that MS now shows you all of the files that
have been shared with you in certain views. But, yes, if everybody has
a SP site, and the project has a SP site, pretty soon it is impossible
to find anything. My wife's nephew used to work for MS and told me that
nobody he worked with used it because they switched teams so often that
it did not make sense. Perhaps things have changed, but I wish MS would
overhaul the UI for SP. It is a kludge at best IMHO, and it is the
underpinning of Teams, so behind the scenes it is creating untold
numbers of folds and paths.

--Ken

MadaboutDana wrote:

Ha, yes, everything you say is spot on. We experimented with SharePoint
a few years ago, and after desperately trying to control the
efflorescence, I finally gave up and sacked it.

We even had a special extranet running on SharePoint for one of our
major clients at the time (part of Sony); it had lots of useful info on
it, including archives of advertising/marketing material and all sorts.
Did anybody ever use it? Nope. Members of the client’s marketing
department would visit it, admire it… and then leave it alone.

There’s something basically confusing about SharePoint; it’s
what I would call a “fuzzy” app, something that appears to
be friendly but isn’t really. Craft, a wonderful, gorgeous app for
macOS/iOS, is similar – the structures are fuzzy, making it
quite difficult to organise things in logical ways. But at least Craft
has a really great search function (now that they’ve upgraded it),
whereas the SharePoint search function almost feels bolted on as an
afterthought. Or it did – I must confess I haven’t used SP
in anger for several years.

And I shouldn’t really criticise Craft; it’s brand new, and
its developers produce new, improved features at a stonking pace,
whereas SP? Just seems to get fuzzier and fuzzier…

Cheers!
Bill

Don't even get me started on SP. Fuzzy would be a kind word for it if I was to describe it at length. I would say that having Teams as a "front end" to it helps a bit, but it still is a kludge as far as friendliness and UI.

--Ken

This exchange caused me to go back and take another look at a piece of software I experimented with many years ago: EssentialPIM (available for Windows, Android, iOS), a very capable three-pane outliner (on Windows; on Android and iOS it’s more like a one-pane/two-pane outliner).

I was pleased to see (a) that it’s still up and running, and (b) that it appears to be making steady progress.

It’s an alternative to the Microsoft all-in-one approach originally envisioned in Outlook. Of course Microsoft’s solution has now been profoundly compromised (especially on Mac) as they appear to be trying to drive users into using their online versions of the software – with typically Microsoft-fragmented solutions and results. Enraging!

But EssentialPIM still bundles everything together: task management, contacts, rich-text notes, passwords – and yes, e-mail as well. It’s a remarkably comprehensive package, and can also be run by a small business using a Firebird server (running on Linux, macOS or Windows: Firebird is open-source and cross-platform). This allows you to share most of the data with colleagues across your LAN. EssentialPIM supply a user-friendly version that’s very easy to install and set up.

It’s much cheaper than comparable packages, and uses an excellent rich-text editor for notes across all its modules.

I wouldn’t say it’s suitable for use in big-business environments, but certainly a very reasonable alternative for SMEs. More info on essentialpim.com (for a comparison of Pro with Free editions – yes, there is a free edition – see https://www.essentialpim.com/pc-version/pro-vs-free

There is even an online offering (EPIM Cloud), although I’ve not tried it.

Cheers,
Bill

Coincidentally Bill, I got an email post today from Essential PIM announcing release of version 10. It looks good.

https://www.essentialpim.com/blog-and-news

I do remember Essential PIM and am glad to see it has survived through the years.

--Ken