In this somewhat longer (holiday) blog, we’ll explore the limitations and possibilities of text analysis technology and show how text analysis becomes an essential tool to help process and analyze today’s enormous amounts of enterprise information for various critical business applications in a timely fashion.

An in-depth white paper with more detailed information and several supporting graphics, can be downloaded from here: http://www.zylab.com/Resources/WhitePapers.aspx in the educational white paper on Text Analysis.  

Finding Without Knowing Exactly What to Look For

 In general, text analysis refers to the process of extracting interesting and non-trivial information and knowledge from unstructured text. Text analysis differs from traditional search in that, whereas search requires a user to know what he or she is looking for, text analysis attempts to discover information in a pattern that is not known beforehand (through the use of advanced techniques such as pattern recognition, natural language processing, machine learning and so on). By focusing on patterns and characteristics, text analysis can produce better search results and deeper data analysis, thereby providing quick retrieval of information that otherwise would remain hidden.

Text analysis is particularly interesting in areas where users must discover new information, such as, in criminal investigations, legal discovery and when performing due diligence investigations. Such investigations require 100% recall; i.e., users can not afford to miss any relevant information. In contrast, a user who uses a standard search engine to search the internet for background information simply requires any information as long as it is reliable. During due diligence, a lawyer certainly wants to find all possible liabilities and is not interested in finding only the obvious ones.

Beyond the Google Standard

With web search engines like Google, most companies and organizations place a premium on being found as close to the top of search list as possible. Experienced users have become quite savvy in utilizing search engine optimization techniques to enhance high rankings.

Now, an entire generation of tech-savvy computer users exist whose expectations and perceptions of full-text search functionality and performance are almost completely influenced by the “Google effect.” In most instances, this type of approach works fine if users only need to find the most appropriate website for answering general questions. Users type in full-text keywords and expect to see the most relevant document or website appear at the top of a result list. Page-link and similar popularity- based algorithms work very well in this context.

However, a lot of information that may be vital for them to know may not come to light using only these basic search techniques. If, for example, a user’s search is related to fraud and security investigations, (business) intelligence, or legal or patent issues, other searching techniques are needed that support different sets of issues and requirements, such as the following:

• Focusing on optimized relevance: the first requirement of broader search applications is that not only does the best document need to be found, but all potentially relevant documents need to be located and sorted in a logical order, based on the investigator’s strategic needs. 
• Handling massive data collections: another issue impacting effective strategic searching is how to conduct extensive searches among extremely large data collections. For example, if email collections need to be investigated, these repositories are no longer gigabytes in size; rather, they can be a terabyte or more. When handling this volume of data, plain full-text search simply cannot effectively support finding, analyzing, reviewing and organizing all potentially relevant documents. 
• Finding information based on words not located in the document. In this context, consider investigators who may have some piece of information concerning an investigation but don’t necessary know other details they may be looking for. Who is associated with a suspect? What organizations are involved? What aliases are associated with bank accounts, addresses, phone records or financial transactions? Traditional precision-focused, full-text approaches are not going to help users find hidden or obscure information in these contexts. 
• Defining relevancy:when defining a search’s relevance, all factors that could be in play during a specific search instance must be accounted for (in the context of overall goals). Using the investigative example again, consider possible involved parties and what “relevance” would mean to their actual search:
  • Investigators want to comb documents to find key facts or associations (the “smoking gun”);
  • Lawyers need to find privileged or responsive documents;
  • Patent lawyers need to search for related patents or prior art;
  • Business intelligence professionals want to find trends and analyses; and
  • Historians need to find and analyze precedents and peer-reviewed data.

All of these instances require not only sophisticated search capabilities but also different context-specific functionalities for sorting, organizing, categorizing, classifying, grouping and otherwise structuring data based on additional meta-information, including document key fields, document properties and other context-specific meta-information. Utilizing this additional information will require a whole spectrum of additional search techniques, such as clustering, visualization, advanced (semantic) relevance ranking, automatic document grouping and categorization.

Challenges Facing Text Analysis

Due to the global reach of many investigations, a lot of interest also exists with text analysis in multi-language collections. Multi-language text analysis is much more complex than it appears because, in addition to differences in character sets and words, text analysis makes intensive use of statistics as well as the linguistic properties (such as conjugation, grammar, tenses or meanings) of a language. A number of multi-language issues will be addressed later in this article.

But perhaps the biggest challenge with text analysis is that increasing recall can compromise precision, meaning that users end up having to browse large collections of documents to verify their relevance. Standard approaches to countering decreasing precision rely on language-based technology, but when text collections are not in one language, are not domain-specific and/or contain documents of variable sizes and types, these approaches often fail or are too sophisticated for users to comprehend what processes are actually taking place, thereby diminishing their control.

Furthermore, according to Moore’s Law, computer processor and storage capacities double every 18 months, which, in the modern context, also means that the amount of information stored will double during this timeframe as well. The continual, exponential growth of information means most people and organizations are always battling with the specter of information overload. Although effective and thorough information retrieval is a real challenge, the development of new computing techniques to help control this mountain of information is advancing quickly as well. Text analysis is at the forefront of these new techniques, but it needs to be used correctly and understood according to the particular context in which it’s applied. For example, in an international environment, a suitable text analysis solution may consist of a combination of standard relevance- ranking with adaptive filtering and interactive visualization, which is based on utilizing features (i.e. metadata elements) that have been extracted earlier.

Control of Unstructured Information

More than 90% of all information is unstructured, and the absolute amount of stored unstructured information increases daily. Searching within this information, or performing analysis using database or data mining techniques, is not possible, as these techniques work only on structured information. The situation is further complicated by the diversity of stored information: scanned documents, email and multimedia files (speech, video and photos).

Text analysis neutralizes these concerns through the use of various mathematical, statistical, linguistic and pattern-recognition techniques that allow automatic analysis of unstructured information as well as the extraction of high quality and relevant data. (“High quality” here refers to the combination of relevance [i.e. finding a needle in a haystack] and the acquiring of new and interesting insights.) With text analysis, instead of searching for words, we can search for linguistic word patterns which enable a much higher level of search.

Different Levels of Semantic Information Extraction

 Several options exist for extraction and text analysis within its products. These options vary from simple extraction methods such as file and document property extraction to more advanced text analysis options:

• File system extraction: extraction of file properties such as file name, file size, modified date, creation date, attributes, mime type, etc.
• Document property extraction: extraction of specific document properties depending on the document format such as Title, Author, Publisher, Version, etc.
• Email property extraction: extraction of common email properties such as Sender, Recipient, Sent Date, Subject, Conversation topic and other properties such as Internet Headers, Original Sender, etc.
• Microsoft SharePoint property extraction: extractions of all Microsoft SharePoint document properties as these are stored in SharePoint with the document including security settings.
• Hash calculation: calculation of hash values for identification purposes, supporting several hash types such as MD5 and SHA1.
• Duplicate detection: calculating hash values based on the content for email messages or binaries for other file types to find and detect duplicate documents.
• Language detection: detection of document language, support for over 400 languages.
• Concept extraction: extraction of predefined (full-text) queries that identify document and meta information content with specific combinations of keywords or (fuzzy and wildcard) word patterns in.
• Entity Extraction: extraction of basic entities that can be found in a text such as: people, companies, locations, products, countries, and cities.
• Fact Extraction: these are relationships between entities, for example, a contractual relationship between a company and a person.
• Attributes extraction: extraction of the properties of the found entities, such as function title, a person’s age and social security number, addresses of locations, quantity of products, car registration numbers, and the type of organisation.
• Events extraction: these are interesting events or activities that involve entities, such as: “one person speaks to another person”, “a person travels to a location”, and “a company transfers money to another company”.
• Sentiment detection: finding documents that express a sentiment and determine the polarization and importance of the sentiment expressed.
• Extended natural language processing:  Part-of-Speech (POS) tagging for pronoun, co-reference and anaphora resolution, semantic normalization, grouping, entity boundary and co-occurrence resolution.

Other examples of the application of text analytics and sentiment mining can be found in the examples below which are related to the FCPA and the UK Bribery Act: in the future it will be more and more important to find potential violations of these acts to prevent expensive investigations, consequential serial litigation and loss of reputation and business!

Co-reference and Anaphora Resolution

 One of the biggest problems in the discovery and identification of events is the resolving of the so called anaphora and co-references. This is the linguistic problem to associate pairs of linguistic expressions that refer to the same entities in the real world.

Consider the following text:

“A man walks to the station and try to catch the train. His name is John Doe. Later he meets his colleague, who has just bought a card for the same train. They work together at the Rail Company as technical employees and they are going to a meeting with colleagues in Washington DC.”

The text contains are various references and co-references. Various anaphora and co-references will have to be disambiguated before it is possible to fully understand and extract the more complex patterns of events. The following list shows some examples of these (mutual) references:

• Pronominal Anaphora: he, she, we, oneself, etc.
• Proper Name Co-reference: For example, multiple references to the same name.
• Apposition: the additional information given to an entity, such as “John Doe, the father of Peter Doe”.
• Predicate Nominative: the additional description given to an entity, for example “John Doe, who is the chairman of the football club”.
• Identical Sets: A number of reference sets referring to equivalent entities, such as “Ajax”, “the best soccer team”, and the “group of players” which all refer to the same group of people.

With advanced computational linguistics, one can resolve co-references, pronouns and other anaphora and easily find 2 to 4 times more relevant patterns which dramatically improves the quality of these types of analyses. So, for real in-depth insights, one cannot ignore pronoun and co-reference resolution! 

Faceted Search and Information Visualization

Text analysis is often mentioned in the same sentence with faceted search and information visualization in large part because visualization is one of the viable technical tools for information analysis after unstructured information has been structured. Extracted facts, entities, and events from data and can be presented in advanced data visualization tools such as a “treemap,, start tree or other powerfull visual analytical tools. Colored-coding, zoom, sizeshow interrelationships and content volume quickly (see the white paper for the graphical examples. These types of visualization techniques are ideal for allowing an easy insight into large email collections. Alongside the structure that text analysis techniques can deliver, use can also be derived from the available attributes such as “sender,” “recipient,” “subject,” “date,” etc.

Faceted search, also called faceted navigation or faceted browsing, is a technique for accessing a collection of information represented using a faceted classification, allowing users to explore by filtering available information. A faceted classification system allows the assignment of multiple classifications to an object, enabling the classifications to be ordered in multiple ways, rather than in a single, pre-determined, taxonomic order. Each facet typically corresponds to the possible values of a property common to a set of digital objects.

Facets are often derived by analysis of the text of an item using entity extraction techniques or from pre-existing fields in the database such as author, descriptor, language, and format. This approach permits existing web-pages, product descriptions or articles to have this extra metadata extracted and presented as a navigation facet.

Exploratory search is a specialization of information exploration which represents the activities carried out by searchers who are either:

• unfamiliar with the domain of their goal (i.e. need to learn about the topic in order to understand how to achieve their goal)
• unsure about the ways to achieve their goals (either the technology or the process) or even unsure about their goals in the first place.

Consequently, exploratory search covers a broader class of activities than typical information retrieval, such as investigating, evaluating, comparing, and synthesizing, where new information is sought in a defined conceptual area; exploratory data analysis is another example of an information exploration activity. Typically, therefore, such users generally combine querying and browsing strategies to foster learning and investigation.

Text Analysis on Non-English Documents

As mentioned earlier, many language dependencies need to be addressed when text- analysis technology is applied to non-English languages.

First, basic low-level character encoding differences can have a huge impact on the general search ability of data: where English is often represented in basic ASCII, ANSI or UTF-8, foreign languages can use a variety of different code-pages and UNICODE (UTF-16), all of which map characters differently. Before a particular language’s archive can be full-text indexed and processed, a 100% matching character mapping process must be performed. Because this process may change from file to file, and may also be different for different electronic file formats, this exercise can be significant and labor intensive. In fact, words that contain such language-specific special characters as ñ, Æ, ç, or ß (and there are hundreds more of such characters) will not be recognized at all.

Next, the language needs to be recognized and the files need to be tagged with the proper language identifications. For electronic files that contain text that is derived from an optical character recognition (OCR) process or for data that needs to be OCRed, this process can be extra complicated.

Straightforward text-analysis applications use regular expressions, dictionaries (of entities) or simple statistics (often Bayesian or hidden Markov models) that all depend heavily on knowledge of the underlying language. For instance, many regular expressions use US phone number or US postal address conventions, and these structures will not work in other countries or in other languages. Also, regular expressions used by text analysis software often presume words that start with capitals to be named entities, which is not the case with German. Another example is the fact that in languages such as German and Dutch, words can be concatenated to new words, which is never anticipated by English text analysis tools. More examples of linguistic structures exist that cannot be handled by many US-developed text analysis tools.

In order to recognize the start and end of named entities and to resolve anaphora and co-references, more advanced text analysis approaches tag words in sentences with “part-of- speech” techniques. These natural language processing techniques depend completely on lexicons and on morphological, statistical and grammatical knowledge of the underlying language. Without extensive knowledge of a particular language, none of the developed text analysis tools will work at all.

A few text analysis and text-analytics solutions exist that provide real coverage for languages other than English. Due to large investments by the US government, languages such as Arabic, Farsi, Urdu, Somali, Chinese and Russian are often well covered, but German, Spanish, French, Dutch and Scandinavian languages are almost always not fully supported. These limitations need to be taken into account when applying text analysis technology in international cases.

Content Analytics on Multimedia files: Audio Search on Sound and Video Files

Written text, such as transcripts from audio recordings, cannot fully convey intent, nuance or emotions which are only discernable by human listeners. Additionally, speech-to-text technology is generally limited to dictionary entries.

In contrast, state-of-the-art Audio Search technology transforms audio recordings into a phonetic representation of the way in which words are pronounced so that investigators can search for dictionary terms, but also proper names, company names, or brands without the need to “re-ingest” the data. 

With Audio Search investigators can quickly identify relevant audio clips from multimedia files and from ubiquitous business tools such as fixed-line telephone, VOIP, mobile, and specialist platforms like Skype or MSN Live.  The intuitive software enables technical and non-technical users involved in legal disputes, forensics, law enforcement, and lawful data interception to search, review and analyze audio data with the same ease as more traditional forms of Electronically Stored Information (ESI).

A Prosperous Future for Text Analysis

 Even with some of the limitations and challenges profiled here, we already see the extensive application of data mining in two areas: e-discovery and compliance. Associated with these are the cognate areas of bankruptcy settlements, due-diligence processes and the handling of data rooms during a takeover or a merger.

The final application in this context will unfold as major legislative changes and stricter control systems will undoubtedly take place in the short term: companies will have to carry out regular (real time) internal preventative investigations, deeper audits and risk analyses. Text analysis technology will become an essential tool to help process and analyze the enormous amount of information in a timely fashion.

Although changes in the legal and financial world are typically evolutions rather than revolutions, a significant role for text analysis in e-discovery and e-disclosure certainly exists. Data collections are just getting too large to be reviewed sequentially, and collections need to be pre-organized and pre-analyzed. With text analysis, reviews can be implemented more efficiently and deadlines can be made easier. The challenge will be to convince courts and auditors of the correctness of these new tools.

Examples of applications are automatic redaction, machine assisted document review, data monitoring, legacy information clean-up, enterprise information archiving and other future legal, governance and investigative power application.

The following observations stem from that session and the others I had an opportunity to attend.

1. European Privacy and Data Protection Regulations: Effective Defense Tools in eDiscovery

European companies are often at a disadvantage when they are up against a US company or regulator in a civil, regulatory or criminal investigation which involves a large eDiscovery. Failing to follow the US rules (the Federal Rules of Civil Procedure –FRCP) and best practices, as set by The Sedona Conference, EDRM and other court opinions, will undoubtedly lead to sanctions, fines, penalties and sometimes even a default decision. US parties may leverage this as a strategy to force European companies into unfavorable settlements. As a result, European companies often engage US law firms to process all data, often leading to violations of European privacy and data protection regulations, but also to huge costs and future risks, because you never know where your data will end up in US courts and in the hands of a hostile opposing party. European privacy and data protection regulations in combination with international treaties can help European companies as a great defensive strategy to:

a. Use technology to find, isolate and produce only documents that do not contain any data which violates privacy or data protection acts.
b. Process and review data in Europe to avoid cross border issues.
c. Use machine assisted review technology to reduce the amount of data that human eyes must review.
d. Use random sampling for legal defensibility of your machine-assisted, automated processes.
e. Maintain control over your data, and thus over cost and risk control.
f. Implement true early case assessment in Europe; Find what really matters and use this to negotiate a more favorable settlement on your terms. Understand the impact of search terms before agreeing to them.
g. Avoid penalties from European regulators for violation of European privacy and data protection acts.
h. Produce and disclose less information in the US.
i. And as a result have lower costs and less risk for court sanctions and penalties.

2. European Privacy and Data Protection Regulations: Also a Compliance Risk!

Many companies have terabytes of legacy information. In almost all cases, the privacy or data protection officer does not know all details of what resides within all of this data. Every day, we read news stories pertaining to data leakage and other violations of privacy and data protection acts. Given the seriousness of this problem, regulators continue to increase fines and penalties. In addition, reputations and revenue streams can be negatively impacted. As a result, and even more than for other compliance officers, it is very important for privacy and data protection officers to understand what is in the legacy data and to handle compliance issues before something goes wrong. Automation by using technology can help to quickly identify information, understand what is there and find documents that contain sensitive information such as names, addresses, credit card numbers, social security numbers, bank accounts, medical information, etc. Pattern based intelligent redaction, data transfer or data retention are then among the possible options.
It became clear during these wonderful sessions that if one should run into trouble in this area, it will be absolutely essential to engage a high-quality lawyer with extensive experience and knowledge in this area. The field is moving very rapidly and there are many different rules and regulations in Europe. This, in combination with new regulations and judgments continues to increase the importance of engaging top-quality lawyers who understand the intricacies–especially when the opposing party is not as skilled and knowledgeable as your lawyer!
In summary, it was a great conference and I’ll definitely go back next year.

• Gartner analyst Debra Logan provided an excellent keynote on trends and developments in eDiscovery and a separate session on Intelligent Information Governance.
• Daan Lunsingh Scheurleer, partner with NautaDutilh, led a session on eDiscovery, Compliance and Litigation Readiness.
• Gonzalo de Cesare of the European Union and formerly IM specialist at the UN war crimes tribunals in Cambodia, Yugoslavia and Rwanda discussed the deployment of ZyLAB technology in several tribunals and new rules of law worldwide.
• Barry Derksen, research director of the Business & IT Trends Institute, presented on the changing role of IT and how IT can best collaborate with the business and legal professionals in their organizations.

Debra Logan undoubtedly convinced the audience that eDiscovery and litigation are not ‘American problems’, that social media is of great concern in eDiscovery, that SaaS and Cloud solutions should be considered when regulators and lawyers are involved, that Legal and IT have to work together to solve eDiscovery problems, and that it is critical for organizations to be proactive about e-Disclosure.

In her second presentation: “Intelligent Information Governance”, Debra Logan helped the audience clearly understand information governance and enterprise information management, in particular. In order for companies to manage their information assets properly, they need to clearly understand enterprise information management. In this session the differences between EIM and related terms were outlined and an organizational model was presented to identify new roles for managing enterprise information effectively.

Gonzalo de Cesare shared his presentation on “Enabling Prosecution of the Unspeakable”, where he discussed the challenges faced by the UN Information Management team in the Office of the Prosecutor of the war crimes tribubals, which were exceptional in scale and often gruesome in content. His experience shed light on more day-to-day challenges faced by many businesses and public organizations today.

Daan Lunsingh Scheurleer explained how his company has setup a firm’s Class Action Team, providing immediate legal assistance to organizations that are involved in regulatory investigations and defending class actions relating to securities, financial services, prospectus liability and financial regulation. His experience in various collective settlements, both on an opt-in basis and on an opt-out basis under the Collective Settlement of Mass Claims Act, including Dexia, Shell Reserves,and  Converium , provided invaluable insight into the need for compliance and litigation readiness programs. He also clarified the unique attributes of litigation matters in the Netherlands compared to other jurisdictions, such as US-based litigation and how to work closely with US counsel.

Last but not least, Barry Derksen  shared the insight garnered from his extensive background working with KPMG Information & Risk Management  as an advisor, auditor, and interim manager for various governmental and commercial organizations. Specifically, Barry explained how IT can and must collaborate effectively with business and legal departments in order to make a company more competitive and to reduce costs and risk. IT should operate by proactively aligning legal, business and IT interests, they should cross the chasm, and they should initiate effective communication, understand the business, see how they can add value, create internal and external partnerships, and assist with implementing governance and compliance programs. As a result, they will become better aligned with legal and subsequently create better value and return on investment for the organization.

The esteemed faculty provided a wonderful curriculum, and the active participation from all attendees helped to make it a magnificent conference all around. The collective group of IT, legal, investigative, compliance, and information management professionals openly discussed challenges and potential solutions to the hurdles associated with eDiscovery, compliance, law enforcement and enterprise information management.

At the end of the day, ZyLAB’s VP of Sales and Marketing for EMEA/APAC, Victor Cohen, provided an excellent wrap up and conclusion of the day:

We all have to work together because:

1. eDiscovery and e-Disclosure are not challenges that are unique to the US
2. New regulations are created every day, and regulators are getting more aggressive
3. Information continues to grow
4. Social Media and working in the cloud bring new challenges
5. Legal & IT alignment yields litigation readiness, increased performance, and higher ROI.

I would like to thank all presenters and participants for contributing to this wonderful event. ZyLAB Universe 2012 demonstrated how effective collaboration among IT, legal, compliance and investigators benefits everyone.

E-mail is where the high costs and risks of e-discovery are concentrated. People keep their e-mails because it is easy, but these e-mail archives (PSTs) rapidly swell to GBs of information. Problems fester because the information in these PST folders is often completely unstructured. For example, potentially sensitive HRM-related e-mails (such as performance reviews or confidential financial or medical information) are frequently in the same collection (i.e. Sent Mail) as other, unrelated messages. This common situation is problematic on two fronts: non-relevant e-mails are kept, and confidential e-mails that can be classified as “privileged” in a legal discovery are not stored in separate folders.

Exchange server mailboxes and PST repositories are not designed for, and should not be used as, document archives. All relevant e-mails and documents must be archived in assigned repositories. Some tips to consider:

• Implement an appropriate e-mail archiving tool
• Set an automatic deletion date for all messages, calendar items, journals, and tasks older than 90 days that still reside on your MS Exchange server in personal, shared, or functional mailboxes and in central repositories (public folders and the list server). This wholesale deletion will occur every three months.
• Old e-mail repositories (PST and server-based mailboxes) also need to be sorted out and cleaned up before a set date. Choose a group to help support this activity. Consider using the same group that works on electronic discovery projects because performing clean-up activities provides a good training environment for e-discovery team members.

The e-mail archiving method can proceed as follows:

• Create a copy of the filing plan in every user’s mailbox. Users can then drag and drop relevant e-mails into these folders and create subfolders where needed.
• Make sure that software is in place that provides an option to automatically archive Sent messages to a designated location on a regular pre-defined basis.

Collecting from and cleaning file shares

Collecting from file shares is not as hard as it may seem, as long as the right software is in place. With many of these tools retention policies can be executed and early case assessment can be implemented. It is important to be sure one can full-text index all data (also incrementally) and to be sure that whatever data manipulation action one performs is audited.

Cleaning up MS-SharePoint repositories

More difficult than the old unstructured file servers, is MS-SharePoint that has replacereplaced these many traditional file shares in several many organizations. Nowadays we are creating large unstructured data collections in MS SharePoint, which is harder to access than the old file shares.

In case of an e-Discovery, SharePoint presents significant challenges for IT departments. When using MS-SharePoint organizations need to ensure they can:

• Archive projects and documents based on various policies (closed, size, age, people involved, set retention or expiration date, activity) into a open sustainable file format (such as XML and native files).
• Do this with or without stubbing (replacing an object with a pointer to another low-cost storage location, so less expansive memory is occupied on the MS-SharePoint server).
• Implement real-time archiving of files and projects.
• Optionally, include all (hidden) meta-information in your archiving.

 Allow Federated search to your archives from within MS-SharePoint.

Audio records management

The nature of data changes from textual data to multimedia visual and audio data. Audio data exists on traditional fixed-line phone systems, VOIP, mobile and specialist platforms like Skype or MSN Live.

But sound, pictures, phone, video and other multimedia information cannot be searched easily, if at all. Strong audio-search solutions are needed. In order to combat market abuse, insider dealing and market manipulation, the Federal Security Agency now requires organizations that handle client orders to record and maintain records of transactions conducted over telephone lines. These records must be “readily accessible” should the relevant authorities require them. FRCP regulations in the U.S. now allow “sound recordings” to be considered for inclusion in the list of discoverable items that may be requested as part of case preparation and evidence gathering. The wider implications of Sarbanes-Oxley and SEC regulations also influence the frequency with which audio files are called upon as a source of evidence.

Archiving and cleaning other databases

Within an organization, there are also many repositories containing structured information such as financial records, logistical transactions, CRM, HRM, ERP, production and other important information. Companies have to include these repositories in their data map, as this data also needs to be managed as part of the overall filing plan.  Since most such systems have proprietary, the best approach is often to archive relevant data in an open format such as XML, or to use specialized software to collect information from these repositories to assist records managers with the identification, transfer and retention of such information.

Archiving and cleaning from the cloud and remote storages

The location of data moves from being within the firewall to being everywhere and nowhere; on home computers, mobile devices, cell phones and of course in the cloud. Companies need to have well defined service level agreements with their cloud, SaaS or outsourcing partners to make sure that they have access to their corporate data when they need it and that it is actually destroyed or transferred when required.

Organization should update their data retention policy to include:

• SharePoint, blogs, social media
• Unified messaging, voice files, Video
• ADP and other financial service providers
• Salesforce and other CRM systems
• FexEx, UPS and other shipping
• BaseCamp, Google docs and other collaboration tools

It is also very important to know which protocols the provider has in place for collection in terms of speed and quality. What can be expected from the cloud provider? What to include into the Service Level agreements (SLA’s)?

e-Discovery and enterprise information management technology puts you in command of boundless enterprise data in order to mitigate risk, reduce costs, investigate matters and elicit business productivity and intelligence.

The convergence of information management and eDiscovery can help you to manage your content assets (and liabilities) and at the same time, cost-effectively mine them when an investigation ensues or when you really wish to share your corporate knowledge.

Records retention always carries with it some type of legal obligation, regardless of the type or size of an organization. The de facto standard for how records-retention requirements should fit together in a comprehensive records management environment are based upon such internationally acknowledged frameworks and guidelines such as the National Archives (UK), National Archives of Australia, US Dept of Defense (DoD) 5015.2-STD, PRO-TNA, ISO 15489, MoReq2, and so on. Even though these sorts of standards are highly detailed and most applicable to large, bureaucratic and multi-tiered organizations, they do, however, provide the necessary thematic guidelines for implementation across any size or type of organization, as well as outline the required legal framework affecting retention obligations. The above standards, as examples, provide a clear structure for organizing file plans and streamlining the records management policies that can fit the needs of most organizations.

Though high-level records management principles are going to be similar across the globe, the type of specific policies a particular organization puts in place will depend heavily on the business sector(s) and geographic regions in which they are active, as well as their own actual levels of commitment (by both management and staff), allocated resources and perceived legal risk.

Mitigating legal risk

Certainly, many instances still exist in which some organizations think about retention in terms of a very narrow “documentation management” model (i.e. storage focused rather than process focused), which falls short of meeting regulatory or performance expectations. Organizations can even be fully aware of the benefits of good records management but feel that setting up a formal and committed records management structure is a hassle and that the benefits don’t outweigh the costs or effort involved. These types of organizations are the ones that really face the most risk, as they are only motivated to move forward when prompted with a reactive stimulus such as some type of litigation.

Right mix of components

On the flip side, although regulatory guidelines exist, every organization is different, and even those organizations committed to creating and implementing a sound records management structure may not hit on the right solution if they try to exactly match or replicate a records management model used by another organization. The point is that there are basic tenants and principles to good records management—not just in terms of “managing records” but also in terms of creating a positive impact on the organization’s overriding knowledge management goals—but this construct doesn’t mean that there is a “one size fits all” solution to every situation.

A fixed, static model of “good records management” doesn’t exist. The focal point of good records management is to have the right mix of records-management components in place so that an organization has the appropriate level of reliability, consistency, and flexibility necessary to anticipate and adapt to changing regulations, business needs, technology, data formats and available resources.

Being “litigation ready” is often too vague and subjective to be meaningful. The following 8-Point Inspection add some objectivity so that you can more clearly assess how prepared your organization truly is.

If you are already in-tune with your information management practices, set aside a few minutes to complete the Self Assessment on the subsequent page. Compare your total score with the Index below and use that as a benchmark for your litigation readiness initiatives:

http://www.scribd.com/doc/45028471/ZyLAB-8-Point-Inspection-of-Litigation-Readiness

Conclusion

In a survey conducted by ZyLAB it was revealed that 88% of FTSE companies in the UK were actually at risk of litigation through not having the right RM & KM framework in place.

Recent market research assessing the FTSE 100 companies’ vulnerability to ten key risk factors indicates that 88% of the FTSE 100 is at risk of litigation. Through addressing these issues you can support you organization in measuring the individual status of the information management practices within your own organization and benchmark your level of preparedness. Take control of your information now!

 More and more electronically stored information (ESI) is non-text based or does not contain any searchable text components: sound recordings, video and pictures are growing exponentially in size and more and more collaborative and social network applications support (only) these information formats. In addition, a whole generation is growing up that no longer uses written communication forms such as letters or emails: they only use social networks and other new media forms for communication and collaboration.

 Search Challenges in Visual Information Retrieval

 This transformation results in a huge future search problem, because: 

• Electronic files containing one of more text components or embedded objects with text components can be searched by using text-based queries.
•  Document scans (images) and even pictures can be enriched with the text of the original document or even with recognizable logo’s in the pictures. The same technology can also be applied to video shots. 
• Audio and the audio component of a video file can be processed by a phonetic search engine and users can search the content by looking for specific words or phoneme sequences. 
• In addition, audio-, pictures- and video files can be searched on contextual information such as the file name, added meta-information or text that surrounds the picture or the video on a web page.

Furthermore, in general, it is not possible to search a picture or a video on its content.

 Web search engines such as Google, Bing and Yahoo use primarily contextual text information from pictures and video’s to search on these object. This text can be tagged by users or can be found in the file name, file location, surrounding text on the webpage, etc. In some cases, words that are recognized in the images and videos with Optical Character Recognition (OCR) technology is used, or nudity is recognized and filtered, but that is about it. There is not or limited influence from pure visual information retrieval technology such as: give me all outdoor pictures or all images with a helicopter in it.

Additional Challenges in Visual Information Retrieval

There are a number of additional challenges in visual information retrieval that are related to the various input formats of files, internal encoding and compression (aka Codex for video), the query format (query by example of query by text), the result list format (text-based or visual-based result navigation with thumbnails and video summaries) and the viewer for the image and video files.

 State-of-the-art visual search technology should address all of these aspects and support both text-based as image or video example based querying, result navigation and viewing.

 Various Codecs

Various image input formats are more or less supported by the ZyLAB Platform, but for proper video support, one needs one of many Codex engines in order to view the video. In some examples, video is treated as a “set of images” without taking into account the proper temporal relationships. Others have a more thorough and complete internal representation, allowing for faster and more accurate viewing and navigation.

The best approach is to convert all videos and images to one common format with the same dimensions, codec and compression. Only then, extracted image features can be compared properly. There are a number of open source standards to realize this. Most vendors use the same open source LIBAVCODEC libraries form the FFMPEG project.

Enormous File Sizes

Images and videos in particular can be of enormous file size. 20 Gb for a video file is more rule than exception. As a result, processing the data often required specialized hardware with very fast and large hard disks and special graphical processing power. Viewing files requires smart streaming techniques to prevent band width overload.

There are many open source solutions available to solve these problems and many vendors use the same open source libraries.

Browsing Video and Images

When searching images and videos, the best result is almost never on the #1 position. It is even possible that it is not among the first 10! Ranking images is based on complex statistics and other mathematical properties that are not always intuitive to humans.  Users need a much more exploratory and visual result list that uses all available dimensions when searching images and videos.

A result list as it is used in text-based information retrieval does not work for searching images and video.

An example of a well defined video or image result list is shown hereunder (University of Amsterdam Forkbrowser, http://www.science.uva.nl/research/mediamill/demo/forkbrowser.php):

Use cases for Visual Information Retrieval

There are many use cases in the field of visual information retrieval varying from searching pictures on the internet to recognizing faces of hooligans at the entrance of a high risk football match, monitoring airports with surveillance cameras and investigating child abuse.

 Many of these applications are highly specialized applications requiring a lot of specialized knowledge and experience to work effectively.

However, I expect that in the next year or five, real visual information retrieval will become a core component of in-house Enterprise Information Management systems as more and more information consists of pictures and videos that are not annotated and therefore hard to find.

A legal department for example needs a state of the art software solution to deal with a crucial litigation, while the IT department needs to reduce spend by 35% to justify a new server they believe to be more essential to the smooth running of the business.

IT departments understand Records Management (RM) and strive for good Knowledge Management (KM) as a strategic part of the business value, making their lives much easier. They do however not necessarily understand the value of an RM framework that also provides e-discovery. The crucial question here is: “Can they afford not to?”

The information referenced within a RM solution is vital to the business and their strategic use is freeing up staff from spending time ‘keeping the lights on’. Nevertheless, any effective RM solution must be multifaceted and have at least some level of robust search and e-discovery capabilities to be scalable, especially in the face of the myriad of growing regulations concerning information transparency, security and expectations for data archiving and retention.

Integrating Records Management, eDiscovery and Knowledge Management

Integrating a system that comprises a comprehensive search framework into an RM solution carries with it capabilities for better overall knowledge management. This type of highly efficient, integrative RM approach can lead to enhanced competitiveness, more agility to meet specific customer needs, minimized legal risks and, at the end of the day, higher profits.

A comprehensive records management solution, supported by enterprise search and advanced content analytics functionality, can lead to better overall knowledge management and optimized pro-active e-discovery capabilities, minimized legal risks and an enhanced profit potential.

In fact, e-discovery and its supporting functionalities serve as the critical pillars upon which true records management can be built. And it is only in this foundational context that a practical, compliant, and comprehensive enterprise-wide knowledge management solution can become fully actualized.

Benefits beyond risk aversion and legal preparedness

The importance of these concepts on a realistic market level should not just be viewed through the somewhat negative prism of risk aversion and legal preparedness, though. Rather, even though the primary motivator for many companies wanting to establish solid records management policies and systems comes from the need to avoid certain risks, the structural and procedural environment created from such an initiative not only creates a strong position with which to accomplish better overall knowledge management across the organization, but it typically provides a variety of additional positive byproducts, including more nimble and proactive customer service capabilities, an unencumbered environment for innovation, optimized maneuverability and preparedness in a rapidly shifting competitive marketplace and stronger positioning for long-term profitability.

Protecting reputation and brand

Another driver to act is being ‘seen’ to have good ethical and transparent working practices. In the current court of public opinion more business are concerned with reputation and brand, and the perception of doing things right. Without good policies and the right solutions to uphold these working practices then an organization falls short of meeting external expectations. This exposure to risk makes them a potential liability to other business partners, who may shy away from an association in today’s climate of risk aversion. Therefore, not only does this leave the organization open to risk, but also can prevent other companies from doing business with your organization, having a directly negative impact on profitability.

Knowledge management allows for the establishment of an environment that promotes knowledge to be created, shared, learned, enhanced, organized and utilized for the benefit of the organization. Therefore, without properly structured records and a file plan, knowledge management is an unthinkable step for any organization. True knowledge management cannot reach fruition if an organization has not reached a suitable and committed comfort level in the management of its records.

So, on its surface, although knowledge management does not necessarily require a file plan or records series to exist, the components must be in place in order for “knowledge” to be used effectively within an organization. Without a strong records management practice, sources of knowledge cannot be easily identified or distributed.

After an organization has reached a comfortable level in the structuring of its records, it can then move on to implement a knowledge management strategy. So, in a way the natural progression is document management to records management to knowledge management.

Knowledge management puts a strong emphasis on people and joins people to records, records to experiences and experiences to people. Records management, then, is really the most immediate, tangible way in which an organization sees, uses, and understands the knowledge is has in place. At its core, records management is the pragmatic face of knowledge management.

• Creates additional flexibility regarding the way records are actually defined within the organization;
• Supports the management of records as they are currently used (a file plan);
• Provide a comprehensive and workable strategy for secure and enduring retention of records;
• Enable the accurate and secure final disposition of records, based on pertinent rules and regulations.
• Enable the management of documents and record files within the organization to enhanced competitiveness, minimized legal risks and realize timely and adequate responds to an e-discovery or e-disclosure.

Creating a flexible definition of records

Every organization should have its own definition of a record, and this definition should be based on the actual business plan and nature of that organization. The bigger and more complicated the organization, the more complicated the records series (i.e. a numeric matrix of the records in an organization), which is the basic framework upon which a file plan is based.  Because record series can become rather complicated, actual records should be defined based on content (and usability)— thereby allowing the automation of a retention schedule within each of the records series—as opposed to medium (e.g. e-mail) or document type (e.g. letter). This approach is consistent with that defined by the Department of Defense’s influential standard (DoD 5015.2-STD), which, among other guidelines, supports the notion that all documents, including e-mail, be treated according to the defined records series (and file plan).

An organization can define records as project-based collections of information if all records within that collection pertain exclusively to a particular project and if all records have the exact same lifespan (legal obligations on retention). However, a more efficient method is to create a multi-tiered file plan for project records. Using this approach means that records can be defined at the highest level as belonging to a particular project, at which point they can then be subdivided into “content” categories as the file plan is created. This structure allows for the automation of the retention schedule. Although organizations are free to define their records series and subsequent file plans, these items need to take into account the legal framework and jurisdiction where they operate and legal obligations for retention, privacy and disclosure.

Supporting a workable file plan

Any organization that is ready to implement the most up-to-date records management policies, especially to enhance overall efficiency, must consider the following factors in the context of its operation, priorities, and available resources:

• File recording
• Workable file structures
• Classification and metadata
• Evaluation and prioritization
• Retention policies
• Legal obligations
• Execution of plan
• Verification structure

Having a solid file plan in place is critical to being able to organize and enhance the efficiency of back-office operations. As previously stated, file plans are directly related to the complexity of an organization and are based on its business needs. Therefore, for the US Department of Defense, for instance, the intricacy of the file plan is enormous, but there is no other workable alternative for an organization of such size and complexity. Smaller or simpler structures within organizations would require a much simpler and more straightforward file plan.

Creating a comprehensive retention strategy

Structuring records serves another goal, namely the length of time that any organization is required to keep records. Keeping all records forever is counterproductive, as this approach would lead to unmanageable sets of data. For example, in the case of some financial organizations this would actually be breaking regulatory requirements of things such as PCI DSS. Rather, records series and file plans provide a structure to allow for the establishment of a retention schedule leading to final disposition.

After a file plan is defined, the implementation of a retention schedule attached to each record series becomes easier. This situation can be further enhanced by setting it up to provide an automated warning message that is sent to appointed officials. These officials can then affect the final disposition of particular records. Legal obligations apply, depending on the jurisdiction where the organization operates. For instance, in most parts of Europe, financial records need to be legally kept in their entirety for a period of seven years, whereas personnel records may need to be kept for up to 72 years, and so forth. In addition, depending on the policies that are mandated by law, an organization may decide to extend its retention schedule if the records in question constitute “substantial records” (i.e. records that are absolutely necessary for the re-start of operations in the event of a calamity and constitute a small percentage of the overall records of the organization). These records are required for the organization throughout its lifespan and should be kept forever.

Disposition of records

After a retention schedule is established, disposition of records can go from permanent archival/inactive status (substantial records) to destruction. Records should strictly follow this retention schedule if an effective records management policy is to be applied.

Retention schedules are complicated, and the only way to effectively apply them is through the use of an automated system, setting alarms and warnings to allow for the implementation of the retention policy. Realistically, no organization is going to perform these activities in a manual way. Logic holds, then, that a retention policy will not be possible without the presence of a strong records management policy that has clear and transparent file plans and records series.

Retention schedules can and should also be suspended in case of extenuating circumstances. In example of a court case, if certain records scheduled to be disposed of fall under disclosure obligations, these records need to be held until further notice. Destruction of this information is considered a crime in itself. Therefore, the retention schedule should have the proper flexibility to accommodate these requirements, regardless of what type of system is implemented to monitor retention processes.

Executing the file plan

The best option is the have a computer controlled filing plan execution with data inside repositories that are controlled by the RMA application. This solution is DoD compliant and is the most efficient and effective of all solutions, working well in both large, complex organizations as well as with small/linear ones. Furthermore, this method reduces the chance of human error and provides for transparent and auditable records management policies. Traditionally smaller organizations shy away from such solutions, scared by potential costs. Efficient EDRMS (RMA) solutions are scalable and costs are directly proportional to the size of the application. Consequently, this solution may well be the most cost-effective one available for any organization.

 By focusing on two components—a file plan and a final disposition plan—and using those two factors to evaluate the needs of particular business units, an organization can then build up a laundry list of the types of records management tools and capabilities they need on an organizational scale. Basing a file plan on content (theme) rather than on medium will also contribute to finding commonalities among the different groups within specific organization, allowing for an easier implementation of a records management policy. In addition, an efficient file plan should take into account particular nuances that will allow customized implementation per area as required.

Realizing timely and adequate responds to an e-discovery

An effective record-management system should enable the management of all assets within the organization, whether these are documents (paper, electronic, email) or record types.  A good system is compliant with the internationally recognized US DoD 5015.2 and Sox standards.

An integrative RM approach leads to enhanced competitiveness, more agility to meet specific customer needs, minimized legal risks, better positioning for higher profits, and supports legal departments to realize timely and adequate responds to an e-discovery or e-disclosure.

Technology is essential and the technology options abound: advanced culling; processing and (forensic)  full-text indexing; concept and fuzzy search; automatic pre-classification of documents into privileged, claim-related and non-related documents; intelligent redaction; machine translation; pattern recognition; relevance ranking; exact and near de-duplication; email trail visualizations; and many other smart and innovative solutions.

However, organizations need more than just software. They should rely on professional services from specialists and—just as importantly— best-practice methodologies, including quality control, reporting and auditing that will help organizations bring e-discovery, records management, risk control and compliance in-house in a defensible manner.

Bringing E-Discovery In-house

There are several key questions an organization must consider in today’s business climate. Is your organization prepared to respond to an e-discovery demand imposed by regulators, civil parties or a competitor? Is your organization able to find and produce all relevant documents? If so, at what price and how fast?

Only about 1% of organizations are reportedly prepared for full-scale e-discovery activities. As a result, the vast majority of organizations facing litigation are forced into a costly reaction mode in order to respond to discovery requests within court-imposed timelines.

So, when the clock is ticking and the meter is running, how do you sift through internal databases, networks, computer systems, servers, archives, backup or recovery systems, laptops, PDAs, mobile phones and pagers to assess your legal risk and defend your organization? And further, how can you possibly meet the rigorous demands of the impending e-discovery while also preparing your organization for litigation that may be looming?

Take Control of Your Information Now

In most cases, the review of information for relevance and privilege and the processing of vast volumes of data in preparation for formal legal review are the most expensive elements of e-discovery—accounting for as much as 50%-80% of the budget when using external sources. Therefore, more and more organizations have acquired advanced tools to help them to control the costs and risks of the e-discovery process. Most notably, organizations have started to deploy information management software and systems to help them respond to a specific legal matter now and prepare for future litigation.

Litigation Response and Readiness are Moving Targets

 While the latest e-discovery and information management technologies have made great progress toward putting organizations in control of their information assets and liabilities, the environment is dynamic and requires solutions that can scale and adapt to new realities:

• Even after 50 years, Moore’s Law still applies: Every 18 months, the volume of our stored data doubles. At that rate, the amount of information stored by an organization will have grown 100 times over in 10 years.
• The legal industry is one of the most conservative industries out there. Adoption of new technology is not a revolutionary process, but an evolutionary one which leads to higher costs from prolonged inefficiency.
• The use of new social media such as Twitter and Facebook and non-searchable multimedia platforms like YouTube is growing exponentially;.
• With the introduction of cloud computing, information is everywhere and nowhere at the same time.
• The number of law suits will only increase in the coming years due to the credit crisis, but also due to the increasingly litigious nature of our society.
• Therefore, the only real solution to solve the e-discovery burden now and in the future is to start properly managing all of your enterprise information as part of daily operations. Figure 2 shows the relation between e-discovery and information management.

Proactive eDiscovery: from litigation responds to litigation readiness

While in-house e-discovery systems are generally implemented to investigate a specific legal matter, more and more organizations are looking for a solid and robust foundation from which to pursue proactive, enterprise wide objectives for information management.

In fact, the costly and disruptive nature of e-discovery often triggers a broader information management initiative championed by executive leadership.

Proper information management initiatives may be based upon principles similar to e-discovery (i.e., identification, collection, review, etc.), but it often requires additional technology, procedures and user training to help organizations achieve corporate governance despite the continued— and rapid—growth of their data populations.

Information management involves not only archiving, but also the implementation and enforcement of retention and disposition schedules to ensure organizations are not storing vast volumes of information unnecessarily. It is actually quite similar to the culling process during e-discovery, yet in a broader form that is widely adopted as part of daily operations in all departments.

Email, SharePoint, HRM files, project files, customer files, official company records and legal contracts all contain potential future legal liability and cost considerations.  For example, retaining 100,000 emails—which may have important files attached—in one’s inbox, sent folder or hard disk exposes the organization to more risk and unnecessary storage costs. And the vast information stored on backup tapes and in MS SharePoint collections in a completely unstructured format presents its own potential risks, especially if the data is related to projects or litigation involving human resources or C-level management. Organizations will always have these types of unstructured data sets, but their potential risk can be managed by the right information management technology and protocol.

Information collections such as the ones noted above need to be ordered and classified in a filing plan, which can be done in a rather straightforward and pragmatic way. This action limits the amounts of data (and thus the potential cost of litigation), makes early case assessment possible on your live data, and limits the need for the expensive processes to collect and preserve all data in advance.

In the end, one will become litigation ready by making litigation response as unobtrusive as any other daily operating procedure.