GitLab

Included OLD_HTML_OUTPUT into the main build as this functionality is still required for LatinoInterfaces/ClowdFlows/TextFlows which fails to build otherwise.

Corrected LatinoWorkflows2010 project to be compilable with the latest VS2010 releases of Latino, SemWeb and OpenNLP projects. Included file config.cs & reference to System.Web.Extensions.

Separated code page detection and language detection

Fixed links in README.md

Added avro persistance module to the LatinoWorkflows2010.csproj. It provides reader and writer classes for streaming lwf documents to avro files. Extendable classes are provided to add support for arbitrary schemes (eg. html docs).

Moved boilerplate tag (skip tag) assessment from HtmlTokenizerComponent to UrlTreeBoilerplateRemoverComponent

HtmlTokenizerComponent: added additional skip tags (forms, embedded content, header, footer, nav, aside...)

UrlTreeBoilerplateRemoverComponent now takes linkToTextRatio into account (simple heuristic)

Fixed a bug in DocumentWriterComponent

First step towards improved boilerplate remover: HTML tokenizer now assigns isLink feature to text blocks

Added overloads of Document's WriteXmlCompressed and ReadXmlCompressed functions which now accept stream parameters (before the only option was a file).

DocumentWriterComponent: implemented retry-on-deadlock, bug fix (html -> xml in file names written to DB)
Added .gitignore

DocumentCorpusWriterComponent.cs: not writing HTMLs anymore
DocumentWriterComponent.cs: now writes old corpus and document ID into DB