Experiments in Data and Tool Blending: Around the World in 8 Languages

In this series, we are experimenting with the most interesting blends of data and tools. Whether it’s mixing traditional sources with modern data lakes, open-source DevOps on the cloud with protected internal legacy tools, SQL with NoSQL, web-wisdom-of-the-crowd with in-house handwritten notes, or IoT sensor data with idle chatting, we’re curious to find out: will they blend? Want to find out what happens when IBM Watson meets Google News, Hadoop Hive meets Excel, R meets Python, or MS Word meets MongoDB?

No doubt you are familiar with the adventure novel Around the World in 80 Days in which British gentleman Phileas Fogg makes a bet that he can circumnavigate the world in 80 days. Today, we will be attempting a similar journey. However, ours is unlikely to be quite as adventurous as the one Phileas made. We won’t be riding elephants across the Indian mainland, nor rescuing our travel companion from the circus. And we certainly won’t be getting attacked by Native American Sioux warriors!

Our adventure will begin from our offices on the Lake of Constance in Germany. From there, we will travel down to Italy, stopping briefly to see the Coliseum. Then, across the Mediterranean to see the Pyramids of Egypt and on through the Middle East to the ancient city of Persepolis. After a detour via Russia to see the Red Square in Moscow, our next stop will be the serene beaches of Thailand for a short break before we head off to walk the Great Wall of China (or at least part of it). On the way home, we will stop in and say hello to our colleagues in the Texas office.

Like all good travelers, we want to stay up-to-date with the news the entire time. Our goal is to read the local newspapers… in the local language, of course! This means reading the news in German, Italian, Arabic, Farsi, Chinese, Russian, Thai, and lastly, English. Impossible you say? Well, we’ll see.

The real question is: will all those languages blend?

To get the RSS news feeds, we used the RSS Feed Reader node. The RSS Feed Reader node connects to the RSS Feed URL, downloads the feed articles, parses them with the help of a tokenizer, and saves them in a document type column at the output port.

Figure 1: Final workflow connecting to several RSS feed URLs for news in eight different languages, pre-processing the articles, and creating a word cloud from the most frequent words. The workflow is available on the KNIME EXAMPLES server under 08_Other_Analytics_Types/01_Text_Processing/20_BlendLanguagesInTagCloud *. The workflow comes in a folder containing also two metanode templates necessary to the workflow execution.

It is important to notice that the text parsing operation performed by the RSS Feed Reader node is language-dependent. A few language specific tokenizers are embedded into the node, such as for English, Spanish, German, and Chinese. However, for most of the languages, no specific tokenizer is available in the RSS Feed Reader node nor in the other Text Processing nodes. In these cases, the two basic Open NLP Simple Tokenizer and Open NLP Whitespace Tokenizer might do.

Then, for each language, a Table Creator node with URLs of the local news is fed to the RSS Feed Reader node. You can see the different language branches in the final workflow in Figure 1.

Document pre-processing mainly involves clean-up tasks such as keep Document column only, remove all numbers, and erase punctuation signs.

Since those operations are language-independent, the same metanode can be used for all languages — not only eight replicas of the same metanode, but eight links to one single metanode template!

Note: The adoption of one metanode template makes maintenance much easier than using eight replicas.

To create a metanode template, just right-click the metanode in the workflow and in the context menu select Metanode > Save as Template, then select the folder destination in the KNIME Explorer panel and click OK.