Add support for language detection

[richardwilly98]

Tika has a language detection feature [1]. It would be very useful to include in this plugin.

There is already a plugin for language detection in ES [2] but I was not able to get it working for attachment type.

[1] - https://tika.apache.org/1.4/detection.html#Language_Detection
[2] - https://github.com/jprante/elasticsearch-langdetect

[richardwilly98]

Nice article about language detection accuracy [1].

[1] - http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html

[jprante]

The langdetect plugin can now be combined with the attachment mapper plugin. See jprante/elasticsearch-langdetect#5

[dadoonet]

Heya,

I started to play with your PR (rebasing, changing some part of code, adding some other tests). See branch here: https://github.com/dadoonet/elasticsearch-mapper-attachments/tree/issue/44-langdetect

Some comments:

• accuracy: https://github.com/dadoonet/elasticsearch-mapper-attachments/blob/4e0422403abfaccf7602d3a33a7d28ad3207c8bc/src/test/java/org/elasticsearch/index/mapper/xcontent/LanguageDetectionAttachmentMapperTests.java#L102 gives strange results. I try to detect :-) and I get back lt
• performance: I think we need to test the performance cost here. As stated in the article you mentioned, detection comes with a cost. If we want to add this feature, I think we should add a setting like index.mapping.attachment.language_detection default to false.

What do you think?

[richardwilly98]

I am not sure about accuracy. I have used elasticsearch-langdetect [1] but did not find it accurate.
I tested Tika version as well and found it pretty good (but I have not done extensive testing). And your example is funny :-)

Regarding performance I definitely agree language detection should be disabled by default.

[1] - https://github.com/jprante/elasticsearch-langdetect

[jprante]

Regarding accuracy, the language-detection which is used in elasticsearch-langdetect was found to be more accurate than Tika:

http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html

If you have observed accuracy issues, feel free to open an issue https://github.com/jprante/elasticsearch-langdetect/issues

[richardwilly98]

This link is the main reason I have been using elasticsearch-langdetect plugin.

I have create a new issue jprante/elasticsearch-langdetect#8