D15758: Use new character width code based on Unicode 11

[Mariusz Glebocki]

mglb created this revision.
mglb added reviewers: Konsole, VDG.
mglb added a project: Konsole.
mglb requested review of this revision.

REVISION SUMMARY
  Adds a code for getting character width togeter with LUTs generated
  using uni2characterwidth from Unicode 11 lists.
  
  Skin tone, flags, gender, and other emoji with and modifer are not
  joined (you will see e.g. a skin tone square + generic yellow emoji).
  I think joining them would cause problems in most editors, command line
  prompts, and other programs which use character width data, as the
  characters would behave as combining or emoji depending on context (like
  ligatures).
  
  Examples:
  
  - light thumb up: 👍🏻
  - dark thumb up:  👍🏿
  - Polish flag:    🇵🇱
  
  This behavior is allowed:
  
  - https://unicode.org/reports/tr51/#Emoji_Modifiers_Display
  - https://unicode.org/reports/tr51/#Emoji_ZWJ_Sequences
  
  It is possible to add support for sequences, but those would work
  only for a string width functions.
  
  Some characters which can be presented as emoji are narrow (e.g. ✖️, ©️).
  Those characters are listed without "presentation" mode, which means
  they should be rendered as text by default (real presentation depends on
  renderer and/or font). Noto Sans Color Emoji renders them as wide,
  DejaVu Sans as narrow. Vim, bash and zsh treat them as narrow, so I made
  them narrow.
  
  https://unicode.org/reports/tr51/#Presentation_Style
  
  BUG: 396435
  BUG: 378124
  BUG: 392171
  BUG: 339439
  
  Depents on D15757 <https://phabricator.kde.org/D15757>

TEST PLAN
  - Look at emoji_test.txt - emojis should look "normal" (two characters
  
  width).
  
  - Look at GLASS.txt - characters width should look correct.
  - CharacterWidthTest should pass.

BRANCH
  arc/396435/Use-new-character-width-code-based-on-Unicode-11 (branched from master)

REVISION DETAIL
  https://phabricator.kde.org/D15758

AFFECTED FILES
  COPYING.Unicode
  src/CMakeLists.txt
  src/Character.h
  src/CharacterWidth.cpp
  src/CharacterWidth.h
  src/CharacterWidth.src.cpp
  src/Filter.cpp
  src/TerminalCharacterDecoder.cpp
  src/TerminalDisplay.cpp
  src/autotests/CharacterWidthTest.cpp
  src/konsole_wcwidth.cpp
  src/konsole_wcwidth.h
  tools/uni2characterwidth/overrides.txt
  tools/uni2characterwidth/template.example

To: mglb, #konsole, #vdg
Cc: konsole-devel, herrold, ngraham, maximilianocuria, hindenburg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/konsole-devel/attachments/20180926/f846551b/attachment.html>