How to mitigate highlighting issues when switching colorschemes?

[lifepillar]

As you know, filetype-specific syntax items and colorscheme's switching don't go very well together—the emblematic discussion of the topic likely being altercation/solarized#102.

In a nutshell, the issue can be reproduced as follows (using Vim 8.1.1300):

1. Create a minimal colorscheme ~/.vim/colors/test.vim:

set background=dark
hi clear
if exists('syntax_on')
  syntax reset
endif
let g:colors_name = 'test'
hi! link vimCommand ErrorMsg
hi vimOption cterm=reverse gui=reverse

2. Edit your vimrc;

3. :colorscheme test

4. :colorscheme default

You will see that vimCommand is still linked to ErrorMsg and vimOption is cleared.

I have thought hard about possible workarounds and my conclusion is that this issue has no solution—except avoiding using syntax items in colorschemes—unless some new mechanism (a new command?) is introduced in Vim to deal specifically with it. Am I right?

Note that avoiding syntax items in colorschemes is somewhat limiting, as one can realize by looking at two of the most popular colorschemes (Solarized and Gruvbox): people constantly ask to tweak the highlighting for specific filetypes.

[andymass]

I've traced all code paths and there is no way to clear/restore a highlight group so that "hi def" will work as expected- switching colorschemes will always leave problems behind.

This function call would work, but there's no vim script which will use this combination of parameters.

  do_highlight(name, TRUE, TRUE);    // force init a group

There is no way to restore the highlights to "factory" settings (i.e., after starting vim and enabling syntax).

[mg979]

IMHO, the problem would be easily solved, if:

hi default link {group}

could reset the highlight group that is defined in the syntax file, when the group has been cleared. Instead this command fails if the group exists and has been cleared, while this command

hi! default link {group}

restores the highlight, but does it even if it's not currently cleared (it totally ignores the default part). It's the default option that is not working as it should (IMHO).

[lifepillar]

It's not as simple as that, I believe. For that to work, syntax reset (or something else) should be extended to clear all syntax groups Vim currently knows about. That might be easy to implement, I guess. That would also clear highlight groups defined in a user's vimrc: this would break backward compatibility, but Vim might start recommending putting one's customizations in after/syntax/<filetype>.vim rather than in vimrc (in :help hi-default).

But then, Vim should also reload syntax/<filetype>.vim for each filetype with a buffer currently opened before loading the new colorscheme (but after syntax reset), and after/syntax/<filetype>.vim after the colorscheme has been loaded. Failing to do so would result in syntax highlighting be off for most of your code. Or should the user be left responsible for reloading a filetype plugin after changing the colorscheme?

[andymass]

Vim should also reload syntax/.vim for each filetype with a buffer currently opened

vim actually does reload the syntax files for each buffer when the commands :syntax on or :syntax enable are used. See

vim/runtime/syntax/syntax.vim

Line 40 in 110289e

doautoall syntaxset FileType

The help says "If you want Vim to overrule your settings with the defaults, use: :syntax on" although this doesn't seem to be completely true.

[mg979]

For what I could understand, the sequence is:

1. a colorscheme is loaded (with all groups it defines)
2. syntax is loaded, when filetype is applied. hi def link fills the gaps, if the colorscheme didn't define some groups.
3. you load a new colorscheme. All syntax is cleared and reset. hi def link this time won't do anything at all: groups that have been cleared, won't be restored to their default.
4. the new colorscheme applies its own groups: if some groups had previously been defined, they've been cleared, and if the new colorscheme does not redefine them, they stay cleared. And this is the problem.

If this is right, imho having hi def link (called from the syntax file) to be able to restore a group that has been cleared, would solve the issue. The new colorscheme would then apply its own groups, but default groups would still be working, even if the new colorscheme does not define them.

What would be missing, would be the group that are defined by the old scheme, and that are not defined in the syntax file (hi def link) or in the new colorscheme. These groups would be cleared and stay cleared, but I think it would be a good thing, why should they carry over to schemes that don't define them, if they don't belong to the syntax file either? They belong to that colorscheme only and it's right if they are removed as well.

[lifepillar]

vim actually does reload the syntax files for each buffer when the commands :syntax on or :syntax enable are used.

But a colorscheme only executes syntax reset. Am I missing something? Or are you suggesting that a colorscheme use syntax on instead?

@mg979 When you load a new colorscheme, groups with settings are cleared, but groups that link to another group are not touched (see vimCommand in my first post). How do you deal with those?

[andymass]

But a colorscheme only executes syntax reset

Yes you are right. My point was, a user who really wants a colorscheme to look exactly as intended might think to do this when changing schemes:

let g:colors_name = "new_scheme"
syntax on

This almost works. But because of the "hi def" issue it doesn't always.

[mg979]

I must be wrong then. I tried to use this command in a syntax file:

command! -nargs=1 HiDefault call s:HiDefault(<q-args>)

function! s:HiDefault(groups)
  try
    let newHi = split(a:groups)[0]
    if match(execute('hi ' . newHi), 'xxx cleared') >= 0
      exe 'hi! default link' a:groups
      echom newHi 'had been cleared'
    endif
  catch
    exe 'hi default link' a:groups
  endtry
endfunction

and replaced all hi def link commands with this one, but I can't see any change. If I run filetype detect after colorscheme change, broken highlight gets fixed though.

Edit: the command is able to restore broken highlight on filetype detect at least, while it doesn't work without it. So I still think that hi def link is misbehaving when a syntax has been cleared by a colorscheme.

[romainl]

How about a solution similar to b:undo_ftplugin and b:undo_indent?

let g:undo_colorscheme = 'highlight clear'
    \ .. ' highlight! link <one of our linked group> NONE'
    \ .. ' highlight! link <another one of our linked group> NONE'
    \ .. ' highlight! link <and another one of our linked group> NONE'

Like those features, g:undo_colorscheme could be executed to undo everything done in the current colorscheme and, essentially, go back to a pristine default before sourcing the new colorscheme.

[adscriven]

I like @romainl's idea.

I've tried to automate this in the past. But I found it distasteful and stopped using it shortly thereafter, so I'm not sure how reliable it is. Autocommands are tricky things.

The idea: record existing links before changing them. For filetype-specific syntax items, record the existing link, unless there is none yet. In that case defer creating the new link until after the syntax file has been sourced. When switching to another colorscheme, restore links using the gathered information.

I only do this below when linking, but I guess you could also make s:fthi() for setting colours directly too.

" colors/linktest.vim
hi clear
let s:name = expand('<sfile>:t:r')
let colors_name = s:name

fun! s:hi(grp, fg, bg, attr)
    exe 'hi! ' . a:grp . ' ctermfg='.a:fg . ' ctermbg='.a:bg . ' cterm='.a:attr
endfun
com! -nargs=+ Hi call s:hi(<f-args>)

let s:links = {}
fun! s:link(to, ...)
    for o in a:000
        sil! let old = matchstr(execute('hi ' . o), 'links to \zs.*')
        let s:links[o] = old == '' ? 'NONE' : old
        exe 'hi! link ' . o . ' ' . a:to
    endfor
endfun
com! -nargs=+ Link call s:link(<f-args>)

" only checks first item for a link
fun! s:ftlink(ft, to, ...)
    sil! let old = matchstr(execute('hi ' . a:000[0]), 'links to \zs.*')
    if old == ''
        exe 'au colorscheme_' . s:name . ' filetype ' . a:ft .
                \ ' call s:link("' . a:to . '", "' .
                \ join(a:000, '", "') . '")'
    else
        call call('s:link', [a:to]+a:000)
    endif
endfun

com! -nargs=+ Ftlink call s:ftlink(<f-args>)

fun! s:cleanup()
    for [link, old] in items(s:links)
        exe 'hi link ' . link . ' ' . old
    endfor
endfun

exe 'augroup colorscheme_' . s:name
au!
au colorschemepre * call s:cleanup()
exe 'au colorschemepre * au! colorscheme_' . s:name
augroup END

Hi normal 7 0 NONE
Hi emphasis bg fg NONE
Hi surprise 15 5 bold
Link emphasis directory whatever
Ftlink help emphasis helphypertextjump otherstuff
Ftlink vim surprise vimcomment vimstring

delcom Hi
delcom Link
delcom Ftlink

[mg979]

I think vimscript masochism in this case won't help, imho the solution is easy (but must be implemented in the C source): don't keep cleared groups around. If a group has been cleared, delete it, so that hi default link will be able to restore the default highlight group as defined by the syntax file.

[adscriven]

Yeah, I know, but workarounds were being discussed. However I couldn't agree more. That's why I stopped trying to use a workaround myself -- it's not the right thing to do.