Text Tokenizer
This extension is developed to improve the process of replacing text parts with tokens (e.g. localization of hardcoded values) and to provide visual hints. To enable all features it's required to configure the extension.
Text tokenization
Each time you replace a string with a token it is stored in memory. You can make multiple replacement operations one by one and finally generate the results as a JSON file which can be easily processed for any application.
Inline and mouseover hints
If properly configured (see "Configuration" section below) it would provide:
- inline hints (useful to detect tokens without assigned values)
- full-text value preview on mouseover.

Configuration
It's recommended to define custom options in the workspace settings.
Text tokenization configuration
text-tokenizer.tokenWrapper
The default scenario is to have a token inside some wrapper (e.g. a localization function or an HTML tag) Define a text-tokenizer.tokenWrapper
and it would be sed as a fallback value if no override is configured in text-tokenizer.tokenWrappersByLanguageId
.
{
"text-tokenizer.tokenWrapper": "$translate(\"%token%\")"
}
text-tokenizer.tokenWrappersByLanguageId
To have different wrappers for different file types you can define a text-tokenizer.tokenWrappersByLanguageId
param. It relies on language ID detected by VS Code:
{
"text-tokenizer.tokenWrappersByLanguageId": {
"html": "{{ $translate(\"%token%\") }}",
"typescript": "useTranslator(\"%token%\")"
}
}
Inline and mouseover hints configuration
text-tokenizer.tokenLookupRegExps
It is possible to provide multiple regular expressions to let the extension to look up for different patterns to extract token. Each regular expression should have "token" named capturing group.
{
"text-tokenizer.tokenLookupRegExps": [
"translate\\((['\"`])(?<token>.+?)\\1\\)",
"useTranslator\\((['\"`])(?<token>.+?)\\1\\)"
]
}
text-tokenizer.tokenCollectionPath
This option should be set up with the text-tokenizer.tokenCollectionGetter
. It is used to watch file changes and update token list only when it is necessary.
- The path should be absolute.
- If
text-tokenizer.tokenCollectionGetter
is set to the preset (e.g. json-array
, json-map
) the path should point to a valid JSON file.
- If
text-tokenizer.tokenCollectionGetter
is a path to a custom function, this path would be used only for tracking file changes to get token updates.
{
"text-tokenizer.tokenCollectionPath": "/absolute/path/to/file.json"
}
text-tokenizer.tokenCollectionGetter
It's possible to enable the functionality of getting code hints for the existing tokens (in-memory ones would be shown anyway) by providing the preset key or an absolute path to the file with the default export of the function.
Available values:
json-map
would read JSON file from text-tokenizer.tokenCollectionPath
absolute path. Data should be in the following format:
// workspace.json
{
"text-tokenizer.tokenCollectionGetter": "json-map",
"text-tokenizer.tokenCollectionPath": "/absolute/path/to/token-collection.json"
}
// token-collection.json
{
"token1": "value1",
"token2": "value2",
...
}
json-array
would read JSON file from text-tokenizer.tokenCollectionPath
absolute path. Data should be in the following format:
// workspace.json
{
"text-tokenizer.tokenCollectionGetter": "json-array",
"text-tokenizer.tokenCollectionPath": "/absolute/path/to/token-collection.json"
}
// token-collection.json
[
{
"token": "token1",
"value": "value1"
},
{
"token": "token2",
"value": "value2"
}
]
/absolute/path/to/token-getter-function.js
pointing to a file containing default export with a function which returns a Promise with an array of tokens and values in the following format:
// workspace.json
{
"text-tokenizer.tokenCollectionGetter": "/absolute/path/to/token-getter-function.js",
"text-tokenizer.tokenCollectionPath": "/absolute/path/to/token-collection.json"
}
// token-collection.json
{
tokens: [
{
"token": "token.number.one",
"value": "The corresponding value One"
},
{
"token": "token.number.two",
"value": "The corresponding value Two"
}
]
}
// token-getter-function.js
const { readFile } = require("fs/promises");
module.exports = async function (tokenCollectionPath) {
const fileContent = await readFile(tokenCollectionPath, {
encoding: "utf-8",
});
return JSON.parse(fileContent).tokens;
};
Inline hints configuration
Inline hints would be helpful if basic "mouseover hints" configuration is done.
text-tokenizer.inlineHints
Enable or disable inline hints. It would visualize values preview for each token and mark tokens without values.
{
"text-tokenizer.inlineHints": true
}
text-tokenizer.inlineValueCSS
CSS to be applied to inline hints when value is found.
{
"text-tokenizer.inlineValueCSS": "display: inline-block;margin: 0 5px 0 5px;padding: 0 5px 0 5px;"
}
text-tokenizer.inlineValueNotFoundCSS
CSS to be applied to inline hints when value is not found.
{
"text-tokenizer.inlineValueNotFoundCSS": "display: inline-block;margin: 0 5px 0 5px;padding: 0 5px 0 5px;"
}
Commands
Text tokenizer: Replace selection with a token
Select a text fragment you wish to replace and execute the command. You'll see a prompt, enter a token, and it would replace your selection.
You can execute the command multiple times. All selected text-token pairs would be stored until you run the "Text tokenizer: Generate results and clean cache" command.
Text tokenizer: Generate results and clean cache
This command would open a new document with all replaced selection-to-token pairs.
The text tokenizer store would be cleaned, so copy the result before closing the generated document
[
{
"token": "token.one",
"value": "selection-1"
},
{
"token": "token.two",
"value": "selection-2"
}
]
Limitations
I'm trying to improve this extension in my free time, but still there is much to do. The most important limitation is caused by the fact that now this extension relies on RegExps, so:
- it can't detect dynamic tokens like
errors.${key}
(language server implementation PR would be welcome).