This is expected behavior. The size of the key is measured in bytes. When converting data from one codepage to another, the size of the data may increase due to the codepage and collation used.
For a single-byte codepage, one character is represented in an index key with one byte. For UTF-8 with BASIC collation, which contains some non-ASCII data, each character could be longer than one byte.
When using ICU collations then there is a much larger impact on key size, to store multi-level comparison data in the key.
When an index key is created, the limit is 2000 bytes (this used to be 200 bytes, but was raised within the OE10 timeframe). How many characters this is depends on the codepage and collation, and how many components (fields) there are in the index.