tcltk/tcl
1
0
Fork 0
mirror of https://github.com/tcltk/tcl.git synced 2026-05-29 00:27:49 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Attempt to clarify various encoding and string representation terms

Browse Source
This commit is contained in:
apnadkarni
2025-11-06 06:47:43 +00:00
parent ccb3dd5f64
commit ac8edcf794
4 changed files with 121 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
doc/Tcl.n
View File

@@ -267,6 +267,36 @@ except for argument expansion as specified in rule [5].
For example, during variable substitution the entire value of
the variable becomes part of a single word, even if the variable's
value contains spaces.
.IP "Tcl values"
Conceptually, all values in Tcl are strings, or to be precise, sequences of
Unicode code points. This characteristic of all values being strings is
commonly referred to as the \fIEverything Is A String (EIAS)\fR principle.
.IP
For the most part, Tcl strings can be thought of as strings of characters
instead of sequences of code points and following common usage, command
descriptions refer to them as such. However, the distinction is to be noted as
some characters may have multiple representations as Unicode code point
sequences and in operations like comparisons, Tcl compares individual
code points and not individual abstract characters.
.IP
While all values are strings, individual commands may treat or
expect their arguments to be representations of data of a particular type.
For example, the \fBincr\fR command expects its operands to be integers.
.IP
The most important consequence of the EIAS principle is that all commands must
treat arguments that have a particular string representation exactly the same
irrespective of how that value was produced. Thus the \fBincr\fR command
must treat \fB100\fR produced by appending \fB0\fR to \fB10\fR exactly the
same as \fB100\fR produced by multiplying 10 by itself.
.IP
Note this EIAS principle is only true at a conceptual level and for reasons of
efficiency the implementation may choose different internal representations
for a value depending on its usage. For example, arithmetic operations on
integers may internally store the value as a native machine word. Similarly, a
\fIbinary string\fR is a sequence of Unicode code points in the range U+0000
to U+00FF and may be stored as an array of bytes. In all cases this internal
representation is irrelevant at the Tcl language level but exposed in the Tcl
C API.
.SH KEYWORDS
backslash, command, comment, script, substitution, variable
'\" Local Variables:

160
doc/Utf.3
View File

@@ -8,7 +8,7 @@
.so man.macros
.BS
.SH NAME
Tcl_UniChar, Tcl_UniCharToUtf, Tcl_UtfToUniChar, Tcl_UtfToChar16, Tcl_UtfToWChar, Tcl_UniCharToUtfDString, Tcl_UtfToUniCharDString, Tcl_Char16ToUtfDString, Tcl_UtfToWCharDString, Tcl_UtfToChar16DString, Tcl_WCharToUtfDString, Tcl_WCharLen, Tcl_Char16Len, Tcl_UniCharLen, Tcl_UniCharNcmp, Tcl_UniCharNcasecmp, Tcl_UniCharCaseMatch, Tcl_UtfNcmp, Tcl_UtfNcasecmp, Tcl_UtfCharComplete, Tcl_NumUtfChars, Tcl_UtfFindFirst, Tcl_UtfFindLast, Tcl_UtfNext, Tcl_UtfPrev, Tcl_UniCharAtIndex, Tcl_UtfAtIndex, Tcl_UtfBackslash \- routines for manipulating UTF-8 strings
Tcl_UniChar, Tcl_UniCharToUtf, Tcl_UtfToUniChar, Tcl_UtfToChar16, Tcl_UtfToWChar, Tcl_UniCharToUtfDString, Tcl_UtfToUniCharDString, Tcl_Char16ToUtfDString, Tcl_UtfToWCharDString, Tcl_UtfToChar16DString, Tcl_WCharToUtfDString, Tcl_WCharLen, Tcl_Char16Len, Tcl_UniCharLen, Tcl_UniCharNcmp, Tcl_UniCharNcasecmp, Tcl_UniCharCaseMatch, Tcl_UtfNcmp, Tcl_UtfNcasecmp, Tcl_UtfCharComplete, Tcl_NumUtfChars, Tcl_UtfFindFirst, Tcl_UtfFindLast, Tcl_UtfNext, Tcl_UtfPrev, Tcl_UniCharAtIndex, Tcl_UtfAtIndex, Tcl_UtfBackslash \- routines for manipulating UTF-8 encoded byte sequences
.SH SYNOPSIS
.nf
\fB#include <tcl.h>\fR
@@ -100,21 +100,21 @@ Tcl_Size
.AS "const Tcl_UniChar" *uniPattern in/out
.AP char *buf out
Buffer in which the UTF-8 representation of the Tcl_UniChar is stored. At most
4 bytes are stored in the buffer.
\fBTCL_UTF_MAX\fR bytes are stored in the buffer.
.AP int ch in
The Unicode character to be converted or examined.
.AP Tcl_UniChar *chPtr out
Filled with the Tcl_UniChar represented by the head of the UTF-8 string.
Filled with the Tcl_UniChar represented by the head of the UTF-8 byte sequence.
.AP unsigned short *uPtr out
Filled with the utf-16 represented by the head of the UTF-8 string.
Filled with the utf-16 represented by the head of the UTF-8 byte sequence.
.AP wchar_t *wPtr out
Filled with the wchar_t represented by the head of the UTF-8 string.
Filled with the wchar_t represented by the head of the UTF-8 byte sequence.
.AP "const char" *src in
Pointer to a UTF-8 string.
Pointer to a UTF-8 byte sequence.
.AP "const char" *cs in
Pointer to a UTF-8 string.
Pointer to a UTF-8 byte sequence.
.AP "const char" *ct in
Pointer to a UTF-8 string.
Pointer to a UTF-8 byte sequence.
.AP "const Tcl_UniChar" *uniStr in
A sequence of \fBTcl_UniChar\fR units with null-termination optional
depending on function.
@@ -145,15 +145,15 @@ If negative, the length includes all bytes until the first null.
.AP "Tcl_DString" *dsPtr in/out
A pointer to a previously initialized \fBTcl_DString\fR.
.AP "const char" *start in
Pointer to the beginning of a UTF-8 string.
Pointer to the beginning of a UTF-8 byte sequence.
.AP Tcl_Size index in
The index of a character (not byte) in the UTF-8 string.
The index of a character (not byte) in the UTF-8 byte sequence.
.AP int *readPtr out
If non-NULL, filled with the number of bytes in the backslash sequence,
including the backslash character.
.AP char *dst out
Buffer in which the bytes represented by the backslash sequence are stored.
At most 4 bytes are stored in the buffer.
At most \fBTCL_UTF_MAX\fR bytes are stored in the buffer.
.AP int nocase in
Specifies whether the match should be done case-sensitive (0) or
case-insensitive (1).
@@ -161,16 +161,26 @@ case-insensitive (1).
.SH DESCRIPTION
.PP
These routines convert between UTF-8 strings and Unicode/Utf-16 characters.
A UTF-8 character is a Unicode character represented as a varying-length
sequence of up to \fB4\fR bytes. A multibyte UTF-8 sequence
consists of a lead byte followed by some number of trail bytes.
N.B. The use of the term \fIUTF-8\fR in this documentation refers to the modified UTF-8
encoding used internally by Tcl. This differs from the UTF-8 encoding defined in
the Unicode standard with respect to the encoding of the NUL character U+0000 which
Tcl encodes internally as the byte sequence 0xC0 0x80 and not a single 0x00 bytes
as per the standard. The term \fIUTF-8 byte sequence\fR refers to a byte sequence
representing one or more Unicode code points encoded using this modified UTF-8.
The term \fIcharacter\fR refers to a Unicode code point and is used interchangeably
with it.
.PP
\fBTCL_UTF_MAX\fR is the maximum number of bytes that \fBTcl_UtfToUniChar\fR
can consume in a single call.
The routines described here convert between UTF-8 encoded byte sequences and
other representation forms.
.PP
\fBTcl_UniCharToUtf\fR stores the character \fIch\fR as a UTF-8 string
in starting at \fIbuf\fR. The return value is the number of bytes stored
The \fBTcl_UniChar\fR type is an C integer type wide enough to hold a single
Unicode code point value. A UTF-8 byte sequence encoding a single code point may
have a maximum length of 4 bytes, defined as the C preprocessor symbol
\fBTCL_UTF_MAX\fR. This is also the maximum number of bytes that
\fBTcl_UtfToUniChar\fR can consume in a single call.
.PP
\fBTcl_UniCharToUtf\fR encodes the character \fIch\fR as a UTF-8 byte sequence,
storing it starting at \fIbuf\fR. The return value is the number of bytes stored
in \fIbuf\fR. The character \fIch\fR can be or'ed with the value TCL_COMBINE
to enable special behavior, compatible with Tcl 8.x. Then, if ch is a high
surrogate (range U+D800 - U+DBFF), the return value will be 1 and a single
@@ -178,13 +188,14 @@ byte in the range 0xF0 - 0xF4 will be stored. If \fIch\fR is a low surrogate
(range U+DC00 - U+DFFF), an attempt is made to combine the result with
the earlier produced bytes, resulting in a 4-byte UTF-8 byte sequence.
.PP
\fBTcl_UtfToUniChar\fR reads one UTF-8 character starting at \fIsrc\fR
\fBTcl_UtfToUniChar\fR reads a UTF-8 byte sequence
starting at \fIsrc\fR and encoding a single code point,
and stores it as a Tcl_UniChar in \fI*chPtr\fR. The return value is the
number of bytes read from \fIsrc\fR. The caller must ensure that the
source buffer is long enough such that this routine does not run off the
end and dereference non-existent or random memory; if the source buffer
is known to be null-terminated, this will not happen. If the input is
a byte in the range 0x80 - 0x9F, \fBTcl_UtfToUniChar\fR assumes the
is known to be null-terminated, this will not happen. If the input starts
with a byte in the range 0x80 - 0x9F, \fBTcl_UtfToUniChar\fR assumes the
cp1252 encoding, stores the corresponding Tcl_UniChar in \fI*chPtr\fR
and returns 1. If the input is otherwise
not in proper UTF-8 format, \fBTcl_UtfToUniChar\fR will store the first
@@ -194,13 +205,13 @@ byte of \fIsrc\fR in \fI*chPtr\fR as a Tcl_UniChar between 0x00A0 and
\fBTcl_UniCharToUtfDString\fR converts the input in the form of a
sequence of \fBTcl_UniChar\fR code points to UTF-8, appending the result to the
previously initialized output \fBTcl_DString\fR. The return value is a pointer
to the UTF-8 representation of the \fBappended\fR string.
to the UTF-8 encoded representation of the \fBappended\fR string.
.PP
\fBTcl_UtfToUniCharDString\fR converts the input in the form of
a UTF-8 encoded string to a \fBTcl_UniChar\fR sequence
appending the result in the previously initialized \fBTcl_DString\fR.
The return value is a pointer to the appended result which is also
terminated with a \fBTcl_UniChar\fR null character.
terminated with a \fBTcl_UniChar\fR NUL character.
.PP
\fBTcl_WCharToUtfDString\fR and \fBTcl_UtfToWCharDString\fR are similar to
\fBTcl_UniCharToUtfDString\fR and \fBTcl_UtfToUniCharDString\fR except they
@@ -218,75 +229,75 @@ the number of UTF-16 units until the null.
characters. It accepts a null-terminated \fBwchar_t\fR sequence and returns
the number of \fBwchar_t\fR units until the null.
.PP
\fBTcl_UniCharLen\fR corresponds to \fBstrlen\fR for Unicode
characters. It accepts a null-terminated Unicode string and returns
the number of Unicode characters (not bytes) in that string.
\fBTcl_UniCharLen\fR corresponds to \fBstrlen\fR for Tcl_UniChar
characters. It accepts a null-terminated Tcl_UniChar string and returns
the number of Tcl_UniChar's (not bytes) in that string.
.PP
\fBTcl_UniCharNcmp\fR and \fBTcl_UniCharNcasecmp\fR correspond to
\fBstrncmp\fR and \fBstrncasecmp\fR, respectively, for Unicode characters.
They accept two null-terminated Unicode strings and the number of characters
to compare. Both strings are assumed to be at least \fIuniLength\fR characters
long. \fBTcl_UniCharNcmp\fR compares the two strings character-by-character
according to the Unicode character ordering. It returns an integer greater
than, equal to, or less than 0 if the first string is greater than, equal
to, or less than the second string respectively. \fBTcl_UniCharNcasecmp\fR
is the Unicode case insensitive version.
\fBTcl_UniCharNcmp\fR and \fBTcl_UniCharNcasecmp\fR correspond to \fBstrncmp\fR
and \fBstrncasecmp\fR, respectively, for Tcl_UniChar code points. They accept
two null-terminated Tcl_UniChar strings and the number of Tcl_UniChar code
points to compare. Both strings are assumed to be at least \fIuniLength\fR
characters long. \fBTcl_UniCharNcmp\fR compares the code points in two strings
in order according to the Unicode character ordering. It returns an integer
greater than, equal to, or less than 0 if the first string is greater than,
equal to, or less than the second string respectively. \fBTcl_UniCharNcasecmp\fR
is the case insensitive variant of \fBTcl_UniCharNcmp\fR.
.PP
\fBTcl_UniCharCaseMatch\fR is the Unicode equivalent to
\fBTcl_StringCaseMatch\fR. It accepts a null-terminated Unicode string,
a Unicode pattern, and a boolean value specifying whether the match should
\fBTcl_StringCaseMatch\fR. It accepts a null-terminated Tcl_UniChar string,
a Tcl_UniChar pattern, a boolean value specifying whether the match should
be case sensitive and returns whether the string matches the pattern.
.PP
\fBTcl_UtfNcmp\fR corresponds to \fBstrncmp\fR for UTF-8 strings. It
accepts two null-terminated UTF-8 strings and the number of characters
to compare. (Both strings are assumed to be at least \fIlength\fR
characters long.) \fBTcl_UtfNcmp\fR compares the two strings
character-by-character according to the Unicode character ordering.
\fBTcl_UtfNcmp\fR corresponds to \fBstrncmp\fR and accepts two null-terminated
UTF-8 encoded strings each of which should represent a sequence of at least
\fIlength\fR code points. \fBTcl_UtfNcmp\fR compares the code points represented
by each of the encoded strings in order.
It returns an integer greater than, equal to, or less than 0 if the
first string is greater than, equal to, or less than the second string
respectively.
.PP
\fBTcl_UtfNcasecmp\fR corresponds to \fBstrncasecmp\fR for UTF-8
\fBTcl_UtfNcasecmp\fR corresponds to \fBstrncasecmp\fR for UTF-8 encoded
strings. It is similar to \fBTcl_UtfNcmp\fR except comparisons ignore
differences in case when comparing upper, lower or title case
characters.
.PP
\fBTcl_UtfCharComplete\fR returns 1 if the source UTF-8 string \fIsrc\fR
\fBTcl_UtfCharComplete\fR returns 1 if the source UTF-8 byte sequence \fIsrc\fR
of \fInumBytes\fR bytes is long enough to be decoded by
\fBTcl_UtfToUniChar\fR/\fBTcl_UtfNext\fR, or 0 otherwise. This function
does not guarantee that the UTF-8 string is properly formed. This routine
does not guarantee that the UTF-8 byte sequence is properly formed. This routine
is used by procedures that are operating on a byte at a time and need to
know if a full Unicode character has been seen.
.PP
\fBTcl_NumUtfChars\fR corresponds to \fBstrlen\fR for UTF-8 strings. It
returns the number of Tcl_UniChars that are represented by the UTF-8 string
\fIsrc\fR. The length of the source string is \fIlength\fR bytes. If the
length is negative, all bytes up to the first null byte are used.
\fBTcl_NumUtfChars\fR corresponds to \fBstrlen\fR for UTF-8 byte sequences. It
returns the number of Tcl_UniChars that are represented by the UTF-8 byte
sequence \fIsrc\fR. The length of the source string is \fIlength\fR bytes. If
the length is negative, all bytes up to the first null byte are used.
.PP
\fBTcl_UtfFindFirst\fR corresponds to \fBstrchr\fR for UTF-8 strings. It
returns a pointer to the first occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 string \fIsrc\fR. The null terminator is
considered part of the UTF-8 string.
\fBTcl_UtfFindFirst\fR corresponds to \fBstrchr\fR for UTF-8 byte sequences.
It returns a pointer to the first occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 byte sequence \fIsrc\fR. The null terminator is
considered part of the byte sequence.
.PP
\fBTcl_UtfFindLast\fR corresponds to \fBstrrchr\fR for UTF-8 strings. It
returns a pointer to the last occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 string \fIsrc\fR. The null terminator is
considered part of the UTF-8 string.
\fBTcl_UtfFindLast\fR corresponds to \fBstrrchr\fR for UTF-8 byte sequences.
It returns a pointer to the last occurrence of the Unicode character \fIch\fR
in the null-terminated UTF-8 byte sequence \fIsrc\fR. The null terminator is
considered part of the byte sequence.
.PP
Given \fIsrc\fR, a pointer to some location in a UTF-8 string,
\fBTcl_UtfNext\fR returns a pointer to the next UTF-8 character in the
string. The caller must not ask for the next character after the last
Given \fIsrc\fR, a pointer to some location in a UTF-8 byte sequence,
\fBTcl_UtfNext\fR returns a pointer to the start of the UTF-8 byte sequence
corresponding to the next character
The caller must not ask for the next character after the last
character in the string if the string is not terminated by a null
character. \fBTcl_UtfCharComplete\fR can be used in that case to
make sure enough bytes are available before calling \fBTcl_UtfNext\fR.
.PP
\fBTcl_UtfPrev\fR is used to step backward through but not beyond the
UTF-8 string that begins at \fIstart\fR. If the UTF-8 string is made
up entirely of complete and well-formed characters, and \fIsrc\fR points
to the lead byte of one of those characters (or to the location one byte
past the end of the string), then repeated calls of \fBTcl_UtfPrev\fR will
return pointers to the lead bytes of each character in the string, one
character at a time, terminating when it returns \fIstart\fR.
\fBTcl_UtfPrev\fR is used to step backward through but not beyond the UTF-8
byte sequence that begins at \fIstart\fR. If the byte sequence is made
up entirely of complete and well-formed characters, and \fIsrc\fR points to
the lead byte of one of those characters (or to the location one byte past the
end of the string), then repeated calls of \fBTcl_UtfPrev\fR will return
pointers to the lead bytes of each character in the string, one character at a
time, terminating when it returns \fIstart\fR.
.PP
When the conditions of completeness and well-formedness may not be satisfied,
a more precise description of the function of \fBTcl_UtfPrev\fR is necessary.
@@ -303,25 +314,24 @@ byte \fIsrc[0]\fR nor the byte \fIstart[-1]\fR nor the byte
\fIsrc[-5]\fR.
.PP
\fBTcl_UniCharAtIndex\fR corresponds to a C string array dereference or the
Pascal Ord() function. It returns the Unicode character represented at the
specified character (not byte) \fIindex\fR in the UTF-8 string
Pascal Ord() function. It returns the Unicode code point represented at the
specified character (not byte) \fIindex\fR in the UTF-8 byte sequence
\fIsrc\fR. The source string must contain at least \fIindex\fR
characters. If \fIindex\fR is negative it returns -1.
.PP
\fBTcl_UtfAtIndex\fR returns a pointer to the specified character (not
byte) \fIindex\fR in the UTF-8 string \fIsrc\fR. The source string must
byte) \fIindex\fR in the UTF-8 byte sequence \fIsrc\fR. The source must
contain at least \fIindex\fR characters. This is equivalent to calling
\fBTcl_UtfToUniChar\fR \fIindex\fR times. If \fIindex\fR is negative,
the return pointer points to the first character in the source string.
the return pointer points to the first character in the source.
.PP
\fBTcl_UtfBackslash\fR is a utility procedure used by several of the Tcl
commands. It parses a backslash sequence and stores the properly formed
UTF-8 character represented by the backslash sequence in the output
buffer \fIdst\fR. At most 4 bytes are stored in the buffer.
UTF-8 encoding of the character represented by the backslash sequence in the output
buffer \fIdst\fR. At most \fBTCL_UTF_MAX\fR bytes are stored in the buffer.
\fBTcl_UtfBackslash\fR modifies \fI*readPtr\fR to contain the number
of bytes in the backslash sequence, including the backslash character.
The return value is the number of bytes stored in the output buffer.
.PP
See the \fBTcl\fR manual entry for information on the valid backslash
sequences. All of the sequences described in the Tcl manual entry are
supported by \fBTcl_UtfBackslash\fR.

4
doc/lsearch.n
View File

@@ -103,8 +103,8 @@ one takes precedence. The default is \fB\-ascii\fR.
.TP
\fB\-ascii\fR
.
The list elements are to be examined as Unicode strings (the name is
for backward-compatibility reasons.)
The list elements are to be examined as Tcl strings (i.e. sequences of
Unicode code points; the name is for backward-compatibility reasons.)
.\" OPTION: -dictionary
.TP
\fB\-dictionary\fR

4
doc/string.n
View File

@@ -17,6 +17,10 @@ string \- Manipulate strings
.SH DESCRIPTION
.PP
Performs one of several string operations, depending on \fIoption\fR.
.PP
\fBFollowing common usage, the command descriptions below use the
term \fIcharacter\fB in place of the more accurate term \IBcode point\fR.
.PP
The legal \fIoption\fRs (which may be abbreviated) are:
.\" METHOD: cat
.TP