Module encoding_rs::mem
source · [−]Expand description
Functions for converting between different in-RAM representations of text and for quickly checking if the Unicode Bidirectional Algorithm can be avoided.
By using slices for output, the functions here seek to enable by-register (ALU register or SIMD register as available) operations in order to outperform iterator-based conversions available in the Rust standard library.
Note: “Latin1” in this module refers to the Unicode range from U+0000 to U+00FF, inclusive, and does not refer to the windows-1252 range. This in-memory encoding is sometimes used as a storage optimization of text when UTF-16 indexing and length semantics are exposed.
The FFI binding for this module are in the encoding_c_mem crate.
Enums
Classification of text as Latin1 (all code points are below U+0100), left-to-right with some non-Latin1 characters or as containing at least some right-to-left characters.
Functions
Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.
Checks whether a potentially invalid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.
Checks whether a potentially invalid UTF-16 buffer contains code points that trigger right-to-left processing or is all-Latin1.
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system.
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 with potentially insufficient output space.
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-16.
Converts valid UTF-8 to valid UTF-16.
If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
Converts potentially-invalid UTF-8 to valid UTF-16 with errors replaced with the REPLACEMENT CHARACTER.
Converts potentially-invalid UTF-8 to valid UTF-16 signaling on error.
If the input is valid UTF-16 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system.
Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.
Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER.
Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER with potentially insufficient output space.
Copies ASCII from source to destination up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).
Copies ASCII from source to destination zero-extending it to UTF-16 up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).
Copies Basic Latin from source to destination narrowing it to ASCII up to the first non-Basic Latin code unit (or the end of the input if it is Basic Latin in its entirety).
Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.
If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
Replaces unpaired surrogates in the input with the REPLACEMENT CHARACTER.
Checks whether the buffer is all-ASCII.
Checks whether the buffer is all-Basic Latin (i.e. UTF-16 representing only ASCII characters).
Checks whether a scalar value triggers right-to-left processing.
Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing.
Checks whether the buffer represents only code points less than or equal to U+00FF.
Checks whether a potentially-invalid UTF-8 buffer contains code points that trigger right-to-left processing.
Checks whether the buffer is valid UTF-8 representing only code points less than or equal to U+00FF.
Checks whether a UTF-16 buffer contains code points that trigger right-to-left processing.
Checks whether a UTF-16 code unit triggers right-to-left processing.
Checks whether the buffer represents only code point less than or equal to U+00FF.
Returns the index of first byte that starts a non-Latin1 byte sequence, or the length of the string if there are none.
Returns the index of first byte that starts an invalid byte sequence or a non-Latin1 byte sequence, or the length of the string if there are neither.
Returns the index of the first unpaired surrogate or, if the input is valid UTF-16 in its entirety, the length of the input.