Tibetan and Himalayan Library - THL

thl header title text

ACIP To Tibetan Converters

This document describes the ACIP->Tibetan converters built atop Jskad.  These converters were initially written by David Chandler, a volunteer with the Tibetan and Himalayan Digital Library, in the latter half of 2003.  They built upon the work of Tony Duff, Edward Garrett, and Than Garson, and they would not be possible without the assistance of David Chapman, Robert Chilton, and Andrés Montano Pellegrini.  (Please correct, and forgive, any omissions from these lists.)

These converters accept Asian Classics Input Project (ACIP) transliteration of Tibetan (using ACIP's Tibetan Input Code), a Roman transliteration scheme.  ACIP has many Buddhist texts available in ACIP transliteration, which alone makes ACIP transliteration (or just ACIP for short) important.

The converters here accept a text file of ACIP and output either a Unicode UTF-8-encoded text file (using any Tibetan Unicode font, such as Tibetan Machine Uni, TMU) or a Rich Text Format (RTF) file of Tibetan Machine Web (TMW).  Unicode is the unified standard encoding for all scripts that will be understood by software for foreseeable future. The latter (TMW) is the legacy version of TMU, which THL previous used for web-pages and makes beautiful hardcopies.

The converters are meant to produce perfect results even for imperfect input.  To give you an idea of the thought and care that went into these converters, consider the following partial list of features:

  • Four tiers of warning and error messages are available.
  • Some transliterations specified by the ACIP standard are not accepted (i.e., they cause errors) because they are used too often improperly in Release V texts (e.g., {\}); some non-standard transliteration is understood because it is used in ACIP Release V texts (e.g., {[DD1]}).
  • Non-standard Unicode character escapes are supported.  (In this way, the glyph that the ACIP {\} refers to according to the standard can in fact be represented, via {\u0F84}.)
  • Color-coding can help find typos in the input.
  • A substitution mechanism allows for correcting erroneous documents on the fly.
  • The converters can output frequency statistics.
  • The "lexical analyzer" and "parser" handle every intricacy of real ACIP Release V texts.
  • The knowledge regarding the TMW font has been verified by independent teams as described here.

The ACIP->Unicode and ACIP->TMW converters are equally good.  There are some differences between the two, though.  The TMW font has only a fixed set of glyphs, whereas Unicode can encode arbitrary Tibetan glyphs.  Thus, the hypothetical ACIP {GAI}, which parses as {G+AI} due to prefix rules, will give an error in an ACIP->TMW conversion because no glyph exists for this stack.  The ACIP->Unicode conversion will succeed, having generated correct Unicode.  This is the only difference between the two conversions.

The converters are actively maintained; your feedback is valued.

Note that there are also TMW->ACIP converters available; this document does not cover them.

In what follows, you will learn how to use the converters, including all the features listed above, and you'll find a list of known bugs and places where there is room for improvement.

Using the Converters

This section briefly describes how the converters are best used.

The GUI and command-line interfaces are both sufficient; the GUI interface is your best bet if you've not used the converters before.  To learn how to invoke these interfaces, read these instructions.

First, review the known bugs and be sure you can live with them.

Now perform a trial conversion of your document with warnings disabled.  You will first ensure that no outright errors appear in the input.  If any do, make a copy of the input, edit the input, and feed it through again.  Feel free to try this out as soon as you're comfortable; the error messages themselves are sometimes self-explanatory.

Once all errors have been corrected, do a conversion with warning level 'Some'.  If any warnings mark real problems, correct those problems.

If you have the patience, now do a conversion with warning level 'Most' and correct further problems.  If any warnings mark real problems, correct those problems.

The 'All' warning level is pedantic; you might find it useful if you're writing software that is to produce ACIP transliteration that is easily read by machines.  If you find any useful warnings at this level, report it as a bug -- such warnings should be 'Most' or 'Some' level.

For best results, produce color-coded output.  Scan the output for non-native tsheg bars and ensure that they match the original document (the one from which the ACIP transliteration was produced).  Color-coding is useful because, for example, {ZHIGN} is probably a typo for {ZHING}; {ZHIGN} will appear colored, whereas {ZHING} is not colored.

Note that the ACIP {%} gives a warning every time.  Use the Unicode escape {\u0F35} if you want to avoid this warning, but note well that Unicode escapes are not part of the ACIP standard.  Thus, other tools that work with ACIP transliteration will likely not understand {\u0F35}.

To save time, you may use the tsheg-bar substitution mechanism when appropriate.

Even if your desired end result is Unicode output, an ACIP->TMW conversion is sometimes useful.  One benefit is that errors will appear for any ACIP tsheg bar that refers to a consonant stack not included in TMW.  These stacks should be scrutinized, because TMW contains over 500 of the most common consonant stacks.

Finally, check a few folios by hand against the original document to be sure that you're satisfied with the conversion.

Diagnostics: Warnings and Errors

These converters are designed such that the output is just what you yourself would create by hand.  Whenever there is doubt about what output is desired, a warning or error is issued.  This means that a helpful warning or error message will appear in the output, and that you will be told at the end of the conversion that one or more warnings or errors have indeed occurred.  You can then search your output document for the text [#ERROR or [#WARNING.

Some warning or error messages refer to lexical errors, that is, errors that occurs when breaking an input text up into tsheg bars.  Others are parsing errors, that is, errors that occur during the interpretation of ACIP tsheg bars.  It helps to understand both these processes.

There are four warning levels: 'None', 'Some', 'Most', and 'All'.  Choose 'None' if you don't want any warnings to appear in your output and be brought to your attention at the end of conversion.  Choose 'Some' if you want to see the most important warnings, 'Most' if you want some real confidence in your output, and 'All' if you've absolutely got to know that the output is right.

Errors will always appear; you cannot disable them.

It is possible to alter the severity of a warning at runtime.  It is not possible to make an error a warning, however, and it is not possible to make a warning into an error (though that might be useful [vote for RFE #954903 if you want it].  To change the severity of a warning, set the system property thdl.acip.to.tibetan.warning.severity.XXX, where XXX is the error number, e.g. 501, to your choice of DISABLED, Some, Most, or All.  Alternatively, alter options.txt, a file found inside the top level of the JAR file, as the comments in that file indicate.  These instructions are for experts; please contact the developers if you need help.

One may choose to have ACIP->Tibetan ERRORS appear in long (i.e., verbose) form or in short (i.e., terse) forms.  When short forms appear, they are embedded in the output like [#ERROR 130: {X}].  The long forms are as follows:

101: There's not even a unique, non-illegal parse for {X}

102: Found an open bracket, 'X', within a [#COMMENT]-style comment. Brackets may not appear in comments.

103: Found a truly unmatched close bracket, 'X'.

104: Found a closing bracket, 'X', without a matching open bracket. Perhaps a [#COMMENT] incorrectly written as [COMMENT], or a [*CORRECTION] written incorrectly as [CORRECTION], caused this.

105: Found a truly unmatched open bracket, '[' or '{', prior to this current illegal open bracket, 'X'.

106: Found an illegal open bracket (in context, this is 'X'). Perhaps there is a [#COMMENT] written incorrectly as [COMMENT], or a [*CORRECTION] written incorrectly as [CORRECTION], or an unmatched open bracket?

107: Found an illegal at sign, @ (in context, this is X). This folio marker has a period, '.', at the end of it, which is illegal.

108: Found an illegal at sign, @ (in context, this is X). This folio marker is not followed by whitespace, as is expected.

109: Found an illegal at sign, @ (in context, this is X). @012B is an example of a legal folio marker.

110: Found //, which could be legal (the Unicode would be \u0F3C\u0F3D), but is likely in an illegal construct like //NYA\\.

111: Found an illegal open parenthesis, '('. Nesting of parentheses is not allowed.

112: Unexpected closing parenthesis, ')', found.

113: The ACIP {?}, found alone, may intend U+0F08, but it may intend a question mark, i.e. '?', in the output. It may even mean that the original text could not be deciphered with certainty, like the ACIP {[?]} does.

114: Found an illegal, unprintable character.

115: Found a backslash, \, which the ACIP Tibetan Input Code standard says represents a Sanskrit virama. In practice, though, this is so often misused (to represent U+0F3D) that {\} always generates this error. If you want a Sanskrit virama, change the input document to use {\u0F84} instead of {\}. If you want U+0F3D, use {/NYA/} or {/NYA\u0F3D}.

116: Found an illegal character, 'X', with ordinal (in decimal) 88.

117: Unexpected end of input; truly unmatched open bracket found.

118: Unmatched open bracket found. A comment does not terminate.

119: Unmatched open bracket found. A correction does not terminate.

120: Slashes are supposed to occur in pairs, but the input had an unmatched '/' character.

121: Parentheses are supposed to occur in pairs, but the input had an unmatched parenthesis, '('.

122: Warning, empty tsheg bar found while converting from ACIP!

123: Cannot convert ACIP {X} because it contains a number but also a non-number.

124: Cannot convert ACIP {X} because {V}, wa-zur, appears without being subscribed to a consonant.

125: Cannot convert ACIP {X} because we would be required to assume that {A} is a consonant, when it is not clear if it is a consonant or a vowel.

126: Cannot convert ACIP {X} because it ends with a '+'.

127: Cannot convert ACIP {X} because it ends with a '-'.

128: Cannot convert ACIP {X} because A: is a "vowel" without an associated consonant.

129: Cannot convert ACIP {X} because + is not an ACIP consonant.

130: The tsheg bar ("syllable") {X} is essentially nothing.

131: The ACIP caret, {^}, must precede a tsheg bar.

132: The ACIP {X} must be glued to the end of a tsheg bar, but this one was not.

133: Cannot convert the ACIP {X} to Tibetan because it is unclear what the result should be. The correct output would likely require special mark-up.

134: The tsheg bar ("syllable") {X} has no legal parses.

135: The Unicode escape 'X' with ordinal (in decimal) 88 is specified by the Extended Wylie Transliteration Scheme (EWTS), but is in the private-use area (PUA) of Unicode and will thus not be written out into the output lest you think other tools will be able to understand this non-standard construction.

136: The Unicode escape with ordinal (in decimal) 88 does not match up with any TibetanMachineWeb glyph.

137: The ACIP {X} cannot be represented with the TibetanMachine or TibetanMachineWeb fonts because no such glyph exists in these fonts. The TibetanMachineWeb font has only a limited number of ready-made, precomposed glyphs, and {X} is not one of them.

138: The Unicode escape 'X' with ordinal (in decimal) 88 is in the Tibetan range of Unicode (i.e., [U+0F00, U+0FFF]), but is a reserved code in that area.

139: Found an illegal open bracket (in context, this is 'X'). There is no matching closing bracket.

140: Unmatched closing bracket, 'X', found. Pairs are expected, as in [#THIS] or [THAT]. Nesting is not allowed.

141: While waiting for a closing bracket, an opening bracket, 'X', was found instead. Nesting of bracketed expressions is not permitted.

142: Because you requested conversion to a Unicode text file, there is no way to indicate that the font size is supposed to decrease starting here and continuing until error 143. That is, this is the beginning of a region in YIG CHUNG.

143: Because you requested conversion to a Unicode text file, there is no way to indicate that the font size is supposed to increase (go back to the size it was before the last error 142, that is) starting here. That is, this is the end of a region in YIG CHUNG.

Just as with ERRORS, one may choose to have WARNINGS appear in either short or long form.  The long forms of warnings are as follows:

501: Using X, but only because the tool's knowledge of prefix rules (see the documentation) says that XX is not a legal Tibetan tsheg bar ("syllable")

502: The last stack does not have a vowel in {X}; this may indicate a typo, because Sanskrit, which this probably is (because it's not legal Tibetan), should have a vowel after each stack.

503: Though {X} is unambiguous, it would be more computer-friendly if '+' signs were used to stack things because there are two (or more) ways to interpret this ACIP if you're not careful.

504: The ACIP {X} is treated by this converter as U+0F35, but sometimes might represent U+0F14 in practice. To avoid seeing this warning again, change the input to use {\u0F35} instead of {X}.

505: There is a useless disambiguator in {X}.

506: There is a stack of three or more consonants in {X} that uses at least one '+' but does not use a '+' between each consonant.

507: There is a chance that the ACIP {X} was intended to represent more consonants than we parsed it as representing -- GHNYA, e.g., means GH+NYA, but you can imagine seeing GH+N+YA and typing GHNYA for it too.

508: The ACIP {X} has been interpreted as two stacks, not one, but you may wish to confirm that the original text had two stacks as it would be an easy mistake to make to see one stack (because there is such a stack used in Sanskrit transliteration for this particular sequence) and forget to input it with '+' characters.

509: The ACIP {X} has an initial sequence that has been interpreted as two stacks, a prefix and a root stack, not one nonnative stack, but you may wish to confirm that the original text had two stacks as it would be an easy mistake to make to see one stack (because there is such a stack used in Sanskrit transliteration for this particular sequence) and forget to input it with '+' characters.

510: A non-breaking tsheg, 'X', appeared, but not like "...," or ".," or ".dA" or ".DA".

511: The ACIP {X} cannot be represented with the TibetanMachine or TibetanMachineWeb fonts because no such glyph exists in these fonts. The TibetanMachineWeb font has only a limited number of ready-made, precomposed glyphs, and {X} is not one of them.

512: There is a chance that the ACIP {X} was intended to represent more consonants than we parsed it as representing -- GHNYA, e.g., means GH+NYA, but you can imagine seeing GH+N+YA and typing GHNYA for it too. In fact, there are glyphs in the Tibetan Machine font for N+N+Y, N+G+H, G+N+Y, G+H+N+Y, T+N+Y, T+S+TH, T+S+N, T+S+N+Y, TS+NY, TS+N+Y, H+N+Y, M+N+Y, T+S+M, T+S+M+Y, T+S+Y, T+S+R, T+S+V, N+T+S, T+S, S+H, R+T+S, R+T+S+N, R+T+S+N+Y, and N+Y, indicating the importance of these easily mistyped stacks, so the possibility is very real.

The above messages are perhaps not verbose enough to help you figure out what the converter thinks is wrong or questionable, so below is further explanation of a few error and warning messages:

  • Error 131 appears for {^  GONG SA}, for example, because only {^GONG SA} and {^ GONG SA} are supported in this implementation.
  • Error 128 appears for the input {:} because {:} cannot appear alone.  (Sloppily, this message exposes you to the internals of the converter, where {:} is thought of as {A:} in some contexts.)
  • Error 132 appears because {%}, {o}, and {x} are really only to be applied to whole tsheg bars, and should not occur alone.
  • Each of warnings 501, 508 and 509 appears because it helps evince the impact of prefix rules, a subtle point with regard to ACIP because they are implied, but not discussed explicitly in depth, by the ACIP standard.
  • Warning 504 appears because some ACIP transliteration out there does use {%} to mean U+0F14.

Coloration

For ACIP->TMW conversions (not ACIP->Unicode), color-coding of tsheg bars is an option.  The command-line converters accept a flag --colors yes|no; the conversion GUI in Jskad has a checkbox for color-coding.

Warnings and errors appear in red; tsheg bars that would parse differently if other prefix rules were used appear in yellow; non-native tsheg bars appear in green.

Tsheg-bar Statistics

The ACIP->Tibetan converters provide a simple-minded accounting mechanism with which one can determine which tsheg bars appear in a conversion or how many times each tsheg bar appears.  This mechanism is for power users only at this point; its user interface leaves much to be desired.  If you wish to produce frequency information, and if you are not familiar with some sort of scripting (via Excel macros, Unix shell scripts, etc.), then the output produced will likely be useless to you.

To support the calculation of frequency statistics, that is, how many times each tsheg bar appears, the converter can output all tsheg bars to the Java error console (i.e., System.err).  Each will appear on the console as many times as it appears in the input.  To activate this functionality, set the system property org.thdl.tib.text.ttt.OutputAllTshegBars to true, and be prepared for voluminous output.  Massaging this output into a friendly tabular format is quite possible but not described here; contact the developers for help.

To support the generation of syllabaries, the converter can output each tsheg bar encountered to the Java error console (i.e., System.err).  Each will appear on the console only once, no matter how many times it appears in the input.  To activate this functionality, set the system property org.thdl.tib.text.ttt.OutputUniqueTshegBars to true, and be prepared for voluminous output.

If desired, each tsheg bar output can be prefixed with a string of your choice by setting the system property org.thdl.tib.text.ttt.PrefixForOutputTshegBars to that string.  This is useful if the converter is producing other output on the console and you want to separate that output from the statistics.

Tsheg-bar Substitution

The ACIP->Tibetan converters provide a mechanism for automatically correcting common transliteration typos.  For example, if your document contains 100 occurrences of {KAsh} that all in fact intend {K+sh}, then you can specify just once the rule {KAsh}->{K+sh}, and all 100 occurrences will be treated correctly.  This mechanism is not very easy to use, but it is completely customizable; you can specify any number of rules.  You can only perform such substitutions at the tsheg bar level, though.  This means, for example, that you cannot specify the rule {GONG SA}->{^GONG SA}; you can only specify {GONG}->{^GONG}, which would affect {GONG LA} just as it would affect {GONG SA}.

To perform substitutions, set the system property org.thdl.tib.text.ttt.ReplacementMap to be a comma-delimited list of x=>y pairs.  For example, if you think BLKU, which parses as B+L+KU, should parse as B-L+KU, and you want KAsh to be parsed as K+sh because the input operators mistyped it, then set org.thdl.tib.text.ttt.ReplacementMap to BLKU=>B-L+KU,KAsh=>K+sh.  Note that this will not cause {B+L+KU} to become {B-L+KU} -- we are doing the replacement during lexical analysis of the input file, not during parsing.  And it will cause {SBLKU} to become {SB-L+KU}, which is parsed as {S+B-L+KU}, probably not what you wanted.  If you fear such things, you can see if they happen by setting the system property org.thdl.tib.text.ttt.VerboseReplacementMap to true, which will cause an informational message to be printed on the Java console every time a replacement is made.

Furthermore, you can use the regular expression notations ^ and $ to denote the beginning and end of the tsheg bar, respectively.  For example, ^BLKU$=>B-L+KU is a useful rule.  Note that full regular expressions are not supported -- the tool just borrows a bit of the notation.  The rule ^BLKU=>B-L+KU means that {BLKUM} and {BLKU} will both be replaced, but {SBLKU} and {SBLKUM} will not be.  The caret, ^, means that we only match if BLKU is at the beginning.  The dollar sign, $, means that we only match if the pattern is at the end.  The rule BLKU$=>B-L+KU will cause {SBLKU} to be replaced, but not {BLKUM}.  Note that performance is far better for ^FOO$ than for ^FOO, FOO$, or FOO alone.

Only one substitution is made per tsheg bar.  ^FOO$-style mappings will be tried first, then ^FOO-style, then FOO$-style, and finally FOO-style.

An example of a useful substitution is o$=>\u0F35.  This is useful because the converters interpret the ACIP {o} as U+0F37 by default, but you might prefer U+0F35 in your output.

Note that you cannot literally replace {FOO} with {BAR} using this mechanism -- because {F} is not an ACIP character, the lex will not get far enough to use this substitution mechanism.  This is not considered a design flaw -- serious errors require user intervention.  Sophisticated users can use something akin to perl, sed, or awk scripts to preprocess the input.

Note also that you cannot use the rule ONYA=>O&, although it would be nice if you could.  Technically, {&} is considered to be punctuation (i.e., that which divides tsheg bars) and is not understood inside a tsheg bar.

Note that this mechanism is also useful for fixing problems in the converter itself rather than in the input.

Unicode Character Escapes

The ACIP->Tibetan converters support some non-standard extensions to the ACIP Tibetan Input Code Standard.  One of those is Unicode character escape sequences.  This extension makes it possible to represent characters that the ACIP standard does not address, and to represent one character, U+0F84, that ACIP does address with the transliteration {\} but that is misused in practice so often to refer to U+0F3C that the ACIP->Tibetan converters always produce an error upon seeing {\}.

Outside of comments and the like, {\uKLMN} is interpreted as referring to the Unicode character with ordinal KLMN, where each of K, L, M, and N are case-insensitive hexadecimal digits.  For example, the ACIP {KA KHA GA NGA } is exactly equivalent to {\u0F40\u0f0B\u0F41\u0F0B\u0F42\u0F0B\u0F44\u0f0b}.  Unicode escapes produce the obvious Unicode in an ACIP->Unicode conversion, and they produce the correct TMW glyph in an ACIP->TMW conversion.  There are limits, though, when converting to TMW; multiple escapes in sequence are not handled correctly.  It would take a Unicode to TMW converter to produce the correct glyphs for {\u0F42\u0F92\u0FB7\u0F7C}.  The escapes for vowels and other characters that are mapped to multiple TMW glyphs are also not handled perfectly.  Best practice is to use escapes only when necessary in an ACIP->TMW conversion.

The Unicode character represented need not be a Tibetan one; for example, {\u0040} produces the at sign, @.

The latest Extended Wylie Transliteration Scheme standard has assigned private-use area (PUA) Unicode codepoints to some TMW glyphs.  ACIP documents that have a Unicode escape in the range U+F021 to U+F0FF, inclusive, are interpreted as intending these TMW glyphs.  ACIP->Unicode produces an error for such an escape because it is font-dependent and not standard.  Other tools will likely not understand such Unicode, so the converter will not produce it.  If you want it in the output, it is there in the error message.

Note well the known bug with regard to whitespace in transliteration that follows a Unicode escape.  In large part, this bug affects characters that can be transliterated by other, simpler, standard means.

If you do want to disable the use of Unicode escapes, set the system property thdl.tib.text.disallow.unicode.character.escapes.in.acip to true.

Breaking a Text Up Into tsheg bars

The ACIP->Tibetan converters all take ACIP transliteration as input.  The first step in conversion is to break up the input into manageable pieces.  (This is known as lexical analysis in the context of programming languages, and you may see the term in diagnostic messages though a linguist who studies human language like Tibetan might balk at the term.)  The correct pieces in this case are tsheg bars (in ACIP, {TSEG BAR}), punctuation, comments, whitespace, folio markers, formatting codes, etc.  In this section, the intracacies of how the converter does that will be laid bare.  With luck, this will help you understand why the converter treated one space character (i.e, ' ', U+0020) as a tsheg and another as Tibetan whitespace.

The Tibetan term tsheg bar refers to "the stuff between the dots".  In the ACIP {BKRA SHIS [# Notice that this comment is embedded in the Tibetan greeting pronounced 'tashi delay']BDE LEGS,}, there are four tsheg bars, 'BKRA', 'SHIS', 'BDE', and 'LEGS'.  In this case 'BDE' is literally "between the dots"; i.e., it is sandwiched by two U+0F0B characters (because comments are in a sense invisible).  One of the "dots" that touches 'LEGS' does not look like a dot -- it is a shad, U+0F0D.  The lexical analyzer also finds one comment, which will appear in a Latin typeface in the output, and it finds four pieces of punctuation -- three tshegs and a shad.

The converter will not allow an illegal character into a tsheg bar.  For example, {jA} is an error and causes an error message to appear in the output.

Now that the basic operation is clear from the above example, let's cover the fine points of how standard ACIP is handled.  We'll also cover some non-standard constructs that appear commonly in actual ACIP Release V texts.

The first construct that deserves explanation is the line break.  By the ACIP standard, line breaks in the input do not become line breaks in the output unless there are two line breaks in the input.  For example, the ACIP snippet below has only one line break in the output although three line breaks appear in the input: 

BKRA SHIS 
BDE LEGS,

THUGS RJE CHE ... and so on ...

One fine point is that the converter does not require a space before a line break. If {SHIS} appears before a line break, the converter inserts a space so that it's treated just like {SHIS } is treated.  This oddity is needed to convert real ACIP documents.

Another fine point is that ACIP's {^} character "eats" a following space or a newline.  This is so that {^ GONG SA } is treated identically to {^GONG SA }.

Text inside a matching pair of square brackets (e.g., [# A COMMENT] or [BP]) is passed through untouched into the output; the brackets remain.  Nesting is not allowed.  Text inside a matching pair of curly brackets (e.g., {# A COMMENT} or {BP}) is passed through untouched into the output; the brackets disappear.  Nesting is not allowed.  (Note that the source code implements two algorithms for handling square and curly brackets; the one described here is presently in use.  But if you desire different handling, please e-mail the developers to ask if it isn't a five-minute job to make that happen.)

The converter also supports several non-standard folio markers.  A review of ACIP Release V texts determined that the following types of folio markers can appear: 

@001
@001A
@001B
@01A.3
@012A.3
@[07B]
@00007B
@00007
@B00007
@[00007A]

The list above was created by a most fallible process of reviewing a large number of ACIP Release V texts.  Your suggestions for additions to this list are highly valued; please contact the developers.

The converters will insert a tsheg in some places where no ACIP { } appears; this happens after {PA} and {DANG,} below: 

GA PA

GA PHA 

DAM,
LHAG

GA CA,

GA

Note that a space appears after {PHA}, and a comma appears after {CA}, but {PA} has nothing between it and a line break.  The converters are smart enough to insert a tsheg regardless.

Also missing from the above ACIP, but inserted automatically by the converters, is Tibetan whitespace; the converter sees {DAM, LHAG} instead of {DAM,LHAG} above.

If such automatic corrections are not desired, try using a Unicode escape before the line break instead of {PA} or {,}.

The converters also treat {NGA,} as a typo for {NGA ,} (actually, {NGA\u0F0C,} since one wouldn't want a line break to occur after the tsheg and cause a shad to begin a line; see the section on formatting Tibetan texts in the Tibetan! 5.1 documentation) because Tibetan typesetting requires that NGA not appear directly before a shad.  (Perhaps {NGA,} would look too much like {KA}.)

The converters embody the rule that a shad does not appear after GA or KA unless a shabs kyu vowel is on the GA or KA.  For example, the space in {MA ,HA} is a tsheg, and the space in {KU ,HA} is a tsheg, but the space in {GA ,HA} is Tibetan whitespace.

If you find that the converters put a tsheg where it does not belong, miss a tsheg, or put whitespace where it does belong, please contact the developers.

Though the ACIP standard does not mention it, it appears that some ACIP Release V texts use a period (i.e., {.}) to indicate a non-breaking tsheg (i.e., U+0F0C).  Search for {NGO.,}, {....,DAM}, etc.  Unless {,}, {.}, or a letter (i.e., 'a' through 'z' or 'A' through 'Z') follows the {.}, it is only grudingly interpreted as a non-breaking tsheg -- a warning is generated, too.

Note that the treatment of the very last line in an input text is circumspect.

Parsing tsheg bars: Greedy Stacking and Nativeness

This section is a technical reference sufficiently detailed so that you can fully understand the inner workings of the converter as it decides which Unicode or TMW to use for a given tsheg bar.  The problem of breaking up a text into tsheg bars is a separate issue; this section describes what happens to a tsheg bar after it's been chipped away from the text.

The ACIP->Tibetan converters have a notion of nativeness.  Each tsheg bar is either native Tibetan or non-native.  For example, in Buddhist texts written in Tibetan, Sanskrit mantras often appear in Tibetan characters.  This "Tibetanized Sanskrit" is non-native.  The tsheg bars that make up this mantra (and here, take "tsheg bar" somewhat literally to mean the characters delimited by punctuation and whitespace) are some native and some non-native in the converter's eyes.  For example, the tsheg bar {MA } appears in some mantras, and is thus in fact non-native.  The converter, however, treats {MA } as native in all contexts.  Thus, "native" is a technical term with a slightly different meaning than usual.

The idea of nativeness is important because it affects how the converter treats a tsheg bar.  In ACIP transliteration, the rule is that consonants stack up until punctuation, whitespace, or a vowel appears.  For example, {RDZYA} is equivalent to {R+DZ+YA}.  ({DZA} always means the letter {DZA} itself, never {D+ZA}.)  But this greedy stacking does not apply to {SOGS}, which is equivalent to {SOG-S}, not {SOG+S}.  Why not?  Because {SOGS} is a native tsheg bar where GA is the suffix and SA is the postsuffix.  Similarly, {GNAD} is {G-NAD}, not {G+NAD}.  Why?  Because GA is a prefix in this native Tibetan tsheg bar.

In this section, we will illustrate the inner workings of this aspect of the converter.  You will be able to determine which snippets of transliteration the converter considers to be native tsheg bars, where greedy stacking does not apply except for the root stack, and which snippets are non-native, and thus wholly subject to greedy stacking.

Anatomy of a Native tsheg bar

First, the lexical analyzer ensures that only the Tibetan and Sanskrit consonants, the vowels {A}, {I}, {U}, {E}, {O}, {OO}, {EE}, {i}, {'A}, {'I}, {'U}, {'E}, {'O}, {'OO}, {'EE}, and {'i}, and the adornments {m} and {:} are allowed in a tsheg bar.

As far as the converter is concerned, a native tsheg bar consists of an optional prefix, a native root stack, an optional suffix, an optional postsuffix (also known as a secondary suffix) that may only be present if a suffix is present, and zero or more appendages (my term, created because I don't know what a grammarian calls such a thing).  An appendage is one of the following stack sequences:

  • {'E}
  • {'I}
  • {'O}
  • {'U}
  • {'US}
  • {'UR}
  • {'UM}
  • {'ONG}
  • {'ONGS}
  • {'OS}
  • {'IS}
  • {'UNG}
  • {'ANG}
  • {'AM}

A tsheg bar is non-native if it has a non-native root stack or if it contains the {:} character.  Any vowel is allowed on a native root stack, even {'EEm}, {i}, or the like.

The rule about native root stacks is important, for example, in determining that {KTYAMS} is {K+T+YAM+SA} instead of {K+T+YAMASA} (because K+T+YA is not a native stack).  Another example is {GNVA}, which is treated like {G+N+VA}, not {G-N+VA}, even though {GNA} is treated like {G-NA} because NA can take a GA prefix.  The complete list of native stacks is the following:

  • KA
  • KHA
  • GA
  • NGA
  • CA
  • CHA
  • JA
  • NYA
  • TA
  • THA
  • DA
  • NA
  • PA
  • PHA
  • BA
  • MA
  • TZA
  • TSA
  • DZA
  • WA
  • ZHA
  • ZA
  • 'A
  • YA
  • RA
  • LA
  • SHA
  • SA
  • HA
  • AA
  • R+KA (RKA)
  • R+GA (RGA)
  • R+NGA (RNGA)
  • R+JA (RJA)
  • R+NYA (RNYA)
  • R+TA (RTA)
  • R+DA (RDA)
  • R+NA (RNA)
  • R+BA (RBA)
  • R+MA (RMA)
  • R+TZA (RTZA)
  • R+DZA (RDZA)
  • L+KA (LKA)
  • L+GA (LGA)
  • L+NGA (LNGA)
  • L+CA (LCA)
  • L+JA (LJA)
  • L+TA (LTA)
  • L+DA (LDA)
  • L+PA (LPA)
  • L+BA (LBA)
  • L+HA (LHA)
  • S+KA (SKA)
  • S+GA (SGA)
  • S+NGA (SNGA)
  • S+NYA (SNYA)
  • S+TA (STA)
  • S+DA (SDA)
  • S+NA (SNA)
  • S+PA (SPA)
  • S+BA (SBA)
  • S+MA (SMA)
  • S+TZA (STZA)
  • K+VA (KVA)
  • KH+VA (KHVA)
  • G+VA (GVA)
  • C+VA (CVA)
  • NY+VA (NYVA)
  • T+VA (TVA)
  • D+VA (DVA)
  • TZ+VA (TZVA)
  • TS+VA (TSVA)
  • ZH+VA (ZHVA)
  • Z+VA (ZVA)
  • R+VA (RVA)
  • SH+VA (SHVA)
  • S+VA (SVA)
  • H+VA (HVA)
  • K+YA (KYA)
  • KH+YA (KHYA)
  • G+YA (GYA)
  • P+YA (PYA)
  • PH+YA (PHYA)
  • B+YA (BYA)
  • M+YA (MYA)
  • K+RA (KRA)
  • KH+RA (KHRA)
  • G+RA (GRA)
  • T+RA (TRA)
  • TH+RA (THRA)
  • D+RA (DRA)
  • P+RA (PRA)
  • PH+RA (PHRA)
  • B+RA (BRA)
  • M+RA (MRA)
  • SH+RA (SHRA)
  • S+RA (SRA)
  • H+RA (HRA)
  • K+LA (KLA)
  • G+LA (GLA)
  • B+LA (BLA)
  • Z+LA (ZLA)
  • R+LA (RLA)
  • S+LA (SLA)
  • R+K+YA (RKYA)
  • R+G+YA (RGYA)
  • R+M+YA (RMYA)
  • R+G+VA (RGVA)
  • R+TZ+VA (RTZVA)
  • S+K+YA (SKYA)
  • S+G+YA (SGYA)
  • S+P+YA (SPYA)
  • S+B+YA (SBYA)
  • S+M+YA (SMYA)
  • S+K+RA (SKRA)
  • S+G+RA (SGRA)
  • S+N+RA (SNRA)
  • S+P+RA (SPRA)
  • S+B+RA (SBRA)
  • S+M+RA (SMRA)
  • G+R+VA (GRVA)
  • D+R+VA (DRVA)
  • PH+Y+VA (PHYVA)

(Some would argue that LVA is notably absent.  It is seen in ACIP Buddhist texts in {AELVA}, {LVAm}, {LVU}, {LVUN}, {LVAR}, {LVE}, {LVANG}, and {LVA}.  Greedy stacking affects none of these tsheg bars' parsing, however.)

Not all characters can be prefixes and the like.  Only the five prefixes (GA, DA, BA, MA, 'A), ten suffixes (GA, NGA, DA, NA, BA, MA, 'A, RA, LA, SA), and two postsuffixes (DA, SA) every Tibetan student knows are allowed, and they cannot appear with vowels.  (In {LE'U}, {'} is not a suffix -- it is part of an appendage.)  In fact, certain prefixes may only appear with certain root stacks.  The reason that these prefix rules matter is that they govern how tsheg bars are parsed.  For example, {GNA} is parsed like {G-NA}, because NA takes a GA prefix.  But {GPA} is parsed like {G+PA}, because PA does not take a GA prefix.

Prefix rules are a topic of some controversy; different grammars give different lists of prefix rules.  For a converter, it is important that the converter's knowledge of prefix rules matches the knowledge of the person who typed in the ACIP transliteration, not that the converter agrees with a grammarian.  For example, if the input technician thought that PA could take a GA prefix, then the converter will produce {G+PA} when {G-PA} was intended.  For this reason, the converter can produce a warning every time a prefix rule prohibited the treatment of one of the five prefixes as a prefix.  For example, {GPA} produces this warning.  However, {GNA} produces no warning, because the converter assumes that it is unlikely that an input technician would enter {GNA} upon seeing {G+NA}.  Part of the reason for this assumption is that the Asian Classics Input Project Entry Operator Transcription Chart as of Spring, 1993, explicitly enumerates the following cases for special treatment by input operators:

  • {BDA'} vs. {B+DA}
  • {DBANG} vs. {D+BA}
  • {DGA'} vs. {D+GA}
  • {DGRA} vs. {D+GRA}
  • {DGYES} vs. {D+GYA}
  • {DMAR} vs. {D+MA}
  • {GDA'} vs. {G+DA}
  • {GNAD} vs. {G+NA}
  • {MNA'} vs. {M+NA}

Regardless, for best results, you should ensure that the input technician's knowledge of prefix rules matches the converter's knowledge.  The following are the legal combinations of prefix and root stack in the converter's eyes:

  • The BA prefix may occur with any of the following stacks:
    • KA
    • SA
    • CA
    • TA
    • TZA
    • GA
    • DA
    • ZHA
    • ZA
    • SHA
    • K+YA (KYA)
    • G+YA (GYA)
    • K+RA (KRA)
    • G+RA (GRA)
    • S+RA (SRA)
    • G+LA (GLA)
    • K+LA (KLA)
    • Z+LA (ZLA)
    • R+LA (RLA)
    • S+LA (SLA)
    • S+KA (SKA)
    • S+GA (SGA)
    • S+NGA (SNGA)
    • S+NYA (SNYA)
    • S+TA (STA)
    • S+DA (SDA)
    • S+NA (SNA)
    • S+TZA (STZA)
    • R+KA (RKA)
    • R+GA (RGA)
    • R+NGA (RNGA)
    • R+JA (RJA)
    • R+NYA (RNYA)
    • R+TA (RTA)
    • R+DA (RDA)
    • R+NA (RNA)
    • R+TZA (RTZA)
    • R+DZA (RDZA)
    • L+CA (LCA)
    • L+TA (LTA)
    • L+DA (LDA)
    • R+K+YA (RKYA)
    • R+G+YA (RGYA)
    • S+K+YA (SKYA)
    • S+G+YA (SGYA)
    • S+K+RA (SKRA)
    • S+G+RA (SGRA)
  • The GA prefix may occur with any of the following stacks:
    • CA
    • DA
    • NA
    • NYA
    • SA
    • SHA
    • TA
    • TZA
    • YA
    • ZA
    • ZHA
  • The 'A prefix may occur with any of the following stacks:
    • GA
    • JA
    • DA
    • BA
    • DZA
    • KHA
    • CHA
    • THA
    • PHA
    • TSA
    • PH+YA (PHYA)
    • B+YA (BYA)
    • KH+YA (KHYA)
    • G+YA (GYA)
    • B+RA (BRA)
    • KH+RA (KHRA)
    • G+RA (GRA)
    • D+RA (DRA)
    • PH+RA (PHRA)
  • The MA prefix may occur with any of the following stacks:
    • KHA
    • GA
    • CHA
    • JA
    • THA
    • TSA
    • DA
    • DZA
    • NGA
    • NYA
    • NA
    • KH+YA (KHYA)
    • G+YA (GYA)
    • KH+RA (KHRA)
    • G+RA (GRA)
  • The DA prefix may occur with any of the following stacks:
    • BA
    • GA
    • KA
    • MA
    • NGA
    • PA
    • B+RA (BRA)
    • B+YA (BYA)
    • G+RA (GRA)
    • G+YA (GYA)
    • K+RA (KRA)
    • K+YA (KYA)
    • M+YA (MYA)
    • P+RA (PRA)
    • P+YA (PYA)

In the above list, the presence of wa-zur (ACIP {V}) does not disallow a prefix-root combination; nor does the presence of any vowel, even {'EEm}.  The presence of {:} does disallow prefix-root combinations; e.g., {GN'EEm} is {G-N'EEm}, but {GNA:} is {G+NA:}.  ({GNVA} is parsed as {G+N+VA} not because NVA cannot take a GA prefix, but because NVA is not a native stack.)

The converter will allow any suffix to go with any native root or prefix-root combination; it will allow any postsuffix to follow any suffix.  It will allow any appendage on any native tsheg bar.

For example, {SOGS}, {BSOGS}, {BS'EEmGS}, {LE'U'I'O} and {BSKYABS-'UR-'UNG-'O} are all native tsheg bars in the converter's eyes.  Note the need for disambiguation: {PAM-'AM} is a native tsheg bar, but {PAM'AM}, which parses as the three stacks {PA}, {M'A}, and {MA}, is not.  (In practice, appendages rarely occur after prefixes.  {BUR-'ANG} appears at least once in ACIP files and {DGA'-'AM} appears at least twice, but these may be typos.  The converter does allow it, though.  It thinks {BIR'U} and {WAN'U} (which also occur, but only very rarely) are both non-native, though, and thus treats {'} as U+0F71 (subscribed) and not U+0F60 (full form) in each case.)

Note a fine point.  When turning a tsheg bar into Tibetan, the ACIP->Tibetan converters assume that subjoined YA and RA consonants are not fixed-form -- not U+0FBB and U+0FBC -- but rather are the usual subjoined forms U+0FB1 and U+0FB2.  The only exceptions are the stacks R+Y, Y+Y, and n+d+Y, which are known to have fixed-form subjoined YA, and the stacks n+d+R+Y (where RA but not YA is full-form) and K+sh+R, which are known to have fixed-form subjoined RA.  (Wa-zur, U+0FAD, is never confused with full-form subjoined WA, U+0FBA, because ACIP represents the former with {V} and the latter with {W}.)  Furthermore, the converter only generates U+0F6A, the fixed-form RA (rango), for the stacks R+W, R+Y, R+SH, R+SH+Y, R+sh, R+sh+n, R+sh+n+Y, R+sh+M, R+sh+Y, and R+S; U+0F62 is always produced for the top-most RA in any other stack.  (Note that U+0F62 is sometimes displayed as a fixed-form RA itself, as in {RNYA}.)

Another fine point: The tool treats {N+DZYA} like {N+DZ+YA}, except that it warns, "There is a stack of three or more consonants in N+DZYA that uses at least one '+' but does not use a '+' between each consonant.".  The tool is inconsistent, however; it will not treat {R+TS+NYA} like {R+T+S+N+YA} (and that would be a terrible idea).

So far, we have spoken about consonants and vowels.  In fact, it is not trivial to determine when something is a consonant and when it is a vowel.  {A} can represent U+0F68, the Tibetan letter, or the implicit vowel.  {'} can represent U+0F71, the subscribed a-chung, or U+0F60, the full-sized consonant a-chung.  The converter treats {TAA} as {T+AA}, not {TA-AA}, but treats {TAAA} like {TA-AA}, not {T+AA-A}.  It treats {PA'AM} like {PA-'A-M}, not {P+A'A-M}.  In short, it first tries out treating {'} and {A} like vowels, but will backtrack if that leads to a clearly invalid tsheg bar.

Finally, a string of numbers can be a tsheg bar also.  It is illegal for numbers and consonants to appear together within one tsheg bar, however.

The above is the complete understanding of the converter's algorithms for parsing tsheg bars.  You the native Tibetan speaker may know that {BSKYABS-'UR-'UNG-'O} is not allowed and thus think that {B+S+K+YAB+S-'UR-'UNG-'O} should be the result, but the converter has no such knowledge, and thinks this is a native tsheg bar equivalent to {B-S+K+YAB-S-'UR-'UNG-'O}.

System Properties

The tsheg-bar substitution mechanism is customizable via system properties.  Java developers likely know what these are, but few users do.  This section will perhaps get a determined person started, but if you have trouble, contact the developers so that we can improve this documentation or create a better user interface.

For the tool to respect the value of a system property, you must invoke the tool from the command line as follows:

java "-Dorg.thdl.tib.text.ttt.ReplacementMap=KAsh=>K+sh,ONYA=>[#ERROR-ONYA-IS-O&]" -Dorg.thdl.tib.text.ttt.VerboseReplacementMap=true -jar Jskad.jar

Known Bugs

This section presents areas where the current tool's behavior is wrong.  Before doing serious work with the converter, familiarize yourself with this section and develop a plan to work around the bugs or to ensure that your documents will not trigger the bugs.  At the same time, if any of these bugs affects you, contact the developers so that we can fix them.  The squeaky wheel surely gets the grease; these bugs may never be fixed if there are no complaints.

The following are all known bugs:

  • When ACIP {MTHARo} is given, the {o} glyph should be centered under the THA glyph in ACIP->TMW conversions.  At present, the {o} glyph appears underneath the rightmost stack.  Similarly, {\u0F35} and {\u0F37} are not centered properly.  [838594]
  • ACIP->TMW conversion for {\u0F3E} is not correct.  Fear not; the character U+0F3E is so rare that no ACIP transliteration exists for it.  [855478]
  • A folio marker {@0B1} can appear in ACIP Release V texts; it gives an error at present.
  • The treatment of the very last line in an input text may be buggy with regard to treatment of ACIP spaces, etc.
  • The treatment of {:} directly before a line break is likely incorrect; a tsheg is inserted right now after the visarga.
  • The number of errors after which processing is aborted (under the assumption that the input is probably not ACIP) is absolute, not a per capita measurement (i.e., one error per 100 characters of input) and is not easily configured.

Room for Improvement

This section presents areas where the current tool could be improved.  None of the current behavior described here is incontrovertibly flawed (i.e., there are no bugs described here, see known bugs for that); current behavior is technically correct.  However, the current behavior is not, in everyone's eyes, perfect.

The following are the current areas in which the tool could be better:

  • The Unicode U+0F43 is equivalent to the sequence U+0F42 followed by U+0FB7.  There are several distinct but similar cases.  The converter should have an option that allows for producing one form or the other instead.  (In practice, doing Unicode normalization on the output is probably going to give you results just as good, and having a separate normalizer facilitates code reuse.)  See issue 946043.
  • The fact that stacks G+N+Y and M+N+Y exist in the TMW font means that the ACIP snippets {GNY} and {MNY} should, in some cases, trigger warning 512.  They do not do so at present, and warning 507 is not given either.  See issue 946058.
  • At present, an error in ACIP->TMW conversion is given when ACIP like {RTSNY} or {NNY} is seen; this is because no glyph R+TS+NY or N+NY is in the TMW font.  For these snippets, though, there is a significant possibility that R+T+S+N+Y or N+N+Y was intended because TMW does have glyphs for both.  It would be best to give an error or stern warning when creating the Unicode for RTSNY etc., and to allow for giving an error when creating TMW for RTSNY etc. (whereas right now warning 512 is given).  See issue 936998.
  • It would be best to produce a warning (but not an error) when converting R+W, Y+Y, n+d+R+Y, K+sh+R, n+d+Y, R+Y, R+SH, R+SH+Y, R+sh, R+sh+n, R+sh+n+Y, R+sh+M, R+sh+Y, or R+S to Tibetan.  The ACIP scheme does not really say when the unusual, full-formed consonant is intended.  See issue 933240.
  • Some warnings are false alarms (false positives).  The only known false positive is the warning [#WARNING CONVERTING ACIP DOCUMENT: There is a chance that the ACIP KshA was intended to represent more consonants than we parsed it as representing -- NNYA, e.g., means N+NYA, but you can imagine seeing N+N+YA and typing NNYA for it too.] when it is given for ACIP constructions that cannot possibly be interpreted any other way.  See issue 932896.
  • The only time Unicode output will contain U+0F6A, full-formed RA, instead of U+0F62, RA (possibly superscribed), is when a Unicode escape sequence for U+0F6A is used or when one of the ten stacks RY, RW, RSH, RSHY, Rsh, Rshn, RshnY, RshM, RshY, or RS appears.  The Unicode standard is not as clear as it could be on the issue of when to use full-formed code points like U+0F6A, so this treatment might not be the best.
  • The glyph TibetanMachineWeb9.61 -- the {O'I} special combination (i.e., the glyph for the Unicode string U+0F7C,U+0F60,U+0F72) -- is never output by the ACIP->TMW converter.  It is sometimes more beautiful than the glyphs that are presently output (three separate glyphs instead of the one).
  • Though the ACIP standard disallows it, you will find in ACIP documents from the Buddhist Canon things like {/NYA\} where the standard demands {/NYA/}.  Presently, this triggers an error; it would be better if this were converted like {/NYA/} is, and triggered only a Most-level warning.
  • The hypothetical comment {[# \u0F40 may have been intended...]} should cause a warning saying that Unicode escapes do not apply within comments.
  • The whitespace after a Unicode escape is not interpreted correctly when that Unicode escape represents something that is part of a tsheg bar.  For example, the space in {KA KHA} is treated as a tsheg (i.e., U+0F0B), but the space in {\u0F40 KHA} is wrongly treated as Tibetan whitespace.  [855482]
  • Though not standard, {:} and {:-} sometimes are intended to represent U+0F14.  The latter causes an error; it should cause a warning suggested that the Unicode escape {\u0F14} be used instead.  The former is always treated as U+0F7F; it should cause a warning in some or all contexts.
  • The tsheg-bar substitution mechanism should be more general.  The useful rule ONYA=>O& should be supported and used by default.
  • The converters should support a white list of acceptable non-native tsheg bars (where the term "tsheg bar" is to be interpreted somewhat literally here as any characters between punctuation).  Non-native tsheg bars not on the list should produce warnings or errors.  Similarly, but perhaps less urgently, a syllabary of native tsheg bars should be supported too.  (A workaround is to use coloring, have your word processor delete everything but the colored text, sort the colored tsheg bars, and inspect them all by hand.  Also, tsheg-bar statistics will help you to find uncommon tsheg bars.)
  • ACIP->Unicode conversions produce Unicode text files at present.  While more compact than Rich Text Format (RTF) files, a text file does not allow for supporting the two font sizes in {KA (KA)}.  A workaround is to use an ACIP->TMW conversion followed by a separate TMW->Unicode conversion.
  • The converter should warn for each occurrence of the vowels {'E}, {'O}, {'EE}, or {'OO}.
  • Default substitution rules should handle {KAsh}, which seems to always mean {K+sh} in ACIP Release V texts.

License

Both the ACIP->Tibetan converters and this document are released under the THDL Open Community License Version 1.0.

Please e-mail us your comments about this page.

The THDL Tools project is generously hosted by: SourceForge Logo


Loading...