PHP: token_get_all - Manual

downloads | documentation | faq | getting help | mailing lists | reporting bugs | php.net sites | links | conferences | my php.net

search for in the

token_name Tokenizer

Last updated: Sat, 27 Jan 2007

view this page in

# token_get_all

(PHP 4 >= 4.2.0, PHP 5)

token_get_all -- Split given source into PHP tokens

## Описание

array token_get_all ( string source )

token_get_all() parses the given `source` string into PHP language tokens using the Zend engine's lexical scanner.

For a list of parser tokens, see Прил. Q, or use token_name() to translate a token value into its string representation.

## Список параметров

`source`

  The PHP source to parse.

## Возвращаемые значения

An array of token identifiers. Each individual token identifier is either a single character (i.e.: `;`, `.`, `>`, `!`, etc...), or a two element array containing the token index in element 0, and the string content of the original token in element 1.

## Примеры

Пример 1. token_get_all() examples

`<?php
$tokens = token_get_all('<?php echo; ?>'); /* => array(
array(T_OPEN_TAG, '<?php'),
array(T_ECHO, 'echo'),
';',
array(T_CLOSE_TAG, '?>') ); */
/* Note in the following example that the string is parsed as T_INLINE_HTML
rather than the otherwise expected T_COMMENT (T_ML_COMMENT in PHP <5).
This is because no open/close tags were used in the "code" provided.
This would be equivalent to putting a comment outside of <?php ?> tags in a normal file. */
$tokens = token_get_all('/* comment */'); // => array(array(T_INLINE_HTML, '/* comment */'));
?>`

add a note User Contributed Notes
token_get_all
phpcomments at majiclab dot com
01-Aug-2005 10:08
`Regarding bertrand at toggg dot com's comment: there is another case of the { } curly braces being used in PHP, but the token_get_all() function treats it just like a code block: string index. Example:
<?php
$text = "Hello";
if ($text{ 0 } == 'H') {
echo "This example uses { for both a PHP block and a string index.";
}
?>
Just in case some people were wondering. Since PHP treats them as the same token, it makes some things a little more interesting for parsing. You can't just assume that { ... } is a code block, it could just be a number referring to an index of a string.`
bertrand at toggg dot com
07-Mar-2005 10:41
`If you want to retrieve the PHP blocks then you will count up the opening curly braces '{' and down the closing ones '}' (counter zero means block finished)
CAUTION: the opening curly braces token can take 3 values:
1) '{' for all PHP code blocks,
2) T_CURLY_OPEN for "protected" variables within strings as "{$var}"
3) T_DOLLAR_OPEN_CURLY_BRACES for extended format "${var}"
On the other hand, closing token is allways '}' !
So counting up must take place on the 3 tokens:
'{' , T_CURLY_OPEN and T_DOLLAR_OPEN_CURLY_BRACES
Have fun with PHP tokenizer !`
bishop
07-Dec-2004 10:58
`You may want to know the line and column number at which a token begins (or ends). Since this tokenizer interface doesn't provide that information, you have to track it manually, like below:
<?php
function update_line_and_column_positions($c, &$line, &$col)
{
// update line count
$numNewLines = substr_count($c, "\n");
if (1 <= $numNewLines) {
// have new lines, add them in
$line += $numNewLines;
$col = 1;
// skip to right past the last new line, as it won't affect the column position
$c = substr($c, strrpos($c, "\n") + 1);
if ($c === false) {
$c = '';
}
}
// update column count
$col += strlen($c);
}
?>
Now use it, something like:
<?php
$line = 1;
$col = 1;
foreach ($tokens as $token) {
if (is_array($token)) {
list ($token, $text) = $token;
} else if (is_string($token)) {
$text = $token;
}
update_line_and_column_positions($text, $line, $col);
}
?>
Note this assumes that your desired coordinate system is 1-based (eg (1,1) is the upper left). Zero-based is left as an exercise for the reader.`
Leon Atkinson
06-Dec-2002 03:17
`This function parses PHP code. Here's an example of it's use.
<?
$code = '<?$a = 3;?>';
foreach(token_get_all($code) as $c)
{
if(is_array($c))
{
print(token_name($c[0]) . ": '" . htmlentities($c[1]) . "'\n");
}
else
{
print("$c\n");
}
}
?>`
add a note
token_name Tokenizer

Last updated: Sat, 27 Jan 2007
show source | credits | stats | sitemap | contact | advertising | mirror sites
Copyright © 2001-2007 The PHP Group
All rights reserved.
This mirror generously provided by: Yahoo! Inc.
Last updated: Sat Mar 10 11:33:19 2007 PST