C extensin for Variable#lax_parse

If we are trying to get the struct from something other than self, then we
should make sure to check the class of the object.  This util functions
make this easier.

.gitignore

@@ -5,3 +5,6 @@ pkg
 *.rbc
 .rvmrc
 .ruby-version
+*.bundle
+/tmp
+Gemfile.lock

.travis.yml

@@ -1,11 +1,14 @@
 rvm:
-  - 1.8.7
   - 1.9.3
-  - ree
-  - jruby-18mode
+  - 2.0.0
+  - 2.1.0
   - jruby-19mode
-  - rbx-18mode
+  - jruby-head
   - rbx-19mode
+matrix:
+  allow_failures:
+    - rvm: rbx-19mode
+    - rvm: jruby-head
 
 script: "rake test"
 

Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

History.md

@@ -1,18 +1,33 @@
 # Liquid Version History
 
-IMPORTANT: Liquid 2.6 is going to be the last version of Liquid which maintains explicit Ruby 1.8 compatability.
-The following releases will only be tested against Ruby 1.9 and Ruby 2.0 and are likely to break on Ruby 1.8.
-
-## 2.6.0 / Master branch (not yet released)
+## 3.0.0 / not yet released / branch "master"
 
 * ...
+* Add a to_s default for liquid drops, see #306 [Adam Doeler, releod]
+* Add strip, lstrip, and rstrip to standard filters [Florian Weingarten, fw42]
+* Make if, for & case tags return complete and consistent nodelists, see #250 [Nick Jones, dntj]
+* Prevent arbitrary method invocation on condition objects, see #274 [Dylan Thacker-Smith, dylanahsmith]
+* Don't call to_sym when creating conditions for security reasons, see #273 [Bouke van der Bijl, bouk]
+* Fix resource counting bug with respond_to?(:length), see #263 [Florian Weingarten, fw42]
+* Allow specifying custom patterns for template filenames, see #284 [Andrei Gladkyi, agladkyi]
+* Allow drops to optimize loading a slice of elements, see #282 [Tom Burns, boourns]
+* Support for passing variables to snippets in subdirs, see #271 [Joost Hietbrink, joost]
+* Add a class cache to avoid runtime extend calls, see #249 [James Tucker, raggi]
+* Remove some legacy Ruby 1.8 compatibility code, see #276 [Florian Weingarten, fw42]
+* Add default filter to standard filters, see #267 [Derrick Reimer, djreimer]
 * Add optional strict parsing and warn parsing, see #235 [Tristan Hume, trishume]
 * Add I18n syntax error translation, see #241 [Simon Hørup Eskildsen, Sirupsen]
 * Make sort filter work on enumerable drops, see #239 [Florian Weingarten, fw42]
 * Fix clashing method names in enumerable drops, see #238 [Florian Weingarten, fw42]
 * Make map filter work on enumerable drops, see #233 [Florian Weingarten, fw42]
-* Fix security issue with map filter, see #230, #232, #234, #237 [Florian Weingarten, fw42]
 * Improved whitespace stripping for blank blocks, related to #216 [Florian Weingarten, fw42]
+* Raise `Liquid::ArgumentError` instead of `::ArgumentError` when filter has wrong number of arguments #309 [Bogdan Gusiev, bogdan]
+
+## 2.6.0 / 2013-11-25 / branch "2.6-stable"
+
+IMPORTANT: Liquid 2.6 is going to be the last version of Liquid which maintains explicit Ruby 1.8 compatability.
+The following releases will only be tested against Ruby 1.9 and Ruby 2.0 and are likely to break on Ruby 1.8.
+
 * Bugfix for #106: fix example servlet [gnowoel]
 * Bugfix for #97: strip_html filter supports multi-line tags [Jo Liss, joliss]
 * Bugfix for #114: strip_html filter supports style tags [James Allardice, jamesallardice]
@@ -21,6 +36,7 @@ The following releases will only be tested against Ruby 1.9 and Ruby 2.0 and are
 * Bugfix for #204: 'raw' parsing bug [Florian Weingarten, fw42]
 * Bugfix for #150: 'for' parsing bug [Peter Schröder, phoet]
 * Bugfix for #126: Strip CRLF in strip_newline [Peter Schröder, phoet]
+* Bugfix for #174, "can't convert Fixnum into String" for "replace" [wǒ_is神仙, jsw0528]
 * Allow a Liquid::Drop to be passed into Template#render [Daniel Huckstep, darkhelmet]
 * Resource limits [Florian Weingarten, fw42]
 * Add reverse filter [Jay Strybis, unreal]
@@ -31,6 +47,21 @@ The following releases will only be tested against Ruby 1.9 and Ruby 2.0 and are
 * Better documentation for 'include' tag (closes #163) [Peter Schröder, phoet]
 * Use of BigDecimal on filters to have better precision (closes #155) [Arthur Nogueira Neves, arthurnn]
 
+## 2.5.4 / 2013-11-11 / branch "2.5-stable"
+
+* Fix "can't convert Fixnum into String" for "replace", see #173, [wǒ_is神仙, jsw0528]
+
+## 2.5.3 / 2013-10-09
+
+* #232, #234, #237: Fix map filter bugs [Florian Weingarten, fw42]
+
+## 2.5.2 / 2013-09-03 / deleted
+
+Yanked from rubygems, as it contained too many changes that broke compatibility. Those changes will be on following major releases.
+
+## 2.5.1 / 2013-07-24
+
+* #230: Fix security issue with map filter, Use invoke_drop in map filter [Florian Weingarten, fw42]
 
 ## 2.5.0 / 2013-03-06
 

README.md

@@ -1,9 +1,9 @@
+[![Build Status](https://secure.travis-ci.org/Shopify/liquid.png?branch=master)](http://travis-ci.org/Shopify/liquid)
 # Liquid template engine
 
 * [Contributing guidelines](CONTRIBUTING.md)
 * [Version history](History.md)
 * [Liquid documentation from Shopify](http://docs.shopify.com/themes/liquid-basics)
-* [Liquid Wiki from Shopify](http://wiki.shopify.com/Liquid)
 * [Liquid Wiki at GitHub](https://github.com/Shopify/liquid/wiki)
 * [Website](http://liquidmarkup.org/)
 
@@ -71,5 +71,3 @@ This is useful for doing things like enabling strict mode only in the theme edit
 
 It is recommended that you enable `:strict` or `:warn` mode on new apps to stop invalid templates from being created.
 It is also recommended that you use it in the template editors of existing apps to give editors better error messages.
-
-[![Build Status](https://secure.travis-ci.org/Shopify/liquid.png)](http://travis-ci.org/Shopify/liquid)

Rakefile

@@ -1,9 +1,8 @@
-#!/usr/bin/env ruby
-
-require 'rubygems'
 require 'rake'
 require 'rake/testtask'
-require 'rubygems/package_task'
+require 'rake/extensiontask'
+$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
+require "liquid/version"
 
 task :default => 'test'
 
@@ -29,14 +28,20 @@ task :test do
   Rake::Task['base_test'].invoke
 end
 
-gemspec = eval(File.read('liquid.gemspec'))
-Gem::PackageTask.new(gemspec) do |pkg|
-  pkg.gem_spec = gemspec
+task :gem => :build
+task :build do
+  system "gem build liquid.gemspec"
 end
 
-desc "Build the gem and release it to rubygems.org"
-task :release => :gem do
-  sh "gem push pkg/liquid-#{gemspec.version}.gem"
+task :install => :build do
+  system "gem install liquid-#{Liquid::VERSION}.gem"
+end
+
+task :release => :build do
+  system "git tag -a v#{Liquid::VERSION} -m 'Tagging #{Liquid::VERSION}'"
+  system "git push --tags"
+  system "gem push liquid-#{Liquid::VERSION}.gem"
+  system "rm liquid-#{Liquid::VERSION}.gem"
 end
 
 namespace :benchmark do
@@ -60,6 +65,10 @@ namespace :profile do
     ruby "./performance/profile.rb"
   end
 
+  task :stackprof do
+    ruby "./performance/stackprof.rb"
+  end
+
   desc "Run KCacheGrind"
   task :grind => :run  do
     system "qcachegrind /tmp/liquid.rubyprof_calltreeprinter.txt"
@@ -71,3 +80,8 @@ desc "Run example"
 task :example do
   ruby "-w -d -Ilib example/server/server.rb"
 end
+
+Rake::ExtensionTask.new "liquid" do |ext|
+  ext.lib_dir = "lib/liquid"
+end
+Rake::Task[:test].prerequisites << :compile

example/server/liquid_servlet.rb

@@ -13,7 +13,7 @@ class LiquidServlet < WEBrick::HTTPServlet::AbstractServlet
   def handle(type, req, res)
     @request, @response = req, res
 
-    @request.path_info =~ /(\w+)$/
+    @request.path_info =~ /(\w+)\z/
     @action = $1 || 'index'
     @assigns = send(@action) if respond_to?(@action)
 

ext/liquid/block.c

@@ -0,0 +1,168 @@
+#include "liquid_ext.h"
+
+VALUE cLiquidBlock;
+ID intern_assert_missing_delimitation, intern_block_delimiter, intern_is_blank, intern_new,
+   intern_new_with_options, intern_tags, intern_unknown_tag, intern_unterminated_tag,
+   intern_unterminated_variable;
+
+struct liquid_tag
+{
+    char *name, *markup;
+    long name_length, markup_length;
+};
+
+static bool parse_tag(struct liquid_tag *tag, char *token, long token_length)
+{
+    // Strip {{ and }} braces
+    token += 2;
+    token_length -= 4;
+
+    char *end = token + token_length;
+    while (token < end && isspace(*token))
+        token++;
+    tag->name = token;
+
+    char c = *token;
+    while (token < end && (isalnum(c) || c == '_'))
+        c = *(++token);
+    tag->name_length = token - tag->name;
+    if (!tag->name_length) {
+        memset(tag, 0, sizeof(*tag));
+        return false;
+    }
+
+    while (token < end && isspace(*token))
+        token++;
+    tag->markup = token;
+
+    char *last = end - 1;
+    while (token < last && isspace(*last))
+        last--;
+    end = last + 1;
+    tag->markup_length = end - token;
+    return true;
+}
+
+static VALUE rb_parse_body(VALUE self, VALUE tokenizerObj)
+{
+    struct liquid_tokenizer *tokenizer = LIQUID_TOKENIZER_GET_STRUCT(tokenizerObj);
+
+    bool blank = true;
+    VALUE nodelist = rb_iv_get(self, "@nodelist");
+    if (nodelist == Qnil) {
+        nodelist = rb_ary_new();
+        rb_iv_set(self, "@nodelist", nodelist);
+    } else {
+        rb_ary_clear(nodelist);
+    }
+
+    struct token token;
+    while (true) {
+        liquid_tokenizer_next(tokenizer, &token);
+        switch (token.type) {
+        case TOKEN_NONE:
+            /*
+             * Make sure that it's ok to end parsing in the current block.
+             * Effectively this method will throw an exception unless the current block is
+             * of type Document
+             */
+            rb_funcall(self, intern_assert_missing_delimitation, 0);
+            goto done;
+        case TOKEN_INVALID:
+        {
+            VALUE token_obj = rb_str_new(token.str, token.length);
+            if (token.str[1] == '%')
+                rb_funcall(self, intern_unterminated_tag, 1, token_obj);
+            else
+                rb_funcall(self, intern_unterminated_variable, 1, token_obj);
+            break;
+        }
+        case TOKEN_TAG:
+        {
+            struct liquid_tag tag;
+            if (!parse_tag(&tag, token.str, token.length)) {
+                // FIXME: provide more appropriate error message
+                rb_funcall(self, intern_unterminated_tag, 1, rb_str_new(token.str, token.length));
+            } else {
+                if (tag.name_length >= 3 && !memcmp(tag.name, "end", 3)) {
+                    VALUE block_delimiter = rb_funcall(self, intern_block_delimiter, 0);
+                    if (TYPE(block_delimiter) == T_STRING &&
+                        tag.name_length == RSTRING_LEN(block_delimiter) &&
+                        !memcmp(tag.name, RSTRING_PTR(block_delimiter), tag.name_length))
+                    {
+                        goto done;
+                    }
+                }
+
+                VALUE tags = rb_funcall(cLiquidTemplate, intern_tags, 0);
+                Check_Type(tags, T_HASH);
+                VALUE tag_name = rb_str_new(tag.name, tag.name_length);
+                VALUE tag_class = rb_hash_lookup(tags, tag_name);
+                VALUE markup = rb_str_new(tag.markup, tag.markup_length);
+                if (tag_class != Qnil) {
+                    VALUE options = rb_iv_get(self, "@options");
+                    if (options == Qnil)
+                        options = rb_hash_new();
+                    VALUE new_tag = rb_funcall(tag_class, intern_new_with_options, 4,
+                                               tag_name, markup, tokenizerObj, options);
+                    if (blank) {
+                        VALUE blank_block = rb_funcall(new_tag, intern_is_blank, 0);
+                        if (blank_block == Qnil || blank_block == Qfalse)
+                            blank = false;
+                    }
+                    rb_ary_push(nodelist, new_tag);
+                } else {
+                    rb_funcall(self, intern_unknown_tag, 3, tag_name, markup, tokenizerObj);
+                    /*
+                     * multi-block tags may store the nodelist in a block array on unknown_tag
+                     * then replace @nodelist with a new array. We need to use the new array
+                     * for the block following the tag token.
+                     */
+                    nodelist = rb_iv_get(self, "@nodelist");
+                }
+            }
+            break;
+        }
+        case TOKEN_VARIABLE:
+        {
+            VALUE markup = rb_str_new(token.str + 2, token.length - 4);
+            VALUE options = rb_iv_get(self, "@options");
+            VALUE new_var = rb_funcall(cLiquidVariable, intern_new, 2, markup, options);
+            rb_ary_push(nodelist, new_var);
+            blank = false;
+            break;
+        }
+        case TOKEN_STRING:
+            rb_ary_push(nodelist, rb_str_new(token.str, token.length));
+            if (blank) {
+                int i;
+                for (i = 0; i < token.length; i++) {
+                    if (!isspace(token.str[i])) {
+                        blank = false;
+                        break;
+                    }
+                }
+            }
+            break;
+        }
+    }
+done:
+    rb_iv_set(self, "@blank", blank ? Qtrue : Qfalse);
+    return Qnil;
+}
+
+void init_liquid_block()
+{
+    intern_assert_missing_delimitation = rb_intern("assert_missing_delimitation!");
+    intern_block_delimiter = rb_intern("block_delimiter");
+    intern_is_blank = rb_intern("blank?");
+    intern_new = rb_intern("new");
+    intern_new_with_options = rb_intern("new_with_options");
+    intern_tags = rb_intern("tags");
+    intern_unknown_tag = rb_intern("unknown_tag");
+    intern_unterminated_tag = rb_intern("unterminated_tag");
+    intern_unterminated_variable = rb_intern("unterminated_variable");
+
+    cLiquidBlock = rb_define_class_under(mLiquid, "Block", cLiquidTag);
+    rb_define_method(cLiquidBlock, "parse_body", rb_parse_body, 1);
+}

ext/liquid/block.h

@@ -0,0 +1,8 @@
+#ifndef LIQUID_BLOCK_H
+#define LIQUID_BLOCK_H
+
+void init_liquid_block();
+
+extern VALUE cLiquidBlock;
+
+#endif

ext/liquid/extconf.rb

@@ -0,0 +1,3 @@
+require 'mkmf'
+$CFLAGS << ' -Wall'
+create_makefile("liquid/liquid")

ext/liquid/liquid_ext.c

@@ -0,0 +1,16 @@
+#include "liquid_ext.h"
+
+VALUE mLiquid;
+VALUE cLiquidTemplate, cLiquidTag, cLiquidVariable;
+
+void Init_liquid(void)
+{
+    mLiquid = rb_define_module("Liquid");
+    cLiquidTemplate = rb_define_class_under(mLiquid, "Template", rb_cObject);
+    cLiquidTag = rb_define_class_under(mLiquid, "Tag", rb_cObject);
+    cLiquidVariable = rb_define_class_under(mLiquid, "Variable", rb_cObject);
+
+    init_liquid_tokenizer();
+    init_liquid_block();
+    init_liquid_variable();
+}

ext/liquid/liquid_ext.h

@@ -0,0 +1,16 @@
+#ifndef LIQUID_EXT_H
+#define LIQUID_EXT_H
+
+#include <stdbool.h>
+#include <ctype.h>
+#include <ruby.h>
+
+#include "tokenizer.h"
+#include "block.h"
+#include "utils.h"
+#include "variable.h"
+
+extern VALUE mLiquid;
+extern VALUE cLiquidTemplate, cLiquidTag, cLiquidVariable;
+
+#endif

ext/liquid/tokenizer.c

@@ -0,0 +1,113 @@
+#include "liquid_ext.h"
+
+VALUE cLiquidTokenizer;
+
+static void free_tokenizer(void *ptr)
+{
+    struct liquid_tokenizer *tokenizer = ptr;
+    xfree(tokenizer);
+}
+
+static VALUE rb_allocate(VALUE klass)
+{
+    VALUE obj;
+    struct liquid_tokenizer *tokenizer;
+
+    obj = Data_Make_Struct(klass, struct liquid_tokenizer, NULL, free_tokenizer, tokenizer);
+    return obj;
+}
+
+static VALUE rb_initialize(VALUE self, VALUE source)
+{
+    struct liquid_tokenizer *tokenizer;
+
+    Check_Type(source, T_STRING);
+    Data_Get_Struct(self, struct liquid_tokenizer, tokenizer);
+    tokenizer->cursor = RSTRING_PTR(source);
+    tokenizer->length = RSTRING_LEN(source);
+    return Qnil;
+}
+
+void liquid_tokenizer_next(struct liquid_tokenizer *tokenizer, struct token *token)
+{
+    if (tokenizer->length <= 0) {
+        memset(token, 0, sizeof(*token));
+        return;
+    }
+    token->type = TOKEN_STRING;
+
+    char *cursor = tokenizer->cursor;
+    char *last = tokenizer->cursor + tokenizer->length - 1;
+
+    while (cursor < last) {
+        if (*cursor++ != '{')
+            continue;
+
+        char c = *cursor++;
+        if (c != '%' && c != '{')
+            continue;
+        if (cursor - tokenizer->cursor > 2) {
+            token->type = TOKEN_STRING;
+            cursor -= 2;
+            goto found;
+        }
+        char *incomplete_end = cursor;
+        token->type = TOKEN_INVALID;
+        if (c == '%') {
+            while (cursor < last) {
+                if (*cursor++ != '%')
+                    continue;
+                c = *cursor++;
+                while (c == '%' && cursor <= last)
+                    c = *cursor++;
+                if (c != '}')
+                    continue;
+                token->type = TOKEN_TAG;
+                goto found;
+            }
+            cursor = incomplete_end;
+            goto found;
+        } else {
+            while (cursor < last) {
+                if (*cursor++ != '}')
+                    continue;
+                if (*cursor++ != '}') {
+                    incomplete_end = cursor - 1;
+                    continue;
+                }
+                token->type = TOKEN_VARIABLE;
+                goto found;
+            }
+            cursor = incomplete_end;
+            goto found;
+        }
+    }
+    cursor = last + 1;
+found:
+    token->str = tokenizer->cursor;
+    token->length = cursor - tokenizer->cursor;
+    tokenizer->cursor += token->length;
+    tokenizer->length -= token->length;
+}
+
+static VALUE rb_next(VALUE self)
+{
+    struct liquid_tokenizer *tokenizer;
+    Data_Get_Struct(self, struct liquid_tokenizer, tokenizer);
+
+    struct token token;
+    liquid_tokenizer_next(tokenizer, &token);
+    if (token.type == TOKEN_NONE)
+        return Qnil;
+
+    return rb_str_new(token.str, token.length);
+}
+
+void init_liquid_tokenizer()
+{
+    cLiquidTokenizer = rb_define_class_under(mLiquid, "Tokenizer", rb_cObject);
+    rb_define_alloc_func(cLiquidTokenizer, rb_allocate);
+    rb_define_method(cLiquidTokenizer, "initialize", rb_initialize, 1);
+    rb_define_method(cLiquidTokenizer, "next", rb_next, 0);
+    rb_define_alias(cLiquidTokenizer, "shift", "next");
+}

ext/liquid/tokenizer.h

@@ -0,0 +1,30 @@
+#ifndef LIQUID_TOKENIZER_H
+#define LIQUID_TOKENIZER_H
+
+extern VALUE cLiquidTokenizer;
+
+enum token_type {
+    TOKEN_NONE,
+    TOKEN_INVALID,
+    TOKEN_STRING,
+    TOKEN_TAG,
+    TOKEN_VARIABLE
+};
+
+struct token {
+    enum token_type type;
+    char *str;
+    int length;
+};
+
+struct liquid_tokenizer {
+    char *cursor;
+    int length;
+};
+
+void init_liquid_tokenizer();
+void liquid_tokenizer_next(struct liquid_tokenizer *tokenizer, struct token *token);
+
+#define LIQUID_TOKENIZER_GET_STRUCT(obj) ((struct liquid_tokenizer *)obj_get_data_ptr(obj, cLiquidTokenizer))
+
+#endif

ext/liquid/utils.c

@@ -0,0 +1,21 @@
+#include <ruby.h>
+
+void raise_type_error(VALUE expected, VALUE got)
+{
+    rb_raise(rb_eTypeError, "wrong argument type %s (expected %s)",
+                             rb_class2name(got), rb_class2name(expected));
+}
+
+void check_class(VALUE obj, int type, VALUE klass)
+{
+    Check_Type(obj, type);
+    VALUE obj_klass = RBASIC_CLASS(obj);
+    if (obj_klass != klass)
+        raise_type_error(klass, obj_klass);
+}
+
+void *obj_get_data_ptr(VALUE obj, VALUE klass)
+{
+    check_class(obj, T_DATA, klass);
+    return DATA_PTR(obj);
+}

ext/liquid/utils.h

@@ -0,0 +1,8 @@
+#ifndef LIQUID_UTILS_H
+#define LIQUID_UTILS_H
+
+void raise_type_error(VALUE expected, VALUE got);
+void check_class(VALUE klass);
+void *obj_get_data_ptr(VALUE obj, VALUE klass);
+
+#endif

ext/liquid/variable.c

@@ -0,0 +1,179 @@
+#include "liquid_ext.h"
+
+VALUE cLiquidVariable;
+extern VALUE mLiquid;
+
+static void free_variable(void *ptr)
+{
+    struct liquid_variable *variable = ptr;
+    xfree(variable);
+}
+
+static VALUE rb_variable_allocate(VALUE klass)
+{
+    VALUE obj;
+    struct liquid_variable *variable;
+
+    obj = Data_Make_Struct(klass, struct liquid_variable, NULL, free_variable, variable);
+    return obj;
+}
+
+static inline int skip_whitespace(char * str, int len) 
+{
+  int skipped = 0; char * ptr = str;
+  while (skipped < len && isspace(*ptr))
+    {skipped++; ptr++;}
+  return skipped;
+}
+
+static char * get_quoted_fragment(char * str, int len, int * ret_size, int * end_offset, bool colon) 
+{
+  int p = 0; /* Current position in string */
+  int start = -1, end = -1; /* Start and end indices for the found string */
+  char quoted_by = -1; /* Is the current part of string quoted by a single or double quote? If so 
+                          ignore any special chars */
+
+  while (p < len) {
+    
+    switch (str[p]) {
+      case '"': 
+        if (start == -1) {start = p; quoted_by = '"';}
+        else if (str[start] == '"') {end = p; goto quoted_fragment_found;}
+        else if (quoted_by == -1) quoted_by = '"';
+        else if (quoted_by == '"') quoted_by = -1;
+        break;
+      case '\'':
+        if (start == -1) {start = p; quoted_by = '\'';}
+        else if (str[start] == '\'') {end = p; goto quoted_fragment_found;}
+        else if (quoted_by == -1) quoted_by = '\'';
+        else if (quoted_by == '\'') quoted_by = -1;
+        break;
+      case ':': 
+        if (colon)
+          if (start != -1 && quoted_by == -1) {end = p-1; goto quoted_fragment_found;} 
+        else
+          if (start == -1) start = p;
+        break;
+      case '|':
+      case ',':
+      case '\n':
+      case '\r':
+      case '\f':
+      case '\t':
+      case '\v':
+      case ' ': 
+        if (start != -1 && quoted_by == -1) {end = p-1; goto quoted_fragment_found;} 
+        break;
+      default: 
+        if (start == -1) start = p; 
+        break;
+    }
+    p++;
+  }
+  if (p == len && start != -1 && end == -1) end = len-1;
+
+quoted_fragment_found:
+  if (end >= start) {
+    *ret_size = end-start+1;
+    *end_offset = end+1;
+    return &str[start];
+  } else {
+    *ret_size = 0;
+    return NULL;
+  }
+}
+
+static VALUE get_filters(char * str, int len, VALUE self) {
+  VALUE filters_arr = rb_ary_new(); 
+
+  int p = 0; 
+  int ret_size, end_offset; 
+  char * f;
+
+  while(p<len) {
+    if (str[p] == '|') {
+      VALUE filter = rb_ary_new();
+      VALUE f_args = rb_ary_new();
+
+      p += skip_whitespace(&str[p+1], len-p-1);
+      f = get_quoted_fragment(&str[p], len-p, &ret_size, &end_offset, true);
+      p += end_offset;
+
+      if (f) {
+        if (f[ret_size-1] == ':') ret_size--;
+        rb_ary_push(filter, rb_str_new(f, ret_size));
+      }
+
+      /* Check for filter arguments */
+      do {
+        if (p<len) {
+          p += skip_whitespace(&str[p], len-p);
+
+          // printf("\n1. %.*s\n", len-p, &str[p]);
+
+          if (str[p] != '|') {
+            f = get_quoted_fragment(&str[p], len-p, &ret_size, &end_offset, false);
+
+            // printf("\n2. %.*s\n", ret_size, f);
+            
+            p += end_offset;
+            p += skip_whitespace(&str[p], len-p);
+
+            if (str[p] == '|') p--;
+
+            if (f) rb_ary_push(f_args, rb_str_new(f, ret_size));
+          
+          } else p--;
+        }
+
+      } while (str[p] == ',' || str[p] == ':');
+
+      rb_ary_push(filter, f_args);
+
+      /* Add to filters_arr array */
+      rb_ary_push(filters_arr, filter);
+    }
+    p++;
+  }
+  return filters_arr;
+}
+
+static VALUE rb_variable_lax_parse(VALUE self, VALUE m) 
+{
+  char * markup = RSTRING_PTR(m);
+  int markup_len = RSTRING_LEN(m);
+
+  char * cursor = markup; int cursor_pos = 0; 
+  VALUE filters_arr; 
+  int size, end_offset;
+
+  /* Extract name */
+  cursor_pos += skip_whitespace(markup, markup_len); 
+  cursor = markup + cursor_pos;
+  cursor = get_quoted_fragment(cursor, markup_len - cursor_pos, &size, &end_offset, false);
+
+  if (cursor == NULL) {
+    rb_iv_set(self, "@name", Qnil);
+    filters_arr = rb_ary_new(); 
+    rb_iv_set(self, "@filters", filters_arr);
+  }
+  else 
+  {
+    rb_iv_set(self, "@name", rb_str_new(cursor, size));
+
+    /* Extract filters */
+    if (end_offset < markup_len) {
+      cursor = &markup[end_offset];
+      filters_arr = get_filters(cursor, markup_len - end_offset, self);
+      rb_iv_set(self, "@filters", filters_arr);
+    }
+  }
+  return filters_arr;
+}
+
+void init_liquid_variable()
+{
+    cLiquidVariable = rb_define_class_under(mLiquid, "Variable", rb_cObject);
+    rb_define_alloc_func(cLiquidVariable, rb_variable_allocate);
+    rb_define_method(cLiquidVariable, "lax_parse", rb_variable_lax_parse, 1);
+}

ext/liquid/variable.h

@@ -0,0 +1,13 @@
+#ifndef LIQUID_VARIABLE_H
+#define LIQUID_VARIABLE_H
+
+#include <regex.h>
+
+struct liquid_variable {
+    char *markup; long markup_len;
+    char *name; long name_len;
+};
+
+void init_liquid_variable();
+
+#endif

lib/liquid.rb

@@ -30,21 +30,18 @@ module Liquid
   VariableSegment             = /[\w\-]/
   VariableStart               = /\{\{/
   VariableEnd                 = /\}\}/
-  VariableIncompleteEnd       = /\}\}?/
   QuotedString                = /"[^"]*"|'[^']*'/
   QuotedFragment              = /#{QuotedString}|(?:[^\s,\|'"]|#{QuotedString})+/o
   StrictQuotedFragment        = /"[^"]+"|'[^']+'|[^\s|:,]+/
   FirstFilterArgument         = /#{FilterArgumentSeparator}(?:#{StrictQuotedFragment})/o
   OtherFilterArgument         = /#{ArgumentSeparator}(?:#{StrictQuotedFragment})/o
-  SpacelessFilter             = /^(?:'[^']+'|"[^"]+"|[^'"])*#{FilterSeparator}(?:#{StrictQuotedFragment})(?:#{FirstFilterArgument}(?:#{OtherFilterArgument})*)?/o
+  SpacelessFilter             = /\A(?:'[^']+'|"[^"]+"|[^'"])*#{FilterSeparator}(?:#{StrictQuotedFragment})(?:#{FirstFilterArgument}(?:#{OtherFilterArgument})*)?/o
   Expression                  = /(?:#{QuotedFragment}(?:#{SpacelessFilter})*)/o
   TagAttributes               = /(\w+)\s*\:\s*(#{QuotedFragment})/o
-  AnyStartingTag              = /\{\{|\{\%/
-  PartialTemplateParser       = /#{TagStart}.*?#{TagEnd}|#{VariableStart}.*?#{VariableIncompleteEnd}/o
-  TemplateParser              = /(#{PartialTemplateParser}|#{AnyStartingTag})/o
   VariableParser              = /\[[^\]]+\]|#{VariableSegment}+\??/o
 end
 
+require 'liquid/liquid'
 require "liquid/version"
 require 'liquid/lexer'
 require 'liquid/parser'

lib/liquid/block.rb

@@ -1,82 +1,26 @@
 module Liquid
   class Block < Tag
-    IsTag             = /^#{TagStart}/o
-    IsVariable        = /^#{VariableStart}/o
-    FullToken         = /^#{TagStart}\s*(\w+)\s*(.*)?#{TagEnd}$/o
-    ContentOfVariable = /^#{VariableStart}(.*)#{VariableEnd}$/o
+    def initialize(tag_name, markup, tokens)
+      super
+      parse_body(tokens)
+    end
 
     def blank?
       @blank || false
     end
 
-    def parse(tokens)
-      @blank = true
-      @nodelist ||= []
-      @nodelist.clear
-
-      # All child tags of the current block.
-      @children = []
-
-      while token = tokens.shift
-        case token
-        when IsTag
-          if token =~ FullToken
-
-            # if we found the proper block delimiter just end parsing here and let the outer block
-            # proceed
-            if block_delimiter == $1
-              end_tag
-              return
-            end
-
-            # fetch the tag from registered blocks
-            if tag = Template.tags[$1]
-              new_tag = tag.new_with_options($1, $2, tokens, @options || {})
-              @blank &&= new_tag.blank?
-              @nodelist << new_tag
-              @children << new_tag
-            else
-              # this tag is not registered with the system
-              # pass it to the current block for special handling or error reporting
-              unknown_tag($1, $2, tokens)
-            end
-          else
-            raise SyntaxError.new(options[:locale].t("errors.syntax.tag_termination", :token => token, :tag_end => TagEnd.inspect))
-          end
-        when IsVariable
-          new_var = create_variable(token)
-          @nodelist << new_var
-          @children << new_var
-          @blank = false
-        when ''
-          # pass
-        else
-          @nodelist << token
-          @blank &&= (token =~ /\A\s*\z/)
-        end
-      end
-
-      # Make sure that it's ok to end parsing in the current block.
-      # Effectively this method will throw an exception unless the current block is
-      # of type Document
-      assert_missing_delimitation!
-    end
-
     # warnings of this block and all sub-tags
     def warnings
       all_warnings = []
       all_warnings.concat(@warnings) if @warnings
 
-      (@children || []).each do |node|
-        all_warnings.concat(node.warnings || [])
+      (nodelist || []).each do |node|
+        all_warnings.concat(node.warnings || []) if node.respond_to?(:warnings)
       end
 
       all_warnings
     end
 
-    def end_tag
-    end
-
     def unknown_tag(tag, params, tokens)
       case tag
       when 'else'
@@ -99,19 +43,20 @@ module Liquid
       @tag_name
     end
 
-    def create_variable(token)
-      token.scan(ContentOfVariable) do |content|
-        return Variable.new(content.first, @options)
-      end
-      raise SyntaxError.new(options[:locale].t("errors.syntax.variable_termination", :token => token, :tag_end => VariableEnd.inspect))
-    end
-
     def render(context)
       render_all(@nodelist, context)
     end
 
     protected
 
+    def unterminated_variable(token)
+      raise SyntaxError.new(options[:locale].t("errors.syntax.variable_termination", :token => token, :tag_end => VariableEnd.inspect))
+    end
+
+    def unterminated_tag(token)
+      raise SyntaxError.new(options[:locale].t("errors.syntax.tag_termination", :token => token, :tag_end => TagEnd.inspect))
+    end
+
     def assert_missing_delimitation!
       raise SyntaxError.new(options[:locale].t("errors.syntax.tag_never_closed", :block_name => block_name))
     end
@@ -135,7 +80,7 @@ module Liquid
           end
 
           token_output = (token.respond_to?(:render) ? token.render(context) : token)
-          context.resource_limits[:render_length_current] += (token_output.respond_to?(:length) ? token_output.length : 1)
+          context.increment_used_resources(:render_length_current, token_output)
           if context.resource_limits_reached?
             context.resource_limits[:reached] = true
             raise MemoryError.new("Memory limits exceeded")

lib/liquid/context.rb

@@ -25,6 +25,15 @@ module Liquid
       squash_instance_assigns_with_environments
 
       @interrupts = []
+      @filters = []
+    end
+
+    def increment_used_resources(key, obj)
+      @resource_limits[key] += if obj.kind_of?(String) || obj.kind_of?(Array) || obj.kind_of?(Hash)
+        obj.length
+      else
+        1
+      end
     end
 
     def resource_limits_reached?
@@ -34,7 +43,7 @@ module Liquid
     end
 
     def strainer
-      @strainer ||= Strainer.create(self)
+      @strainer ||= Strainer.create(self, @filters)
     end
 
     # Adds filters to this context.
@@ -43,11 +52,20 @@ module Liquid
     # for that
     def add_filters(filters)
       filters = [filters].flatten.compact
-
       filters.each do |f|
         raise ArgumentError, "Expected module but got: #{f.class}" unless f.is_a?(Module)
         Strainer.add_known_filter(f)
-        strainer.extend(f)
+      end
+
+      # If strainer is already setup then there's no choice but to use a runtime
+      # extend call. If strainer is not yet created, we can utilize strainers
+      # cached class based API, which avoids busting the method cache.
+      if @strainer
+        filters.each do |f|
+          strainer.extend(f)
+        end
+      else
+        @filters.concat filters
       end
     end
 
@@ -153,15 +171,15 @@ module Liquid
           LITERALS[key]
         else
           case key
-          when /^'(.*)'$/ # Single quoted strings
+          when /\A'(.*)'\z/ # Single quoted strings
             $1
-          when /^"(.*)"$/ # Double quoted strings
+          when /\A"(.*)"\z/ # Double quoted strings
             $1
-          when /^(-?\d+)$/ # Integer and floats
+          when /\A(-?\d+)\z/ # Integer and floats
             $1.to_i
-          when /^\((\S+)\.\.(\S+)\)$/ # Ranges
+          when /\A\((\S+)\.\.(\S+)\)\z/ # Ranges
             (resolve($1).to_i..resolve($2).to_i)
-          when /^(-?\d[\d\.]+)$/ # Floats
+          when /\A(-?\d[\d\.]+)\z/ # Floats
             $1.to_f
           else
             variable(key)
@@ -200,7 +218,7 @@ module Liquid
       #  assert_equal 'tobi', @context['hash["name"]']
       def variable(markup)
         parts = markup.scan(VariableParser)
-        square_bracketed = /^\[(.*)\]$/
+        square_bracketed = /\A\[(.*)\]\z/
 
         first_part = parts.shift
 

lib/liquid/document.rb

@@ -3,12 +3,12 @@ module Liquid
     # we don't need markup to open this block
     def initialize(tokens, options = {})
       @options = options
-      parse(tokens)
+      parse_body(tokens)
     end
 
     # There isn't a real delimiter
     def block_delimiter
-      []
+      nil
     end
 
     # Document blocks don't need to be terminated since they are not actually opened

lib/liquid/drop.rb

@@ -52,6 +52,10 @@ module Liquid
       self
     end
 
+    def to_s
+      self.class.name
+    end
+
     alias :[] :invoke_drop
 
     private
@@ -59,13 +63,12 @@ module Liquid
     # Check for method existence without invoking respond_to?, which creates symbols
     def self.invokable?(method_name)
       unless @invokable_methods
-        # Ruby 1.8 compatibility: call to_s on method names (which are strings in 1.8, but already symbols in 1.9)
-        blacklist = (Liquid::Drop.public_instance_methods + [:each]).map(&:to_s)
+        blacklist = Liquid::Drop.public_instance_methods + [:each]
         if include?(Enumerable)
-          blacklist += Enumerable.public_instance_methods.map(&:to_s)
-          blacklist -= [:sort, :count, :first, :min, :max, :include?].map(&:to_s)
+          blacklist += Enumerable.public_instance_methods
+          blacklist -= [:sort, :count, :first, :min, :max, :include?]
         end
-        whitelist = [:to_liquid] + (public_instance_methods.map(&:to_s) - blacklist.map(&:to_s))
+        whitelist = [:to_liquid] + (public_instance_methods - blacklist)
         @invokable_methods = Set.new(whitelist.map(&:to_s))
       end
       @invokable_methods.include?(method_name.to_s)

lib/liquid/file_system.rb

@@ -31,11 +31,22 @@ module Liquid
   # file_system.full_path("mypartial")       # => "/some/path/_mypartial.liquid"
   # file_system.full_path("dir/mypartial")   # => "/some/path/dir/_mypartial.liquid"
   #
+  # Optionally in the second argument you can specify a custom pattern for template filenames.
+  # The Kernel::sprintf format specification is used.
+  # Default pattern is "_%s.liquid".
+  #
+  # Example:
+  #
+  # file_system = Liquid::LocalFileSystem.new("/some/path", "%s.html")
+  #
+  # file_system.full_path("index") # => "/some/path/index.html"
+  #
   class LocalFileSystem
     attr_accessor :root
 
-    def initialize(root)
+    def initialize(root, pattern = "_%s.liquid")
       @root = root
+      @pattern = pattern
     end
 
     def read_template_file(template_path, context)
@@ -46,15 +57,15 @@ module Liquid
     end
 
     def full_path(template_path)
-      raise FileSystemError, "Illegal template name '#{template_path}'" unless template_path =~ /^[^.\/][a-zA-Z0-9_\/]+$/
+      raise FileSystemError, "Illegal template name '#{template_path}'" unless template_path =~ /\A[^.\/][a-zA-Z0-9_\/]+\z/
 
       full_path = if template_path.include?('/')
-        File.join(root, File.dirname(template_path), "_#{File.basename(template_path)}.liquid")
+        File.join(root, File.dirname(template_path), @pattern % File.basename(template_path))
       else
-        File.join(root, "_#{template_path}.liquid")
+        File.join(root, @pattern % template_path)
       end
 
-      raise FileSystemError, "Illegal template path '#{File.expand_path(full_path)}'" unless File.expand_path(full_path) =~ /^#{File.expand_path(root)}/
+      raise FileSystemError, "Illegal template path '#{File.expand_path(full_path)}'" unless File.expand_path(full_path) =~ /\A#{File.expand_path(root)}/
 
       full_path
     end

lib/liquid/htmltags.rb

@@ -23,7 +23,7 @@ module Liquid
       from = @attributes['offset'] ? context[@attributes['offset']].to_i : 0
       to = @attributes['limit'] ? from + context[@attributes['limit']].to_i : nil
 
-      collection = Utils.slice_collection_using_each(collection, from, to)
+      collection = Utils.slice_collection(collection, from, to)
 
       length = collection.length
 
@@ -55,7 +55,7 @@ module Liquid
 
           col += 1
 
-          result << "<td class=\"col#{col}\">" << render_all(@nodelist, context) << '</td>'
+          result << "<td class=\"col#{col}\">" << super << '</td>'
 
           if col == cols and (index != length - 1)
             col  = 0

lib/liquid/i18n.rb

@@ -24,7 +24,7 @@ module Liquid
 
     private
     def interpolate(name, vars)
-      name.gsub(/%{(\w+)}/) {
+      name.gsub(/%\{(\w+)\}/) {
         # raise TranslationError, "Undefined key #{$1} for interpolation in translation #{name}"  unless vars[$1.to_sym]
         "#{vars[$1.to_sym]}"
       }

lib/liquid/lexer.rb

@@ -15,6 +15,7 @@ module Liquid
     SINGLE_STRING_LITERAL = /'[^\']*'/
     DOUBLE_STRING_LITERAL = /"[^\"]*"/
     NUMBER_LITERAL = /-?\d+(\.\d+)?/
+    DOTDOT = /\.\./
     COMPARISON_OPERATOR = /==|!=|<>|<=?|>=?|contains/
 
     def initialize(input)
@@ -32,6 +33,7 @@ module Liquid
         when t = @ss.scan(DOUBLE_STRING_LITERAL) then [:string, t]
         when t = @ss.scan(NUMBER_LITERAL) then [:number, t]
         when t = @ss.scan(IDENTIFIER) then [:id, t]
+        when t = @ss.scan(DOTDOT) then [:dotdot, t]
         else
           c = @ss.getch
           if s = SPECIALS[c]

lib/liquid/parser.rb

@@ -53,8 +53,7 @@ module Liquid
       elsif token.first == :open_round
         consume
         first = expression
-        consume(:dot)
-        consume(:dot)
+        consume(:dotdot)
         last = expression
         consume(:close_round)
         "(#{first}..#{last})"

lib/liquid/standardfilters.rb

@@ -4,6 +4,8 @@ require 'bigdecimal'
 module Liquid
 
   module StandardFilters
+    HTML_ESCAPE = { '&' => '&amp;',  '>' => '&gt;',   '<' => '&lt;', '"' => '&quot;', "'" => '&#39;' }
+    HTML_ESCAPE_ONCE_REGEXP = /["><']|&(?!([a-zA-Z]+|(#\d+));)/
 
     # Return the size of an array or of an string
     def size(input)
@@ -31,9 +33,7 @@ module Liquid
     end
 
     def escape_once(input)
-      ActionView::Helpers::TagHelper.escape_once(input)
-    rescue NameError
-      input
+      input.to_s.gsub(HTML_ESCAPE_ONCE_REGEXP, HTML_ESCAPE)
     end
 
     alias_method :h, :escape
@@ -43,8 +43,7 @@ module Liquid
       if input.nil? then return end
       l = length.to_i - truncate_string.length
       l = 0 if l < 0
-      truncated = RUBY_VERSION[0,3] == "1.8" ? input.scan(/./mu)[0...l].to_s : input[0...l]
-      input.length > length.to_i ? truncated + truncate_string : input
+      input.length > length.to_i ? input[0...l] + truncate_string : input
     end
 
     def truncatewords(input, words = 15, truncate_string = "...")
@@ -64,6 +63,18 @@ module Liquid
       input.split(pattern)
     end
 
+    def strip(input)
+      input.to_s.strip
+    end
+
+    def lstrip(input)
+      input.to_s.lstrip
+    end
+
+    def rstrip(input)
+      input.to_s.rstrip
+    end
+
     def strip_html(input)
       input.to_s.gsub(/<script.*?<\/script>/m, '').gsub(/<!--.*?-->/m, '').gsub(/<style.*?<\/style>/m, '').gsub(/<.*?>/m, '')
     end
@@ -179,7 +190,7 @@ module Liquid
         return input.to_s
       end
 
-      if ((input.is_a?(String) && !/^\d+$/.match(input.to_s).nil?) || input.is_a?(Integer)) && input.to_i > 0
+      if ((input.is_a?(String) && !/\A\d+\z/.match(input.to_s).nil?) || input.is_a?(Integer)) && input.to_i > 0
         input = Time.at(input.to_i)
       end
 
@@ -245,12 +256,17 @@ module Liquid
       apply_operation(input, operand, :%)
     end
 
+    def default(input, default_value = "")
+      is_blank = input.respond_to?(:empty?) ? input.empty? : !input
+      is_blank ? default_value : input
+    end
+
     private
 
     def flatten_if_necessary(input)
       ary = if input.is_a?(Array)
         input.flatten
-      elsif input.kind_of?(Enumerable)
+      elsif input.is_a?(Enumerable) && !input.is_a?(Hash)
         input
       else
         [input].flatten
@@ -265,7 +281,7 @@ module Liquid
       when Numeric
         obj
       when String
-        (obj.strip =~ /^\d+\.\d+$/) ? BigDecimal.new(obj) : obj.to_i
+        (obj.strip =~ /\A\d+\.\d+\z/) ? BigDecimal.new(obj) : obj.to_i
       else
         0
       end

lib/liquid/strainer.rb

@@ -11,6 +11,11 @@ module Liquid
     @@filters = []
     @@known_filters = Set.new
     @@known_methods = Set.new
+    @@strainer_class_cache = Hash.new do |hash, filters|
+      hash[filters] = Class.new(Strainer) do
+        filters.each { |f| include f }
+      end
+    end
 
     def initialize(context)
       @context = context
@@ -32,10 +37,13 @@ module Liquid
       end
     end
 
-    def self.create(context)
-      strainer = Strainer.new(context)
-      @@filters.each { |m| strainer.extend(m) }
-      strainer
+    def self.strainer_class_cache
+      @@strainer_class_cache
+    end
+
+    def self.create(context, filters = [])
+      filters = @@filters + filters
+      strainer_class_cache[filters].new(context)
     end
 
     def invoke(method, *args)
@@ -44,6 +52,8 @@ module Liquid
       else
         args.first
       end
+    rescue ::ArgumentError => e
+      raise Liquid::ArgumentError.new(e.message)
     end
 
     def invokable?(method)

lib/liquid/tag.rb

@@ -1,7 +1,7 @@
 module Liquid
   class Tag
-    attr_accessor :nodelist, :options
-    attr_reader :warnings
+    attr_accessor :options
+    attr_reader :nodelist, :warnings
 
     def self.new_with_options(tag_name, markup, tokens, options)
       # Forgive me Matz for I have sinned. I know this code is weird
@@ -16,10 +16,6 @@ module Liquid
       @tag_name   = tag_name
       @markup     = markup
       @options    ||= {} # needs || because might be set before initialize
-      parse(tokens)
-    end
-
-    def parse(tokens)
     end
 
     def name
@@ -31,7 +27,7 @@ module Liquid
     end
 
     def blank?
-      @blank || true
+      @blank || false
     end
 
     def parse_with_selected_parser(markup)

lib/liquid/tags/assign.rb

@@ -25,10 +25,13 @@ module Liquid
     def render(context)
       val = @from.render(context)
       context.scopes.last[@to] = val
-      context.resource_limits[:assign_score_current] += (val.respond_to?(:length) ? val.length : 1)
+      context.increment_used_resources(:assign_score_current, val)
       ''
     end
 
+    def blank?
+      true
+    end
   end
 
   Template.register_tag('assign', Assign)

lib/liquid/tags/capture.rb

@@ -27,7 +27,7 @@ module Liquid
     def render(context)
       output = super
       context.scopes.last[@to] = output
-      context.resource_limits[:assign_score_current] += (output.respond_to?(:length) ? output.length : 1)
+      context.increment_used_resources(:assign_score_current, output)
       ''
     end
 

lib/liquid/tags/case.rb

@@ -15,6 +15,10 @@ module Liquid
       super
     end
 
+    def nodelist
+      @blocks.map(&:attachment).flatten
+    end
+
     def unknown_tag(tag, markup, tokens)
       @nodelist = []
       case tag

lib/liquid/tags/comment.rb

@@ -4,6 +4,9 @@ module Liquid
       ''
     end
 
+    def unknown_tag(tag, markup, tokens)
+    end
+
     def blank?
       true
     end

lib/liquid/tags/cycle.rb

@@ -12,8 +12,8 @@ module Liquid
   #    <div class="green"> Item five</div>
   #
   class Cycle < Tag
-    SimpleSyntax = /^#{QuotedFragment}+/o
-    NamedSyntax  = /^(#{QuotedFragment})\s*\:\s*(.*)/o
+    SimpleSyntax = /\A#{QuotedFragment}+/o
+    NamedSyntax  = /\A(#{QuotedFragment})\s*\:\s*(.*)/o
 
     def initialize(tag_name, markup, tokens)
       case markup

lib/liquid/tags/for.rb

@@ -52,6 +52,14 @@ module Liquid
       super
     end
 
+    def nodelist
+      if @else_block
+        @for_block + @else_block
+      else
+        @for_block
+      end
+    end
+
     def unknown_tag(tag, markup, tokens)
       return super unless tag == 'else'
       @nodelist = @else_block = []
@@ -75,8 +83,7 @@ module Liquid
       limit = context[@attributes['limit']]
       to    = limit ? limit.to_i + from : nil
 
-
-      segment = Utils.slice_collection_using_each(collection, from, to)
+      segment = Utils.slice_collection(collection, from, to)
 
       return render_else(context) if segment.empty?
 

lib/liquid/tags/if.rb

@@ -12,6 +12,7 @@ module Liquid
   class If < Block
     Syntax = /(#{QuotedFragment})\s*([=!<>a-z_]+)?\s*(#{QuotedFragment})?/o
     ExpressionsAndOperators = /(?:\b(?:\s?and\s?|\s?or\s?)\b|(?:\s*(?!\b(?:\s?and\s?|\s?or\s?)\b)(?:#{QuotedFragment}|\S+)\s*)+)/o
+    BOOLEAN_OPERATORS = %w(and or)
 
     def initialize(tag_name, markup, tokens)
       @blocks = []
@@ -19,6 +20,10 @@ module Liquid
       super
     end
 
+    def nodelist
+      @blocks.map(&:attachment).flatten
+    end
+
     def unknown_tag(tag, markup, tokens)
       if ['elsif', 'else'].include?(tag)
         push_block(tag, markup)
@@ -63,7 +68,8 @@ module Liquid
           raise(SyntaxError.new(options[:locale].t("errors.syntax.if"))) unless expressions.shift.to_s =~ Syntax
 
           new_condition = Condition.new($1, $2, $3)
-          new_condition.send(operator.to_sym, condition)
+          raise(SyntaxError.new(options[:locale].t("errors.syntax.if"))) unless BOOLEAN_OPERATORS.include?(operator)
+          new_condition.send(operator, condition)
           condition = new_condition
         end
 

lib/liquid/tags/ifchanged.rb

@@ -4,7 +4,7 @@ module Liquid
     def render(context)
       context.stack do
 
-        output = render_all(@nodelist, context)
+        output = super
 
         if output != context.registers[:ifchanged]
           context.registers[:ifchanged] = output

lib/liquid/tags/include.rb

@@ -35,9 +35,6 @@ module Liquid
       super
     end
 
-    def parse(tokens)
-    end
-
     def blank?
       false
     end
@@ -51,13 +48,14 @@ module Liquid
           context[key] = context[value]
         end
 
+        context_variable_name = @template_name[1..-2].split('/').last
         if variable.is_a?(Array)
           variable.collect do |var|
-            context[@template_name[1..-2]] = var
+            context[context_variable_name] = var
             partial.render(context)
           end
         else
-          context[@template_name[1..-2]] = variable
+          context[context_variable_name] = variable
           partial.render(context)
         end
       end

lib/liquid/tags/raw.rb

@@ -1,17 +1,14 @@
 module Liquid
   class Raw < Block
-    FullTokenPossiblyInvalid = /^(.*)#{TagStart}\s*(\w+)\s*(.*)?#{TagEnd}$/o
+    FullTokenPossiblyInvalid = /\A(.*)#{TagStart}\s*(\w+)\s*(.*)?#{TagEnd}\z/o
 
-    def parse(tokens)
+    def parse_body(tokens)
       @nodelist ||= []
       @nodelist.clear
       while token = tokens.shift
         if token =~ FullTokenPossiblyInvalid
           @nodelist << $1 if $1 != ""
-          if block_delimiter == $2
-            end_tag
-            return
-          end
+          return if block_delimiter == $2
         end
         @nodelist << token if not token.empty?
       end

lib/liquid/template.rb

@@ -162,16 +162,9 @@ module Liquid
 
     private
 
-    # Uses the <tt>Liquid::TemplateParser</tt> regexp to tokenize the passed source
     def tokenize(source)
       source = source.source if source.respond_to?(:source)
-      return [] if source.to_s.empty?
-      tokens = source.split(TemplateParser)
-
-      # removes the rogue empty element at the beginning of the array
-      tokens.shift if tokens[0] and tokens[0].empty?
-
-      tokens
+      Tokenizer.new(source.to_s)
     end
 
   end

lib/liquid/utils.rb

@@ -1,5 +1,18 @@
 module Liquid
   module Utils
+
+    def self.slice_collection(collection, from, to)
+      if (from != 0 || to != nil) && collection.respond_to?(:load_slice)
+        collection.load_slice(from, to)
+      else
+        slice_collection_using_each(collection, from, to)
+      end
+    end
+
+    def self.non_blank_string?(collection)
+      collection.is_a?(String) && collection != ''
+    end
+
     def self.slice_collection_using_each(collection, from, to)
       segments = []
       index = 0
@@ -22,9 +35,5 @@ module Liquid
 
       segments
     end
-
-    def self.non_blank_string?(collection)
-      collection.is_a?(String) && collection != ''
-    end
   end
 end

lib/liquid/variable.rb

@@ -11,8 +11,7 @@ module Liquid
   #   {{ user | link }}
   #
   class Variable
-    FilterParser = /(?:#{FilterSeparator}|(?:\s*(?:#{QuotedFragment}|#{ArgumentSeparator})\s*)+)/o
-    EasyParse = /^ *(\w+(?:\.\w+)*) *$/
+    EasyParse = /\A *(\w+(?:\.\w+)*) *\z/
     attr_accessor :filters, :name, :warnings
 
     def initialize(markup, options = {})
@@ -35,22 +34,22 @@ module Liquid
       end
     end
 
-    def lax_parse(markup)
-      @filters = []
-      if match = markup.match(/\s*(#{QuotedFragment})(.*)/o)
-        @name = match[1]
-        if match[2].match(/#{FilterSeparator}\s*(.*)/o)
-          filters = Regexp.last_match(1).scan(FilterParser)
-          filters.each do |f|
-            if matches = f.match(/\s*(\w+)/)
-              filtername = matches[1]
-              filterargs = f.scan(/(?:#{FilterArgumentSeparator}|#{ArgumentSeparator})\s*((?:\w+\s*\:\s*)?#{QuotedFragment})/o).flatten
-              @filters << [filtername, filterargs]
-            end
-          end
-        end
-      end
-    end
+#     def lax_parse(markup)
+#       @filters = []
+#       if match = markup.match(/\s*(#{QuotedFragment})(.*)/o)
+#         @name = match[1]
+#         if match[2].match(/#{FilterSeparator}\s*(.*)/o)
+#           filters = Regexp.last_match(1).scan(FilterParser)
+#           filters.each do |f|
+#             if matches = f.match(/\s*(\w+)/)
+#               filtername = matches[1]
+#               filterargs = f.scan(/(?:#{FilterArgumentSeparator}|#{ArgumentSeparator})\s*((?:\w+\s*\:\s*)?#{QuotedFragment})/o).flatten
+#               @filters << [filtername, filterargs]
+#             end
+#           end
+#         end
+#       end
+#     end
 
     def strict_parse(markup)
       # Very simple valid cases

lib/liquid/version.rb

@@ -1,4 +1,4 @@
 # encoding: utf-8
 module Liquid
-  VERSION = "2.6.0"
+  VERSION = "3.0.0"
 end

liquid.gemspec

@@ -18,9 +18,15 @@ Gem::Specification.new do |s|
   s.required_rubygems_version = ">= 1.3.7"
 
   s.test_files  = Dir.glob("{test}/**/*")
-  s.files       = Dir.glob("{lib}/**/*") + %w(MIT-LICENSE README.md)
+  s.files       = Dir.glob("{lib,ext}/**/*") + %w(MIT-LICENSE README.md)
+  s.extensions  = ['ext/liquid/extconf.rb']
 
   s.extra_rdoc_files  = ["History.md", "README.md"]
 
   s.require_path = "lib"
+
+  s.add_development_dependency 'rake-compiler'
+  s.add_development_dependency 'stackprof'
+  s.add_development_dependency 'rake'
+  s.add_development_dependency 'activesupport'
 end

performance/shopify/shop_filter.rb

@@ -54,7 +54,7 @@ module ShopFilter
 
   def product_img_url(url, style = 'small')
 
-    unless url =~ /^products\/([\w\-\_]+)\.(\w{2,4})/
+    unless url =~ /\Aproducts\/([\w\-\_]+)\.(\w{2,4})/
       raise ArgumentError, 'filter "size" can only be called on product images'
     end
 

performance/stackprof.rb

@@ -0,0 +1,15 @@
+require 'stackprof' rescue fail("install stackprof extension/gem")
+require File.dirname(__FILE__) + '/theme_runner'
+
+profiler = ThemeRunner.new
+profiler.run
+results = StackProf.run(mode: :cpu, out: ENV['FILENAME']) do
+  100.times do
+    profiler.run
+  end
+end
+if results.kind_of?(File)
+  puts "wrote stackprof dump to #{results.path}"
+else
+  StackProf::Report.new(results).print_text(false, 20)
+end

test/liquid/blank_test.rb

@@ -1,5 +1,13 @@
 require 'test_helper'
 
+class FoobarTag < Liquid::Tag
+  def render(*args)
+    " "
+  end
+
+  Liquid::Template.register_tag('foobar', FoobarTag)
+end
+
 class BlankTestFileSystem
   def read_template_file(template_path, context)
     template_path
@@ -22,6 +30,10 @@ class BlankTest < Test::Unit::TestCase
     wrap_in_for(body) + wrap_in_if(body)
   end
 
+  def test_new_tags_are_not_blank_by_default
+    assert_template_result(" "*N, wrap_in_for("{% foobar %}"))
+  end
+
   def test_loops_are_blank
     assert_template_result("", wrap_in_for(" "))
   end

test/liquid/drop_test.rb

@@ -237,4 +237,9 @@ class DropsTest < Test::Unit::TestCase
   def test_nil_value_access
     assert_equal '', Liquid::Template.parse('{{ product[value] }}').render('product' => ProductDrop.new, 'value' => nil)
   end
+
+  def test_default_to_s_on_drops
+    assert_equal 'ProductDrop', Liquid::Template.parse("{{ product }}").render('product' => ProductDrop.new)
+    assert_equal 'EnumerableDrop', Liquid::Template.parse('{{ collection }}').render('collection' => EnumerableDrop.new)
+  end
 end # DropsTest

test/liquid/file_system_test.rb

@@ -26,4 +26,10 @@ class FileSystemTest < Test::Unit::TestCase
       file_system.full_path("/etc/passwd")
     end
   end
+
+  def test_custom_template_filename_patterns
+    file_system = Liquid::LocalFileSystem.new("/some/path", "%s.html")
+    assert_equal "/some/path/mypartial.html"    , file_system.full_path("mypartial")
+    assert_equal "/some/path/dir/mypartial.html", file_system.full_path("dir/mypartial")
+  end
 end # FileSystemTest

test/liquid/output_test.rb

@@ -86,7 +86,7 @@ class OutputTest < Test::Unit::TestCase
     assert_equal expected, Template.parse(text).render(@assigns, :filters => [FunnyFilter])
   end
 
-  def test_variable_piping_with_args
+  def test_variable_piping_with_multiple_args
     text = %! {{ car.gm | add_tag : 'span', 'bar'}} !
     expected = %| <span id="bar">bad</span> |
 

test/liquid/parser_test.rb

@@ -56,6 +56,14 @@ class ParserTest < Test::Unit::TestCase
     assert_equal '"wut"', p.expression
   end
 
+  def test_ranges
+    p = Parser.new("(5..7) (1.5..9.6) (young..old) (hi[5].wat..old)")
+    assert_equal '(5..7)', p.expression
+    assert_equal '(1.5..9.6)', p.expression
+    assert_equal '(young..old)', p.expression
+    assert_equal '(hi[5].wat..old)', p.expression
+  end
+
   def test_arguments
     p = Parser.new("filter: hi.there[5], keyarg: 7")
     assert_equal 'filter', p.consume(:id)

test/liquid/regexp_test.rb

@@ -21,11 +21,11 @@ class RegexpTest < Test::Unit::TestCase
     assert_equal ['<style', 'class="hello">', '</style>'], %|<style class="hello">' </style>|.scan(QuotedFragment)
   end
 
-  def test_quoted_words
+  def test_double_quoted_words
     assert_equal ['arg1', 'arg2', '"arg 3"'], 'arg1 arg2 "arg 3"'.scan(QuotedFragment)
   end
 
-  def test_quoted_words
+  def test_single_quoted_words
     assert_equal ['arg1', 'arg2', "'arg 3'"], 'arg1 arg2 \'arg 3\''.scan(QuotedFragment)
   end
 

test/liquid/standard_filter_test.rb

@@ -62,11 +62,6 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_equal '', @filters.upcase(nil)
   end
 
-  def test_upcase
-    assert_equal 'TESTING', @filters.upcase("Testing")
-    assert_equal '', @filters.upcase(nil)
-  end
-
   def test_truncate
     assert_equal '1234...', @filters.truncate('1234567890', 7)
     assert_equal '1234567890', @filters.truncate('1234567890', 20)
@@ -75,7 +70,7 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_equal "测试...", @filters.truncate("测试测试测试测试", 5)
   end
 
-  def test_strip
+  def test_split
     assert_equal ['12','34'], @filters.split('12~34', '~')
     assert_equal ['A? ',' ,Z'], @filters.split('A? ~ ~ ~ ,Z', '~ ~ ~')
     assert_equal ['A?Z'], @filters.split('A?Z', '~')
@@ -89,7 +84,7 @@ class StandardFiltersTest < Test::Unit::TestCase
   end
 
   def test_escape_once
-    assert_equal '&lt;strong&gt;', @filters.escape_once(@filters.escape('<strong>'))
+    assert_equal '&lt;strong&gt;Hulk&lt;/strong&gt;', @filters.escape_once('&lt;strong&gt;Hulk</strong>')
   end
 
   def test_truncatewords
@@ -140,6 +135,10 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_equal "woot: 1", Liquid::Template.parse('{{ foo | map: "whatever" }}').render("foo" => [t])
   end
 
+  def test_map_on_hashes
+    assert_equal "4217", Liquid::Template.parse('{{ thing | map: "foo" | map: "bar" }}').render("thing" => { "foo" => [ { "bar" => 42 }, { "bar" => 17 } ] })
+  end
+
   def test_sort_calls_to_liquid
     t = TestThing.new
     assert_equal "woot: 1", Liquid::Template.parse('{{ foo | sort: "whatever" }}').render("foo" => [t])
@@ -210,6 +209,21 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_template_result 'foobar', "{{ 'foo|bar' | remove: '|' }}"
   end
 
+  def test_strip
+    assert_template_result 'ab c', "{{ source | strip }}", 'source' => " ab c  "
+    assert_template_result 'ab c', "{{ source | strip }}", 'source' => " \tab c  \n \t"
+  end
+
+  def test_lstrip
+    assert_template_result 'ab c  ', "{{ source | lstrip }}", 'source' => " ab c  "
+    assert_template_result "ab c  \n \t", "{{ source | lstrip }}", 'source' => " \tab c  \n \t"
+  end
+
+  def test_rstrip
+    assert_template_result " ab c", "{{ source | rstrip }}", 'source' => " ab c  "
+    assert_template_result " \tab c", "{{ source | rstrip }}", 'source' => " \tab c  \n \t"
+  end
+
   def test_strip_newlines
     assert_template_result 'abc', "{{ source | strip_newlines }}", 'source' => "a\nb\nc"
     assert_template_result 'abc', "{{ source | strip_newlines }}", 'source' => "a\r\nb\nc"
@@ -233,9 +247,6 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_template_result "12", "{{ 3 | times:4 }}"
     assert_template_result "0", "{{ 'foo' | times:4 }}"
 
-    # Ruby v1.9.2-rc1, or higher, backwards compatible Float test
-    assert_match(/(6\.3)|(6\.(0{13})1)/, Template.parse("{{ '2.1' | times:3 }}").render)
-
     assert_template_result "6", "{{ '2.1' | times:3 | replace: '.','-' | plus:0}}"
 
     assert_template_result "7.25", "{{ 0.0725 | times:100 }}"
@@ -245,9 +256,6 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_template_result "4", "{{ 12 | divided_by:3 }}"
     assert_template_result "4", "{{ 14 | divided_by:3 }}"
 
-    # Ruby v1.9.2-rc1, or higher, backwards compatible Float test
-    assert_match(/4\.(6{13,14})7/, Template.parse("{{ 14 | divided_by:'3.0' }}").render)
-
     assert_template_result "5", "{{ 15 | divided_by:3 }}"
     assert_template_result "Liquid error: divided by 0", "{{ 5 | divided_by:0 }}"
 
@@ -270,6 +278,15 @@ class StandardFiltersTest < Test::Unit::TestCase
     assert_template_result('abc',"{{ a | prepend: b}}",assigns)
   end
 
+  def test_default
+    assert_equal "foo", @filters.default("foo", "bar")
+    assert_equal "bar", @filters.default(nil, "bar")
+    assert_equal "bar", @filters.default("", "bar")
+    assert_equal "bar", @filters.default(false, "bar")
+    assert_equal "bar", @filters.default([], "bar")
+    assert_equal "bar", @filters.default({}, "bar")
+  end
+
   def test_cannot_access_private_methods
     assert_template_result('a',"{{ 'a' | to_number }}")
   end

test/liquid/strainer_test.rb

@@ -22,6 +22,13 @@ class StrainerTest < Test::Unit::TestCase
     assert_equal "public", strainer.invoke("public_filter")
   end
 
+  def test_stainer_raises_argument_error
+    strainer = Strainer.create(nil)
+    assert_raises(Liquid::ArgumentError) do
+      strainer.invoke("public_filter", 1)
+    end
+  end
+
   def test_strainer_only_invokes_public_filter_methods
     strainer = Strainer.create(nil)
     assert_equal false, strainer.invokable?('__test__')
@@ -49,4 +56,15 @@ class StrainerTest < Test::Unit::TestCase
     assert_equal "has_method?", strainer.invoke("invoke", "has_method?", "invoke")
   end
 
+  def test_strainer_uses_a_class_cache_to_avoid_method_cache_invalidation
+    a, b = Module.new, Module.new
+    strainer = Strainer.create(nil, [a,b])
+    assert_kind_of Strainer, strainer
+    assert_kind_of a, strainer
+    assert_kind_of b, strainer
+    Strainer.class_variable_get(:@@filters).each do |m|
+      assert_kind_of m, strainer
+    end
+  end
+
 end # StrainerTest

test/liquid/tags/case_tag_test.rb

@@ -0,0 +1,10 @@
+require 'test_helper'
+
+class CaseTagTest < Test::Unit::TestCase
+  include Liquid
+
+  def test_case_nodelist
+    template = Liquid::Template.parse('{% case var %}{% when true %}WHEN{% else %}ELSE{% endcase %}')
+    assert_equal ['WHEN', 'ELSE'], template.root.nodelist[0].nodelist
+  end
+end # CaseTest

test/liquid/tags/for_tag_test.rb

@@ -294,4 +294,72 @@ HERE
     assigns  = {'items' => [1,2,3,4,5]}
     assert_template_result(expected, template, assigns)
   end
+
+  def test_for_nodelist
+    template = Liquid::Template.parse('{% for item in items %}FOR{% endfor %}')
+    assert_equal ['FOR'], template.root.nodelist[0].nodelist
+  end
+
+  def test_for_else_nodelist
+    template = Liquid::Template.parse('{% for item in items %}FOR{% else %}ELSE{% endfor %}')
+    assert_equal ['FOR', 'ELSE'], template.root.nodelist[0].nodelist
+  end
+
+  class LoaderDrop < Liquid::Drop
+    attr_accessor :each_called, :load_slice_called
+
+    def initialize(data)
+      @data = data
+    end
+
+    def each
+      @each_called = true
+      @data.each { |el| yield el }
+    end
+
+    def load_slice(from, to)
+      @load_slice_called = true
+      @data[(from..to-1)]
+    end
+  end
+
+  def test_iterate_with_each_when_no_limit_applied
+    loader = LoaderDrop.new([1,2,3,4,5])
+    assigns = {'items' => loader}
+    expected = '12345'
+    template = '{% for item in items %}{{item}}{% endfor %}'
+    assert_template_result(expected, template, assigns)
+    assert loader.each_called
+    assert !loader.load_slice_called
+  end
+
+  def test_iterate_with_load_slice_when_limit_applied
+    loader = LoaderDrop.new([1,2,3,4,5])
+    assigns = {'items' => loader}
+    expected = '1'
+    template = '{% for item in items limit:1 %}{{item}}{% endfor %}'
+    assert_template_result(expected, template, assigns)
+    assert !loader.each_called
+    assert loader.load_slice_called
+  end
+
+  def test_iterate_with_load_slice_when_limit_and_offset_applied
+    loader = LoaderDrop.new([1,2,3,4,5])
+    assigns = {'items' => loader}
+    expected = '34'
+    template = '{% for item in items offset:2 limit:2 %}{{item}}{% endfor %}'
+    assert_template_result(expected, template, assigns)
+    assert !loader.each_called
+    assert loader.load_slice_called
+  end
+
+  def test_iterate_with_load_slice_returns_same_results_as_without
+    loader = LoaderDrop.new([1,2,3,4,5])
+    loader_assigns = {'items' => loader}
+    array_assigns = {'items' => [1,2,3,4,5]}
+    expected = '34'
+    template = '{% for item in items offset:2 limit:2 %}{{item}}{% endfor %}'
+    assert_template_result(expected, template, loader_assigns)
+    assert_template_result(expected, template, array_assigns)
+  end
 end

test/liquid/tags/if_else_tag_test.rb

@@ -157,4 +157,15 @@ class IfElseTagTest < Test::Unit::TestCase
     assert_template_result('yes',
                            %({% if 'gnomeslab-and-or-liquid' contains 'gnomeslab-and-or-liquid' %}yes{% endif %}))
   end
+
+  def test_if_nodelist
+    template = Liquid::Template.parse('{% if true %}IF{% else %}ELSE{% endif %}')
+    assert_equal ['IF', 'ELSE'], template.root.nodelist[0].nodelist
+  end
+
+  def test_operators_are_whitelisted
+    assert_raise(SyntaxError) do
+      assert_template_result('', %({% if 1 or throw or or 1 %}yes{% endif %}))
+    end
+  end
 end # IfElseTest

test/liquid/tags/include_tag_test.rb

@@ -48,6 +48,27 @@ class CountingFileSystem
   end
 end
 
+class CustomInclude < Liquid::Tag
+  Syntax = /(#{Liquid::QuotedFragment}+)(\s+(?:with|for)\s+(#{Liquid::QuotedFragment}+))?/o
+
+  def initialize(tag_name, markup, tokens)
+    markup =~ Syntax
+    @template_name = $1
+    super
+  end
+
+  def parse(tokens)
+  end
+
+  def blank?
+    false
+  end
+
+  def render(context)
+    @template_name[1..-2]
+  end
+end
+
 class IncludeTagTest < Test::Unit::TestCase
   include Liquid
 
@@ -163,4 +184,31 @@ class IncludeTagTest < Test::Unit::TestCase
       Template.parse("{% include 'pick_a_source' %}").render({}, :registers => {:file_system => file_system})
     assert_equal 2, file_system.count
   end
+
+  def test_include_tag_within_if_statement
+    assert_equal "foo_if_true",
+      Template.parse("{% if true %}{% include 'foo_if_true' %}{% endif %}").render
+  end
+
+  def test_custom_include_tag
+    original_tag = Liquid::Template.tags['include']
+    Liquid::Template.tags['include'] = CustomInclude
+    begin
+      assert_equal "custom_foo",
+        Template.parse("{% include 'custom_foo' %}").render
+    ensure
+      Liquid::Template.tags['include'] = original_tag
+    end
+  end
+
+  def test_custom_include_tag_within_if_statement
+    original_tag = Liquid::Template.tags['include']
+    Liquid::Template.tags['include'] = CustomInclude
+    begin
+      assert_equal "custom_foo_if_true",
+        Template.parse("{% if true %}{% include 'custom_foo_if_true' %}{% endif %}").render
+    ensure
+      Liquid::Template.tags['include'] = original_tag
+    end
+  end
 end # IncludeTagTest

test/liquid/tags/standard_tag_test.rb

@@ -33,6 +33,13 @@ class StandardTagTest < Test::Unit::TestCase
     assert_template_result('','{% comment %}{% endcomment %}')
     assert_template_result('','{%comment%}comment{%endcomment%}')
     assert_template_result('','{% comment %}comment{% endcomment %}')
+    assert_template_result('','{% comment %} 1 {% comment %} 2 {% endcomment %} 3 {% endcomment %}')
+
+    assert_template_result('','{%comment%}{%blabla%}{%endcomment%}')
+    assert_template_result('','{% comment %}{% blabla %}{% endcomment %}')
+    assert_template_result('','{%comment%}{% endif %}{%endcomment%}')
+    assert_template_result('','{% comment %}{% endwhatever %}{% endcomment %}')
+    assert_template_result('','{% comment %}{% raw %} {{%%%%}}  }} { {% endcomment %} {% comment {% endraw %} {% endcomment %}')
 
     assert_template_result('foobar','foo{%comment%}comment{%endcomment%}bar')
     assert_template_result('foobar','foo{% comment %}comment{% endcomment %}bar')
@@ -47,16 +54,9 @@ class StandardTagTest < Test::Unit::TestCase
                                      {%endcomment%}bar')
   end
 
-  def test_assign
-    assigns = {'var' => 'content' }
-    assert_template_result('var2:  var2:content', 'var2:{{var2}} {%assign var2 = var%} var2:{{var2}}', assigns)
-
-  end
-
   def test_hyphenated_assign
     assigns = {'a-b' => '1' }
     assert_template_result('a-b:1 a-b:2', 'a-b:{{a-b}} {%assign a-b = 2 %}a-b:{{a-b}}', assigns)
-
   end
 
   def test_assign_with_colon_and_spaces
@@ -218,7 +218,12 @@ class StandardTagTest < Test::Unit::TestCase
   end
 
   def test_assign
-    assert_equal 'variable', Liquid::Template.parse( '{% assign a = "variable"%}{{a}}'  ).render
+    assert_equal 'variable', Liquid::Template.parse( '{% assign a = "variable"%}{{a}}').render
+  end
+
+  def test_assign_unassigned
+    assigns = { 'var' => 'content' }
+    assert_template_result('var2:  var2:content', 'var2:{{var2}} {%assign var2 = var%} var2:{{var2}}', assigns)
   end
 
   def test_assign_an_empty_string

test/liquid/template_test.rb

@@ -14,29 +14,17 @@ class TemplateContextDrop < Liquid::Drop
   end
 end
 
+class SomethingWithLength
+  def length
+    nil
+  end
+
+  liquid_methods :length
+end
+
 class TemplateTest < Test::Unit::TestCase
   include Liquid
 
-  def test_tokenize_strings
-    assert_equal [' '], Template.new.send(:tokenize, ' ')
-    assert_equal ['hello world'], Template.new.send(:tokenize, 'hello world')
-  end
-
-  def test_tokenize_variables
-    assert_equal ['{{funk}}'], Template.new.send(:tokenize, '{{funk}}')
-    assert_equal [' ', '{{funk}}', ' '], Template.new.send(:tokenize, ' {{funk}} ')
-    assert_equal [' ', '{{funk}}', ' ', '{{so}}', ' ', '{{brother}}', ' '], Template.new.send(:tokenize, ' {{funk}} {{so}} {{brother}} ')
-    assert_equal [' ', '{{  funk  }}', ' '], Template.new.send(:tokenize, ' {{  funk  }} ')
-  end
-
-  def test_tokenize_blocks
-    assert_equal ['{%comment%}'], Template.new.send(:tokenize, '{%comment%}')
-    assert_equal [' ', '{%comment%}', ' '], Template.new.send(:tokenize, ' {%comment%} ')
-
-    assert_equal [' ', '{%comment%}', ' ', '{%endcomment%}', ' '], Template.new.send(:tokenize, ' {%comment%} {%endcomment%} ')
-    assert_equal ['  ', '{% comment %}', ' ', '{% endcomment %}', ' '], Template.new.send(:tokenize, "  {% comment %} {% endcomment %} ")
-  end
-
   def test_instance_assigns_persist_on_same_template_object_between_parses
     t = Template.new
     assert_equal 'from instance assigns', t.parse("{% assign foo = 'from instance assigns' %}{{ foo }}").render
@@ -86,6 +74,12 @@ class TemplateTest < Test::Unit::TestCase
     @global = nil
   end
 
+  def test_resource_limits_works_with_custom_length_method
+    t = Template.parse("{% assign foo = bar %}")
+    t.resource_limits = { :render_length_limit => 42 }
+    assert_equal "", t.render("bar" => SomethingWithLength.new)
+  end
+
   def test_resource_limits_render_length
     t = Template.parse("0123456789")
     t.resource_limits = { :render_length_limit => 5 }

test/liquid/tokenizer_test.rb

@@ -0,0 +1,64 @@
+require 'test_helper'
+
+class TokenizerTest < Test::Unit::TestCase
+  def test_tokenize_strings
+    assert_equal [' '], tokenize(' ')
+    assert_equal ['hello world'], tokenize('hello world')
+  end
+
+  def test_tokenize_variables
+    assert_equal ['{{funk}}'], tokenize('{{funk}}')
+    assert_equal [' ', '{{funk}}', ' '], tokenize(' {{funk}} ')
+    assert_equal [' ', '{{funk}}', ' ', '{{so}}', ' ', '{{brother}}', ' '], tokenize(' {{funk}} {{so}} {{brother}} ')
+    assert_equal [' ', '{{  funk  }}', ' '], tokenize(' {{  funk  }} ')
+  end
+
+  def test_tokenize_blocks
+    assert_equal ['{%comment%}'], tokenize('{%comment%}')
+    assert_equal [' ', '{%comment%}', ' '], tokenize(' {%comment%} ')
+
+    assert_equal [' ', '{%comment%}', ' ', '{%endcomment%}', ' '], tokenize(' {%comment%} {%endcomment%} ')
+    assert_equal ['  ', '{% comment %}', ' ', '{% endcomment %}', ' '], tokenize("  {% comment %} {% endcomment %} ")
+  end
+
+  def test_tokenize_incomplete_end
+    assert_tokens 'before{{ incomplete }after', ['before', '{{ incomplete }', 'after']
+    assert_tokens 'before{% incomplete %after', ['before', '{%', ' incomplete %after']
+  end
+
+  def test_tokenize_no_end
+    assert_tokens 'before{{ unterminated ', ['before', '{{', ' unterminated ']
+    assert_tokens 'before{% unterminated ', ['before', '{%', ' unterminated ']
+  end
+
+  private
+
+  def assert_tokens(source, expected)
+    assert_equal expected, tokenize(source)
+    assert_equal expected, old_tokenize(source)
+  end
+
+  def tokenize(source)
+    tokenizer = Liquid::Tokenizer.new(source)
+    tokens = []
+    while token = tokenizer.next
+      tokens << token
+    end
+    tokens
+  end
+
+  AnyStartingTag        = /\{\{|\{\%/
+  VariableIncompleteEnd = /\}\}?/
+  PartialTemplateParser = /#{Liquid::TagStart}.*?#{Liquid::TagEnd}|#{Liquid::VariableStart}.*?#{VariableIncompleteEnd}/o
+  TemplateParser        = /(#{PartialTemplateParser}|#{AnyStartingTag})/o
+
+  def old_tokenize(source)
+    return [] if source.to_s.empty?
+    tokens = source.split(TemplateParser)
+
+    # removes the rogue empty element at the beginning of the array
+    tokens.shift if tokens[0] and tokens[0].empty?
+
+    tokens
+  end
+end

test/liquid/variable_test.rb

@@ -51,11 +51,9 @@ class VariableTest < Test::Unit::TestCase
   end
 
   def test_filter_with_date_parameter
-
     var = Variable.new(%! '2006-06-06' | date: "%m/%d/%Y"!)
     assert_equal "'2006-06-06'", var.name
     assert_equal [["date",["\"%m/%d/%Y\""]]], var.filters
-
   end
 
   def test_filters_without_whitespace
@@ -73,7 +71,7 @@ class VariableTest < Test::Unit::TestCase
   end
 
   def test_symbol
-    var = Variable.new("http://disney.com/logo.gif | image: 'med' ", :error_mode => :lax)
+    var = Variable.new("http://disney.com/logo.gif | image: 'med'", :error_mode => :lax)
     assert_equal "http://disney.com/logo.gif", var.name
     assert_equal [["image",["'med'"]]], var.filters
   end

test/test_helper.rb

@@ -2,11 +2,6 @@
 
 require 'test/unit'
 require 'test/unit/assertions'
-begin
-  require 'ruby-debug'
-rescue LoadError
-  puts "Couldn't load ruby-debug. gem install ruby-debug if you need it."
-end
 
 $:.unshift(File.join(File.expand_path(File.dirname(__FILE__)), '..', 'lib'))
 require 'liquid.rb'