Full Code of ko1/yomikomu for AI

Repository: ko1/yomikomu
Branch: master
Commit: 5d26d28f8d4f
Files: 15
Total size: 17.2 KB

Directory structure:
gitextract_zi77dzkm/

├── .gitignore
├── .travis.yml
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── bin/
│   ├── console
│   └── setup
├── exe/
│   └── kakidasu
├── lib/
│   ├── yomikomu/
│   │   └── version.rb
│   └── yomikomu.rb
├── test/
│   ├── test_helper.rb
│   ├── x.rb
│   └── yomikomu_test.rb
└── yomikomu.gemspec

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
/.bundle/
/.yardoc
/Gemfile.lock
/_yardoc/
/coverage/
/doc/
/pkg/
/spec/reports/
/tmp/


================================================
FILE: .travis.yml
================================================
language: ruby
rvm:
  - 2.3.0
before_install: gem install bundler -v 1.10.6


================================================
FILE: Gemfile
================================================
source 'https://rubygems.org'

# Specify your gem's dependencies in yomikomu.gemspec
gemspec


================================================
FILE: LICENSE.txt
================================================
The MIT License (MIT)

Copyright (c) 2015 Koichi Sasada

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


================================================
FILE: README.md
================================================
# Yomikomu

* `yomikomu` gem enables to load compiled instruction sequences (bytecodes).
* `kakidasu` command compiles and stores compiled instruction sequences (bytecodes).

This gem requires Ruby 2.3.0.
Most of code is ported from `ruby/sample/iseq_load.rb`.

## Installation

Add this line to your application's Gemfile:

```ruby
gem 'yomikomu'
```

And then execute:

    $ bundle

Or install it yourself as:

    $ gem install yomikomu

## Usage

### Configuration

You can use the following configuration with environment variables.

* YOMIKOMU_STORAGE (default: fs): choose storage type.
  * 'fs' (default) stores binaries in the same directory (for examlple, `x.rb` will be compiled to `x.rb.yarb` in the same directory).
  * 'fs2' stores binaries in specific directory.
  * 'fsgz' stores files same as 'fs', but gz compressed.
  * 'fs2gz' stores files same as 'fs2', but gz compressed.
  * 'dbm' stores binaries using dbm.
  * 'flatfile' stores binaries in one flat file. Updating is not supported. Use with YOMIKOMU_AUTO_COMPILE.
* YOMIKOMU_STORAGE_DIR (default: "~/.ruby_binaries"): choose directory where binary files are stored.
* YOMIKOMU_AUTO_COMPILE (default: false): if this value is `true`, then compile all required scripts implicitly.
* YOMIKOMU_YOMIKOMANAI (default: not defined): disable all YOMIKOMU features.
* YOMIKOMU_INFO (default: not defined): show some information.
* YOMIKOMU_DEBUG (default: not defined): show more information.

### Compile and store instruction sequences

You only need to use the following command.

```
$ kakidasu [file or dir] ...
```

Compile specified file or files (*.rb) in specified directory.
If no files or directories are specified, then use "libdir" of installed Ruby.

### Load compiled binary

You only need to require `yomikomu`.

## Development

After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ko1/yomikomu.


## License

The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).



================================================
FILE: Rakefile
================================================
require "bundler/gem_tasks"
require "rake/testtask"

Rake::TestTask.new(:test) do |t|
  t.libs << "test"
  t.libs << "lib"
  t.test_files = FileList['test/**/*_test.rb']
end

task :default => :test


================================================
FILE: bin/console
================================================
#!/usr/bin/env ruby

require "bundler/setup"
require "yomikomu"

# You can add fixtures and/or initialization code here to make experimenting
# with your gem easier. You can also use a different console, if you like.

# (If you use this, don't forget to add pry to your Gemfile!)
# require "pry"
# Pry.start

require "irb"
IRB.start


================================================
FILE: bin/setup
================================================
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

bundle install

# Do any other automated setup that you need to do here


================================================
FILE: exe/kakidasu
================================================
#! /usr/bin/env ruby

$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)

require 'yomikomu'
require 'rbconfig'
require 'optparse'

verbose = false
executor = nil

opts = OptionParser.new{|o|
  o.on('--verify [original script]', 'verify mode'){
    executor = Proc.new{|file|
      raise "TODO"
    }
  }
  o.on('--remove [original script]', 'remove specific compiled iseq'){
    executor = Proc.new{|file|
      ::Yomikomu.remove_compiled_iseq(file)
    }
  }
  o.on('--remove-all', 'remove all compiled iseq'){
    ::Yomikomu.remove_all_compiled_iseq
  }
  o.on('-v', '--verbose'){
    $VERBOSE = true
    ENV['YOMIKOMU_DEBUG'] = 'true'
  }
}

executor = Proc.new{|file|
  begin
    ::Yomikomu.compile_and_store_iseq(file)
  rescue SyntaxError => e
    STDERR.puts e
  end
} unless executor # default

opts.parse!(ARGV)

paths = ARGV.empty? ? [RbConfig::CONFIG['libdir']] : ARGV

paths.each{|path|
  if File.directory?(path)
    pattern = File.join(path, '**/*.rb')
    Dir.glob(pattern){|file|
      executor.call(file)
    }
  else
    executor.call(path)
  end
}


================================================
FILE: lib/yomikomu/version.rb
================================================
module Yomikomu
  VERSION = "0.4.1"
end


================================================
FILE: lib/yomikomu.rb
================================================
# frozen_string_literal: true
require "yomikomu/version"

module Yomikomu
  STATISTICS = Hash.new(0)

  def self.prefix
    unless yomu_dir = ENV['YOMIKOMU_STORAGE_DIR']
      yomu_dir = File.expand_path("~/.ruby_binaries")
    end
    Dir.mkdir(yomu_dir) unless File.exist?(yomu_dir)
    "#{yomu_dir}/cb."
  end

  YOMIKOMU_AUTO_COMPILE = ENV['YOMIKOMU_AUTO_COMPILE'] == 'true'
  YOMIKOMU_USE_MMAP = ENV['YOMIKOMU_USE_MMAP'] == 'true'

  def self.status
    STDERR.puts "[YOMIKOMU:INFO] (pid:#{Process.pid}) " +
                ::Yomikomu::STATISTICS.map{|k, v| "#{k}: #{v}"}.join(', ')
  end

  if $VERBOSE || ENV['YOMIKOMU_INFO'] == 'true'
    def self.info
      STDERR.puts "[YOMIKOMU:INFO] (pid:#{Process.pid}) #{yield}"
    end
    at_exit{
      status
    }
  else
    def self.info
    end
  end

  if ENV['YOMIKOMU_DEBUG'] == 'true'
    def self.debug
      STDERR.puts "[YOMIKOMU:DEBUG] (pid:#{Process.pid}) #{yield}"
    end
  else
    def self.debug
    end
  end

  class NullStorage
    def load_iseq fname; end
    def compile_and_store_iseq fname; end
    def remove_compiled_iseq fname; end
  end

  class BasicStorage
    def initialize
      require 'digest/sha1'
    end

    def load_iseq fname
      iseq_key = iseq_key_name(fname)

      if compiled_iseq_exist?(fname, iseq_key) && compiled_iseq_is_younger?(fname, iseq_key)
        ::Yomikomu::STATISTICS[:loaded] += 1
        ::Yomikomu.debug{ "load #{fname} from #{iseq_key}" }
        binary = read_compiled_iseq(fname, iseq_key)
        iseq = RubyVM::InstructionSequence.load_from_binary(binary)
        # p [extra_data(iseq.path), RubyVM::InstructionSequence.load_from_binary_extra_data(binary)]
        # raise unless extra_data(iseq.path) == RubyVM::InstructionSequence.load_from_binary_extra_data(binary)
        iseq
      elsif YOMIKOMU_AUTO_COMPILE
        compile_and_store_iseq(fname, iseq_key)
      else
        ::Yomikomu::STATISTICS[:ignored] += 1
        ::Yomikomu.debug{ "ignored #{fname}" }
        nil
      end
    end

    def extra_data fname
      "SHA-1:#{::Digest::SHA1.file(fname).digest}"
    end

    def compile_and_store_iseq fname, iseq_key = iseq_key_name(fname = File.expand_path(fname))
      ::Yomikomu.debug{ "compile #{fname} into #{iseq_key}" }
      begin
        iseq = RubyVM::InstructionSequence.compile_file(fname)
        binary = iseq.to_binary(extra_data(fname))
        write_compiled_iseq(fname, iseq_key, binary)
        ::Yomikomu::STATISTICS[:compiled] += 1
        iseq
      rescue SyntaxError, RuntimeError => e
        puts "#{e}: #{fname}"
        nil
      end
    end

    # def remove_compiled_iseq fname; nil; end # should implement at sub classes

    private

    def iseq_key_name fname
      fname
    end

    # should implement at sub classes
    # def compiled_iseq_younger? fname, iseq_key; end
    # def compiled_iseq_exist? fname, iseq_key; end
    # def read_compiled_file fname, iseq_key; end
    # def write_compiled_file fname, iseq_key, binary; end
  end

  class FSStorage < BasicStorage
    def initialize
      super
    end

    def remove_compiled_iseq fname
      iseq_key = iseq_key_name(fname)
      if File.exist?(iseq_key)
        Yomikomu.debug{ "rm #{iseq_key}" }
        File.unlink(iseq_key)
      end
    end

    private

    def iseq_key_name fname
      "#{fname}.yarb" # same directory
    end

    def compiled_iseq_exist? fname, iseq_key
      File.exist?(iseq_key)
    end

    def compiled_iseq_is_younger? fname, iseq_key
      File.mtime(iseq_key) >= File.mtime(fname)
    end

    def read_compiled_iseq fname, iseq_key
      File.binread(iseq_key)
    end

    def write_compiled_iseq fname, iseq_key, binary
      File.binwrite(iseq_key, binary)
    end

    def remove_all_compiled_iseq
      raise "unsupported"
    end
  end

  class FSSGZtorage < FSStorage

  end

  class FS2Storage < FSStorage
    def initialize
      super

      require 'fileutils'
      @dir = Yomikomu.prefix + "files"
      unless File.directory?(@dir)
        FileUtils.mkdir_p(@dir)
      end
    end

    def iseq_key_name fname
      File.join(@dir, fname.gsub(/[^A-Za-z0-9\._-]/){|c| '%02x' % c.ord} + '.yarb') # special directory
    end

    def remove_all_compiled_iseq
      Dir.glob(File.join(@dir, '**/*.yarb')){|path|
        Yomikomu.debug{ "rm #{path}" }
        FileUtils.rm(path)
      }
    end
  end

  module GZFileStorage
    def initialize
      require 'zlib'
      super
    end

    def iseq_key_name fname
      super + '.gz'
    end

    def read_compiled_iseq fname, iseq_key
      Zlib::GzipReader.open(iseq_key){|f|
        f.read
      }
    end

    def write_compiled_iseq fname, iseq_key, binary
      Zlib::GzipWriter.open(iseq_key){|f|
        f.write(binary)
      }
    end
  end

  class FSGZStorage < FSStorage
    include GZFileStorage
  end

  class FS2GZStorage < FS2Storage
    include GZFileStorage
  end

  if YOMIKOMU_USE_MMAP
    require 'mmapped_string'
    Yomikomu.info{ "[RUBY_YOMIKOMU] use mmap" }

    module MMapFile
      def read_compiled_iseq fname, iseq_key
        MmappedString.open(iseq_key)
      end
    end

    class FSStorage
      prepend MMapFile
    end

    class FS2Storage
      prepend MMapFile
    end
  end

  class DBMStorage < BasicStorage
    def initialize
      super
      require 'dbm'
      @db = DBM.open(Yomikomu.prefix + 'db')
    end

    def remove_compiled_iseq fname
      @db.delete fname
    end

    private

    def date_key_name fname
      "date.#{fname}"
    end

    def iseq_key_name fname
      "body.#{fname}"
    end

    def compiled_iseq_exist? fname, iseq_key
      @db.has_key? iseq_key
    end

    def compiled_iseq_is_younger? fname, iseq_key
      date_key = date_key_name(fname)
      if @db.has_key? date_key
        @db[date_key].to_i >= File.mtime(fname).to_i
      end
    end

    def read_compiled_iseq fname, iseq_key
      @db[iseq_key]
    end

    def write_compiled_iseq fname, iseq_key, binary
      date_key = date_key_name(fname)
      @db[iseq_key] = binary
      @db[date_key] = Time.now.to_i
    end
  end

  class FlatFileStorage < BasicStorage
    def index_path
      Yomikomu.prefix + 'ff_index'
    end

    def data_path
      Yomikomu.prefix + 'ff_data'
    end

    def initialize
      super
      require 'fileutils'

      @updated = false

      if File.exist?(index_path)
        open(index_path, 'rb'){|f| @index = Marshal.load(f)}
      else
        @index = {}
        open(data_path, 'w'){} # touch
      end

      @data_file = open(data_path, 'a+b')

      at_exit{
        if @updated
          open(index_path, 'wb'){|f| Marshal.dump(@index, f)}
          Yomikomu.info{'FlatFile: update'}
        end
      }
    end

    def remove_compiled_iseq fname
      raise 'unsupported'
    end

    private

    def compiled_iseq_exist? fname, iseq_key
      @index[iseq_key]
    end

    def compiled_iseq_is_younger? fname, iseq_key
      offset, size, date = @index[iseq_key]
      date.to_i >= File.mtime(fname).to_i
    end

    def read_compiled_iseq fname, iseq_key
      offset, size, date = @index[iseq_key]
      @data_file.pos = offset
      binary = @data_file.read(size)
      raise "size is not match" if binary.size != size
      binary
    end

    def write_compiled_iseq fname, iseq_key, binary
      raise "compiled binary for #{fname} already exists. flatfile does not support overwrite." if compiled_iseq_exist?(fname, iseq_key)

      @data_file.seek 0, IO::SEEK_END
      offset = @data_file.tell
      size = binary.size
      date = Time.now.to_i
      @data_file.write(binary)
      @index[iseq_key] = [offset, size, date]

      @updated = true
    end
  end

  class FlatFileGZStorage < FlatFileStorage
    def index_path
      super + '_gz'
    end

    def data_path
      super + '_gz'
    end

    def initialize
      super
      require 'zlib'
    end

    def read_compiled_iseq fname, iseq_key
      binary = super
      Zlib::Inflate.inflate(binary)
    end

    def write_compiled_iseq fname, iseq_key, binary
      binary = Zlib::Deflate.deflate(binary)
      super(fname, iseq_key, binary)
    end
  end

  def self.compile_and_store_iseq fname
    STORAGE.compile_and_store_iseq fname
  end

  def self.remove_compiled_iseq fname
    STORAGE.remove_compiled_iseq fname
  end

  def self.remove_all_compiled_iseq
    STORAGE.remove_all_compiled_iseq
  end

  def self.verify_compiled_iseq fname
    STORAGE.verify_compiled_iseq fname
  end

  # select storage
  STORAGE = case storage = ENV['YOMIKOMU_STORAGE']
            when 'fs'
              FSStorage.new
            when 'fsgz'
              FSGZStorage.new
            when 'fs2'
              FS2Storage.new
            when 'fs2gz'
              FS2GZStorage.new
            when 'dbm'
              DBMStorage.new
            when 'flatfile'
              FlatFileStorage.new
            when 'flatfilegz'
              FlatFileGZStorage.new
            when 'null'
              NullStorage.new
            when nil
              FSStorage.new
            else
              raise "Unknown storage type: #{storage}"
            end

  Yomikomu.info{ "[RUBY_YOMIKOMU] use #{STORAGE.class}" }
end

class RubyVM::InstructionSequence
  if ENV['YOMIKOMU_YOMIKOMANAI'] != 'true'
    def self.load_iseq fname
      ::Yomikomu::STORAGE.load_iseq(fname)
    end
  end
end


================================================
FILE: test/test_helper.rb
================================================
$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)

require 'tmpdir'

tmpdir = Dir.mktmpdir('yomikomu')
ENV['YOMIKOMU_STORAGE_DIR'] = tmpdir
ENV['YOMIKOMU_STORAGE'] = 'fs2'
require 'yomikomu'
require 'minitest/autorun'


================================================
FILE: test/x.rb
================================================

def yomikomi_test_hello(name)
  "hello #{name}"
end


================================================
FILE: test/yomikomu_test.rb
================================================
require 'test_helper'

class YomikomuTest < Minitest::Test
  def test_that_it_has_a_version_number
    refute_nil ::Yomikomu::VERSION
  end

  def test_compile_file
    assert_equal 0, ::Yomikomu::STATISTICS[:loaded]
    assert_equal 0, ::Yomikomu::STATISTICS[:compiled]
    ignored = ::Yomikomu::STATISTICS[:ignored]

    Yomikomu::compile_and_store_iseq File.join(__dir__, 'x.rb')
    assert_equal 1, ::Yomikomu::STATISTICS[:compiled]

    load_file = File.join(__dir__, 'x.rb')

    load load_file

    assert_equal("hello world", yomikomi_test_hello("world"))
    assert_equal 1, ::Yomikomu::STATISTICS[:loaded]
    assert_equal 1, ::Yomikomu::STATISTICS[:compiled]

    Yomikomu::remove_all_compiled_iseq
    load load_file

    assert_equal("hello world", yomikomi_test_hello("world"))
    assert_equal 1, ::Yomikomu::STATISTICS[:loaded]
    assert_equal 1, ::Yomikomu::STATISTICS[:compiled]

    Yomikomu::compile_and_store_iseq File.join(__dir__, 'x.rb')
    assert_equal 2, ::Yomikomu::STATISTICS[:compiled]

    Yomikomu::remove_compiled_iseq load_file
    load load_file

    assert_equal("hello world", yomikomi_test_hello("world"))
    assert_equal 1, ::Yomikomu::STATISTICS[:loaded]
    assert_equal 2, ::Yomikomu::STATISTICS[:compiled]
  end
end


================================================
FILE: yomikomu.gemspec
================================================
# coding: utf-8
lib = File.expand_path('../lib', __FILE__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require 'yomikomu/version'

Gem::Specification.new do |spec|
  spec.name          = "yomikomu"
  spec.version       = Yomikomu::VERSION
  spec.authors       = ["Koichi Sasada"]
  spec.email         = ["ko1@atdot.net"]

  spec.summary       = %q{Dump compiled iseq by binary (kakidasu) and load binary (yomidasu).}
  spec.description   = %q{Dump compiled iseq by binary (kakidasu) and load binary (yomidasu).}
  spec.homepage      = "http://github.com/ko1/yomikomu"
  spec.license       = "MIT"

  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
  spec.bindir        = "exe"
  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
  spec.require_paths = ["lib"]

  spec.required_ruby_version = '>= 2.3.0'

  spec.add_development_dependency "bundler", "~> 1.10"
  spec.add_development_dependency "rake", "~> 10.0"
  spec.add_development_dependency "minitest"
end