Repository: choonkeat/attache Branch: master Commit: eaee2aca211b Files: 46 Total size: 103.5 KB Directory structure: gitextract_4787py6e/ ├── .gitignore ├── .rspec ├── .travis.yml ├── CONTRIBUTING.md ├── Gemfile ├── Guardfile ├── LICENSE ├── Procfile ├── README.md ├── Rakefile ├── app.json ├── attache.gemspec ├── config/ │ ├── puma.rb │ └── vhost.example.yml ├── config.ru ├── docker/ │ ├── Dockerfile │ └── bundler_geminstaller_install_with_timeout.rb ├── exe/ │ └── attache ├── lib/ │ ├── attache/ │ │ ├── backup.rb │ │ ├── base.rb │ │ ├── delete.rb │ │ ├── download.rb │ │ ├── file_response_body.rb │ │ ├── job.rb │ │ ├── resize_job.rb │ │ ├── tasks.rb │ │ ├── tus/ │ │ │ └── upload.rb │ │ ├── tus.rb │ │ ├── upload.rb │ │ ├── upload_url.rb │ │ ├── version.rb │ │ └── vhost.rb │ └── attache.rb ├── public/ │ ├── index.html │ └── vendor/ │ └── roboto/ │ └── Apache License.txt └── spec/ ├── fixtures/ │ └── sample.txt ├── lib/ │ └── attache/ │ ├── backup_spec.rb │ ├── delete_spec.rb │ ├── download_spec.rb │ ├── resize_job_spec.rb │ ├── tus/ │ │ └── upload_spec.rb │ ├── tus_spec.rb │ ├── upload_spec.rb │ ├── upload_url_spec.rb │ └── vhost_spec.rb └── spec_helper.rb ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ vhost.yml ================================================ FILE: .rspec ================================================ --color --require spec_helper --format documentation ================================================ FILE: .travis.yml ================================================ language: ruby bundler_args: --retry=3 --jobs=8 --no-deployment cache: bundler sudo: false rvm: - 2.2.3 matrix: fast_finish: true ================================================ FILE: CONTRIBUTING.md ================================================ # Contributing 1. Fork it ( http://github.com/choonkeat/attache/fork ) 2. Create your feature branch (`git checkout -b my-new-feature`) 3. Commit your changes (`git commit -am 'Add some feature'`) 4. Push to the branch (`git push origin my-new-feature`) 5. Create new Pull Request ================================================ FILE: Gemfile ================================================ source 'https://rubygems.org' ruby "2.2.3" gemspec ================================================ FILE: Guardfile ================================================ guard :rspec, cmd: "bundle exec rspec" do watch(%r{^spec/.+_spec\.rb$}) watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" } end ================================================ FILE: LICENSE ================================================ The MIT License (MIT) Copyright (c) 2015 Chew Choon Keat Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: Procfile ================================================ web: bundle exec puma -C config/puma.rb worker: bundle exec sidekiq -e production -q attache_vhost_jobs -r ./lib/attache.rb ================================================ FILE: README.md ================================================ # attache [![Gem Version](https://badge.fury.io/rb/attache.svg)](https://badge.fury.io/rb/attache) [![Build Status](https://travis-ci.org/choonkeat/attache.svg?branch=master)](https://travis-ci.org/choonkeat/attache) ## But why? If you're interested in the "why", checkout [my slides](http://www.slideshare.net/choonkeat/file-upload-2015) and [the blog post](http://blog.choonkeat.com/weblog/2015/10/file-uploads-2015.html). Your app can easily support - dynamic resize of images (no predefined styles in your app) - all file types since attache let apps [display non-image files as icons through ``](https://github.com/choonkeat/attache/pull/28) - resumeable upload over unreliable (mobile) networks [using TUS protocol](https://github.com/choonkeat/attache/pull/10) ## Run an instance #### Heroku You can run your own instance on your own Heroku server [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy) #### Docker ``` docker run -it -p 9292:5000 --rm attache/attache ``` Also, see [Deploying Attache on Digital Ocean using Docker](https://github.com/choonkeat/attache/wiki/Deploying-Attache-on-Digital-Ocean-using-Docker) #### RubyGem You can install the gem and then execute `attache` command ``` gem install attache attache start -c web=1 -p 9292 ``` NOTE: some config files will be written into your current directory ``` . ├── Procfile ├── config │   ├── puma.rb │   └── vhost.yml └── config.ru ``` #### Bundler You can also use bundler to manage the gem; add this into your `Gemfile` ``` gem 'attache' ``` then execute ``` bundle install bundle exec attache start -c web=1 -p 9292 ``` NOTE: some config files will be written into your current directory (see RubyGems above) #### Source code You can checkout the source code and run it like a regular [a Procfile-based app](https://ddollar.github.io/foreman/): ``` git clone https://github.com/choonkeat/attache.git cd attache bundle install foreman start -c web=1 -p 9292 ``` See [foreman](https://github.com/ddollar/foreman) for more details. ## Configuration `LOCAL_DIR` is where your local disk cache will be. By default, attache will use a system assigned temporary directory which may not be the same everytime you run attache. `CACHE_SIZE_BYTES` determines how much disk space will be used for the local disk cache. If the size of cache exceeds, least recently used files will be evicted after `CACHE_EVICTION_INTERVAL_SECONDS` duration. #### Asynchronous delete By default `attache` will delete files from cloud storage using the lightweight, async processing library [sucker_punch](https://github.com/brandonhilkert/sucker_punch). This requires no additional setup (read: 1x free dyno). However if you prefer a more durable queue for reliable uploads, configuring `REDIS_PROVIDER` or `REDIS_URL` will switch `attache` to use a `redis` queue instead, via `sidekiq`. [Read Sidekiq's documentation](https://github.com/mperham/sidekiq/wiki/Using-Redis#using-an-env-variable) for details on these variables. If for some reason you'd want the cloud storage delete to be synchronous, set `INLINE_JOB=1` instead. #### Virtual Host Cloud Storage `attache` uses a different config (and backup files into a different cloud service) depending on the request hostname that it was accessed by. This means a single attache server can be the workhorse for different apps. Refer to `config/vhost.example.yml` file for configuration details. At boot time, `attache` server will first look at `VHOST` environment variable. If that is missing, it will load the content of `config/vhost.yml`. If neither exist, the `attache` server run in development mode; uploaded files are only stored locally and may be evicted to free up disk space. If you do not want to write down sensitive information like aws access key and secrets into a `config/vhost.yml` file, you can convert the entire content into `json` format and assign it to the `VHOST` environment variable instead. ``` # bash export VHOST=$(bundle exec rake attache:vhost) # heroku heroku config:set VHOST=$(bundle exec rake attache:vhost) ``` #### Virtual Host Authorization By default `attache` will accept uploads and delete requests from any client. Set `SECRET_KEY` to ensure attache only receives upload (and delete commands) from your own app. To most app developers *using* attache in your rails app through a library like [attache-rails gem](https://github.com/choonkeat/attache-rails), how this work may not matter. But if you are developing attache itself or writing a client library for attache, then read on. #### Virtual Host Authorization (Developer) When `SECRET_KEY` is set, `attache` will require a valid `hmac` parameter in the upload request. Upload and Delete requests will be refused with `HTTP 401` error unless the `hmac` is correct. The additional parameters required for authorized request are: * `uuid` is a uuid string * `expiration` is a unix timestamp of a future time. the significance is, if the timestamp has passed, the upload will be regarded as invalid * `hmac` is the `HMAC-SHA1` of the `SECRET_KEY` and the concatenated value of `uuid` and `expiration` i.e. ``` ruby hmac = OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha1'), SECRET_KEY, uuid + expiration) ``` ## APIs The attache server is a reference implementation of these interfaces. If you write your own server, [compatibility can be verified by running a test suite](https://github.com/choonkeat/attache_api#testing-against-an-attache-compatible-server). #### Upload Users will upload files directly into the `attache` server from their browser, bypassing the main app. > ``` > PUT /upload?file=image123.jpg > ``` > file content is the http request body The main app front end will receive a unique `path` for each uploaded file - the only information to store in the main app database. > ``` > {"path":"pre/fix/image123.jpg","content_type":"image/jpeg","geometry":"1920x1080"} > ``` > json response from attache after upload. ##### Upload by url > ``` > GET /upload_url?url=https://example.com/logo.png > ``` Attache will download the file from `url` supplied and uploads it through the regular `/upload` handler. So be expecting the same json response after upload. works with `GET`, `POST`, `PUT`. Data URIs (aka base64 encoded file binaries) can also be uploaded to the same `/upload_url` endpoint through the same `url` parameter. #### Download Whenever the main app wants to display the uploaded file, constrained to a particular size, it will use a helper method provided by the `attache` lib. e.g. `embed_attache(path)` which will generate the necessary, barebones markup. > ``` > > ``` > use [the imagemagick resize syntax](http://www.imagemagick.org/Usage/resize/) to specify the desired output. > > make sure to `escape` the geometry string. > e.g. for a hard crop of `50x50#`, the url should be `50x50%23` > > ``` > > ``` > requesting for a geometry of `original` will return the uploaded file. this works well for non-image file uploads. > requesting for a geometry of `remote` will skip the local cache and serve from cloud storage. * Attache keeps the uploaded file in the local harddisk (a temp directory) * Attache will also upload the file into cloud storage if `FOG_CONFIG` is set * If the local file does not exist for some reason (e.g. cleared cache), it will download from cloud storage and store it locally * When a specific size is requested, it will generate the resized file based on the local file and serve it in the http response * If cloud storage is defined, local disk cache will store up to a maximum of `CACHE_SIZE_BYTES` bytes. By default `CACHE_SIZE_BYTES` will 80% of available diskspace #### Delete > ``` > DELETE /delete > paths=image1.jpg%0Aprefix2%2Fimage2.jpg%0Aimage3.jpg > ``` Removing 1 or more files from the local cache and remote storage can be done via a http `POST` or `DELETE` request to `/delete`, with a `paths` parameter in the request body. The `paths` value should be delimited by the newline character, aka `\n`. In the example above, 3 files will be requested for deletion: `image1.jpg`, `prefix2/image2.jpg`, and `image3.jpg`. #### Backup > ``` > POST /backup > paths=image1.jpg%0Aprefix2%2Fimage2.jpg%0Aimage3.jpg > ``` This feature might be known as `promote` in other file upload solutions. `attache` allows client app to `backup` uploaded images to another bucket for longer term storage. Copying 1 or more files from the default remote storage to the backup remote storage (backup) can be done via a http `POST` request to `/backup`, with a `paths` parameter in the request body. The `paths` value should be delimited by the newline character, aka `\n`. In the example above, 3 files will be requested for backup: `image1.jpg`, `prefix2/image2.jpg`, and `image3.jpg`. If backup remote storage is not configured, this API call will be a noop. If configured, the backup storage must be accessible by the same credentials as default cloud storage as the system. Please refer to the `BACKUP_CONFIG` configuration illustrated in `config/vhost.example.yml` file in this repository. By default, `backup` operation is performed synchronously. Set `BACKUP_ASYNC` environment variable to make it follow the same synchronicity as `delete` The main reason to configure a backup storage is to make the default cloud storage auto expire files; mitigating [abuse](https://github.com/choonkeat/attache/issues/13). You should consult the documentation of your cloud storage provider on how to setup auto expiry, e.g. [here](https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/) or [here](https://cloud.google.com/storage/docs/lifecycle) ## License MIT ================================================ FILE: Rakefile ================================================ if ENV['RACK_ENV'] == 'production' # Heroku # https://gist.github.com/Geesu/d0b58488cfae51f361c6 namespace :assets do task 'precompile' do puts "Not applicable" end end else require "bundler/gem_tasks" require 'rspec/core/rake_task' require 'attache/tasks' RSpec::Core::RakeTask.new(:spec) task :default => :spec end ================================================ FILE: app.json ================================================ { "name": "attache server", "description": "Image server", "repository": "https://github.com/choonkeat/attache", "keywords": ["ruby", "rack", "image", "resize", "direct", "upload"], "env": { "REMOTE_DIR": "attache" } } ================================================ FILE: attache.gemspec ================================================ $:.push File.expand_path("../lib", __FILE__) # Maintain your gem's version: require "attache/version" # Describe your gem and declare its dependencies: Gem::Specification.new do |s| s.name = "attache" s.version = Attache::VERSION s.authors = ["choonkeat"] s.email = ["choonkeat@gmail.com"] s.homepage = "https://github.com/choonkeat/attache" s.summary = "Image server for everybody" s.description = "Standalone rack app to manage files onbehalf of your app" s.license = "MIT" s.files = Dir["{app,lib}/**/*", "MIT-LICENSE", "Rakefile", "README.md", 'exe/**/*', "config/vhost.example.yml", "config/puma.rb", "config.ru", 'public/**/*'] s.bindir = 'exe' s.executables = ['attache'] s.add_runtime_dependency 'rack', '~> 1.6' s.add_runtime_dependency 'activesupport' s.add_runtime_dependency 'paperclip', '~> 4.3' s.add_runtime_dependency 'puma', '~> 2.14' s.add_runtime_dependency 'net-ssh' s.add_runtime_dependency 'fog', '~> 1.34' s.add_runtime_dependency 'excon', '~> 0.45' s.add_runtime_dependency 'sys-filesystem', '~> 0' s.add_runtime_dependency 'disk_store', '~> 0' s.add_runtime_dependency 'celluloid', '< 0.17' # 0.17 has compatibility issues with disk_store s.add_runtime_dependency 'foreman', '~> 0' s.add_runtime_dependency 'connection_pool', '~> 2.2' s.add_runtime_dependency 'sidekiq', '~> 3.4' s.add_runtime_dependency 'sucker_punch', '~> 1.5' # single-process Ruby asynchronous processing library s.add_development_dependency 'rspec', '~> 3.2' s.add_development_dependency 'shoulda', '~> 3.5' s.add_development_dependency 'guard-rspec', '~> 4.6' end ================================================ FILE: config/puma.rb ================================================ workers Integer(ENV['PUMA_WORKERS'] || 1) threads Integer(ENV['MIN_THREADS'] || 1), Integer(ENV['MAX_THREADS'] || 16) preload_app! rackup DefaultRackup port ENV['PORT'] || 3000 environment ENV['RACK_ENV'] || 'development' ================================================ FILE: config/vhost.example.yml ================================================ # This is an example file. You can copy this file as `vhost.yml` and edit # the content with the correct values. # This section will only take effect if a request is made to `google.lvh.me:9292` "google.lvh.me:9292": "SECRET_KEY": CHANGEME # this is the shared secret between your app and this attache server "REMOTE_DIR": CHANGEME # this is the root directory to use in the `bucket`; omit to use root "GEOMETRY_WHITELIST": # this limits the type of `geometry` we resize to; optional - "100x100" - "1024>" "FOG_CONFIG": # "provider": Google # refer to `fog.io/storage` documentation "google_storage_access_key_id": CHANGEME # "google_storage_secret_access_key": CHANGEME # "bucket": CHANGEME # This `bucket` key is not standard Fog config. BUT attache server needs it # This section will only take effect if a request is made to `aws.example.com` "aws.example.com": "SECRET_KEY": CHANGEME "FOG_CONFIG": "provider": AWS "aws_access_key_id": CHANGEME "aws_secret_access_key": CHANGEME "bucket": CHANGEME "region": us-west-1 "BACKUP_CONFIG": "bucket": CHANGEME_BAK # only supports 1 key: `bucket` # This section will only take effect if a request is made to `localhost:9292` "localhost:9292": # This section will apply if a request did not match anything else "0.0.0.0": ================================================ FILE: config.ru ================================================ require 'attache' use Attache::Delete use Attache::UploadUrl use Attache::Upload use Attache::Download use Attache::Tus::Upload use Attache::Backup use Rack::Static, urls: ["/"], root: Attache.publicdir, index: "index.html" run proc {|env| [200, {}, []] } ================================================ FILE: docker/Dockerfile ================================================ FROM ruby:2.2 RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y imagemagick ghostscript RUN curl -sSL https://raw.githubusercontent.com/choonkeat/attache/master/docker/bundler_geminstaller_install_with_timeout.rb | ruby RUN useradd -d /app -m app && \ chown -R app /usr/local/bundle USER app RUN mkdir -p /app/src WORKDIR /app/src RUN curl -sSL http://johnvansickle.com/ffmpeg/releases/ffmpeg-release-32bit-static.tar.xz | tar -xJv ENV PATH "$PATH:/app/src/ffmpeg-2.8.3-32bit-static" RUN echo 'source "https://rubygems.org"' > Gemfile && \ echo 'gem "attache", ">= 2.3.0"' >> Gemfile && bundle && \ gem install --no-ri --no-rdoc attache --version '>= 2.3.0' EXPOSE 5000 CMD ["attache", "start", "-c", "web=1"] ================================================ FILE: docker/bundler_geminstaller_install_with_timeout.rb ================================================ # Usage: # ruby bundler_geminstaller_install_with_timeout.rb target = `which bundle`.chomp *old_lines, last_line = IO.read(target).split(/[\r\n]+/) if (old_lines.grep(/install_with_timeout/)).empty? new_line = DATA.read.strip combined = (old_lines + [new_line, last_line]).join($/) open(target, "w") {|f| f.write(combined) } puts "installed." else puts "already installed." end __END__ require "timeout" require "rubygems/installer" Gem::Installer.class_eval do def install_with_timeout puts "Gem install_with_timeout..." Timeout.timeout(Integer(ENV.fetch("GEM_INSTALL_TIMEOUT", 60))) { install_without_timeout } rescue Timeout::Error @tries = @tries.to_i + 1 raise unless @tries < 5 STDERR.puts "Gem timed out #{$!} (#{@tries})..." retry end alias :install_without_timeout :install alias :install :install_with_timeout end require "bundler/installer/gem_installer" Bundler::GemInstaller.class_eval do def install_with_timeout puts "Bundler install_with_timeout..." Timeout.timeout(Integer(ENV.fetch("GEM_INSTALL_TIMEOUT", 60))) { install_without_timeout } rescue Timeout::Error @tries = @tries.to_i + 1 raise unless @tries < 5 STDERR.puts "Bundler timed out #{$!} (#{@tries})..." retry end alias :install_without_timeout :install alias :install :install_with_timeout end ================================================ FILE: exe/attache ================================================ #!/bin/env ruby require 'fileutils' # attache config if ENV['VHOST'] puts "Using VHOST env" elsif File.exists?("config/vhost.yml") puts "Using config/vhost.yml" else FileUtils.mkdir_p 'config' FileUtils.copy File.expand_path("../config/vhost.example.yml", File.dirname(__FILE__)), 'config/vhost.yml' puts "Initialized config/vhost.yml" end # puma config if File.exists?("config/puma.rb") puts "Using config/puma.rb" else FileUtils.mkdir_p 'config' FileUtils.copy File.expand_path("../config/puma.rb", File.dirname(__FILE__)), 'config/puma.rb' puts "Initialized config/puma.rb" end # procfile if File.exists?("Procfile") puts "Using Procfile" else open("Procfile", "w") do |f| f.write <<-EOM.gsub(/^\s+/, '') web: bundle exec puma -C config/puma.rb worker: bundle exec sidekiq -e production -q attache_vhost_jobs -r #{File.expand_path("../lib/attache.rb", File.dirname(__FILE__))} EOM end puts "Initialized Procfile" end # rakefile if File.exists?("Rakefile") puts "Using Rakefile" else open("Rakefile", "w") do |f| f.write <<-EOM.gsub(/^\s+/, '') require 'attache/tasks' EOM end puts "Initialized Rakefile" end # rack config if File.exists?("config.ru") puts "Using config.ru" else FileUtils.copy File.expand_path("../config.ru", File.dirname(__FILE__)), 'config.ru' puts "Initialized config.ru" end case ARGV.first when 'start' require "foreman/cli" Foreman::CLI.start else puts "" puts "Setup complete: run `foreman start` to begin" puts "" end ================================================ FILE: lib/attache/backup.rb ================================================ class Attache::Backup < Attache::Base def initialize(app) @app = app end def _call(env, config) case env['PATH_INFO'] when '/backup' request = Rack::Request.new(env) params = request.params return config.unauthorized unless config.authorized?(params) if config.storage && config.bucket sync_method = (ENV['BACKUP_ASYNC'] ? :async : :send) threads = [] params['paths'].to_s.split("\n").each do |relpath| threads << Thread.new do Attache.logger.info "BACKUP remote #{relpath}" config.send(sync_method, :backup_file, relpath: relpath) end end threads.each(&:join) end [200, config.headers_with_cors, []] else @app.call(env) end end end ================================================ FILE: lib/attache/base.rb ================================================ class Attache::Base def call(env) if vhost = vhost_for(request_hostname(env)) dup._call(env, vhost) else @app.call(env) end rescue Timeout::Error Attache.logger.error $@ Attache.logger.error $! Attache.logger.error "ERROR 503 #{env['PATH_INFO']} REFERER #{env['HTTP_REFERER'].inspect}" [503, { 'X-Exception' => $!.to_s }, []] rescue Exception Attache.logger.error $@ Attache.logger.error $! Attache.logger.error "ERROR 500 #{env['PATH_INFO']} REFERER #{env['HTTP_REFERER'].inspect}" [500, { 'X-Exception' => $!.to_s }, []] end def vhost_for(host) Attache::VHost.new(Attache.vhost[host] || Attache.vhost['0.0.0.0']) end def request_hostname(env) env['HTTP_X_FORWARDED_HOST'] || env['HTTP_HOST'] || "unknown.host" end def content_type_of(fullpath) Paperclip::ContentTypeDetector.new(fullpath).detect rescue Paperclip::Errors::NotIdentifiedByImageMagickError # best effort only end def geometry_of(fullpath) Paperclip::Geometry.from_file(fullpath).tap(&:auto_orient).to_s rescue Paperclip::Errors::NotIdentifiedByImageMagickError # best effort only end def filesize_of(fullpath) File.stat(fullpath).size end def params_of(env) env['QUERY_STRING'].to_s.split('&').inject({}) do |sum, pair| k, v = pair.split('=').collect {|s| CGI.unescape(s) } sum.merge(k => v) end end def path_of(cachekey) Attache.cache.send(:key_file_path, cachekey) end def rack_response_body_for(file) Attache::FileResponseBody.new(file) end def generate_relpath(basename) File.join(*SecureRandom.hex.scan(/\w\w/), basename) end def json_of(relpath, cachekey, vhost) filepath = path_of(cachekey) json = { path: relpath, content_type: content_type_of(filepath), geometry: geometry_of(filepath), bytes: filesize_of(filepath), } if vhost && vhost.secret_key content = json.sort.collect {|k,v| "#{k}=#{v}" }.join('&') json['signature'] = OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha1'), vhost.secret_key, content) end json.to_json end end ================================================ FILE: lib/attache/delete.rb ================================================ class Attache::Delete < Attache::Base def initialize(app) @app = app end def _call(env, config) case env['PATH_INFO'] when '/delete' request = Rack::Request.new(env) params = request.params return config.unauthorized unless config.authorized?(params) threads = [] params['paths'].to_s.split("\n").each do |relpath| if Attache.cache threads << Thread.new do Attache.logger.info "DELETING local #{relpath}" cachekey = File.join(request_hostname(env), relpath) Attache.cache.delete(cachekey) end end if config.storage && config.bucket threads << Thread.new do Attache.logger.info "DELETING remote #{relpath}" config.async(:storage_destroy, relpath: relpath) end end if config.backup threads << Thread.new do Attache.logger.info "DELETING backup #{relpath}" config.backup.async(:storage_destroy, relpath: relpath) end end end threads.each(&:join) [200, config.headers_with_cors, []] else @app.call(env) end end end ================================================ FILE: lib/attache/download.rb ================================================ require 'connection_pool' class Attache::Download < Attache::Base RESIZE_JOB_POOL = ConnectionPool.new(JSON.parse(ENV.fetch('RESIZE_POOL') { '{ "size": 2, "timeout": 60 }' }).symbolize_keys) { Attache::ResizeJob.new } def initialize(app) @app = app @mutexes = {} end def _call(env, config) case env['PATH_INFO'] when %r{\A/view/} vhosts = {} vhosts[ENV.fetch('REMOTE_GEOMETRY') { 'remote' }] = config.storage && config.bucket && config vhosts[ENV.fetch('BACKUP_GEOMETRY') { 'backup' }] = config.backup parse_path_info(env['PATH_INFO']['/view/'.length..-1]) do |dirname, geometry, basename, relpath| unless config.try(:geometry_whitelist).blank? || config.geometry_whitelist.include?(geometry) return [415, config.download_headers, ["#{geometry} is not supported"]] end if vhost = vhosts[geometry] headers = vhost.download_headers.merge({ 'Location' => vhost.storage_url(relpath: relpath), 'Cache-Control' => 'private, no-cache', }) return [302, headers, []] end thumbnail = case geometry when 'original', *vhosts.keys get_original_file(relpath, vhosts, env) else get_thumbnail_file(geometry, basename, relpath, vhosts, env) end return [404, config.download_headers, []] if thumbnail.try(:size).to_i == 0 headers = { 'Content-Type' => content_type_of(thumbnail.path), }.merge(config.download_headers) [200, headers, rack_response_body_for(thumbnail)] end else @app.call(env) end end private def parse_path_info(geometrypath) parts = geometrypath.split('/') basename = CGI.unescape parts.pop geometry = CGI.unescape parts.pop dirname = parts.join('/') relpath = File.join(dirname, basename) yield dirname, geometry, basename, relpath end def synchronize(key, &block) mutex = @mutexes[key] ||= Mutex.new mutex.synchronize(&block) ensure @mutexes.delete(key) end def get_thumbnail_file(geometry, basename, relpath, vhosts, env) cachekey = File.join(request_hostname(env), relpath, geometry) synchronize(cachekey) do tempfile = nil Attache.cache.fetch(cachekey) do Attache.logger.info "[POOL] new job" tempfile = RESIZE_JOB_POOL.with do |job| job.perform(geometry, basename, relpath, vhosts, env) do # opens up possibility that job implementation # does not require we download original file prior get_original_file(relpath, vhosts, env) end end end.tap { File.unlink(tempfile.path) if tempfile.try(:path) } end end def get_original_file(relpath, vhosts, env) cachekey = File.join(request_hostname(env), relpath) synchronize(cachekey) do Attache.cache.fetch(cachekey) do name_with_vhost_pairs = vhosts.inject({}) { |sum,(k,v)| (v ? sum.merge(k => v) : sum) } get_first_result_present_async(name_with_vhost_pairs.collect {|name, vhost| lambda { Thread.handle_interrupt(BasicObject => :on_blocking) { begin Attache.logger.info "[POOL] looking for #{name} #{relpath}..." vhost.storage_get(relpath: relpath).tap do |v| Attache.logger.info "[POOL] found #{name} #{relpath} = #{v.inspect}" end rescue Exception Attache.logger.error $! Attache.logger.error $@ Attache.logger.info "[POOL] not found #{name} #{relpath}" nil end } } }) end end rescue Exception # Errno::ECONNREFUSED, OpenURI::HTTPError, Excon::Errors, Fog::Errors::Error Attache.logger.error "ERROR REFERER #{env['HTTP_REFERER'].inspect}" nil end # Ref https://gist.github.com/sferik/39831f34eb87686b639c#gistcomment-1652888 # a bit more complicated because we *want* to ignore falsey result def get_first_result_present_async(lambdas) return if lambdas.empty? # queue.pop will never happen queue = Queue.new threads = lambdas.shuffle.collect { |code| Thread.new { queue << [Thread.current, code.call] } } until (item = queue.pop).last do thread, _ = item thread.join # we could be popping `queue` before thread exited break unless threads.any?(&:alive?) || queue.size > 0 end threads.each(&:kill) _, result = item result end end ================================================ FILE: lib/attache/file_response_body.rb ================================================ class Attache::FileResponseBody def initialize(file, range_start = nil, range_end = nil) @file = file @range_start = range_start || 0 @range_end = range_end || File.size(@file.path) end # adapted from rack/file.rb def each @file.seek(@range_start) remaining_len = @range_end while remaining_len > 0 part = @file.read([8192, remaining_len].min) break unless part remaining_len -= part.length yield part end end end ================================================ FILE: lib/attache/job.rb ================================================ class Attache::Job RETRY_DURATION = ENV.fetch('CACHE_EVICTION_INTERVAL_SECONDS') { 60 }.to_i / 3 def perform(method, env, args) config = Attache::VHost.new(env) config.send(method, args.symbolize_keys) rescue Exception Attache.logger.error $@ Attache.logger.error $! Attache.logger.error [method, args].inspect self.class.perform_in(RETRY_DURATION, method, env, args) end # Background processing setup if defined?(::SuckerPunch::Job) include ::SuckerPunch::Job def later(sec, *args) after(sec) { perform(*args) } end def self.perform_async(*args) self.new.async.perform(*args) end def self.perform_in(duration, *args) self.new.async.later(duration, *args) end else include Sidekiq::Worker sidekiq_options :queue => :attache_vhost_jobs sidekiq_retry_in {|count| RETRY_DURATION} # uncaught exception, retry after RETRY_DURATION end end ================================================ FILE: lib/attache/resize_job.rb ================================================ require 'digest/sha1' require 'stringio' class Attache::ResizeJob def perform(target_geometry_string, basename, relpath, vhosts, env, t = Time.now) closed_file = yield return StringIO.new if closed_file.try(:size).to_i == 0 extension = basename.split(/\W+/).last Attache.logger.info "[POOL] start" return make_nonimage_preview(closed_file, basename) if ['pdf', 'txt'].include?(extension.to_s.downcase) thumbnail = thumbnail_for(closed_file: closed_file, target_geometry_string: target_geometry_string, extension: extension) thumbnail.instance_variable_set('@basename', make_safe_filename(thumbnail.instance_variable_get('@basename'))) thumbnail.make rescue Paperclip::Errors::NotIdentifiedByImageMagickError make_nonimage_preview(closed_file, basename) ensure Attache.logger.info "[POOL] done in #{Time.now - t}s" end private BOLD_FONT_FILE = ENV.fetch('FONT_FILE', File.join(Attache.publicdir, "vendor/roboto/Roboto-Medium.ttf")) THIN_FONT_FILE = ENV.fetch('FONT_FILE', File.join(Attache.publicdir, "vendor/roboto/Roboto-Light.ttf")) BORDER_SIZE = ENV.fetch('BORDER_SIZE', "3") FG_COLOR = ENV.fetch('FG_COLOR', "#ffffff") BG_COLOR = ENV.fetch('BG_COLOR', "#dddddd") EXT_COLOR = ENV.fetch('EXT_COLOR', "#333333") TXT_SIZE = ENV.fetch('TXT_SIZE', "12") PREVIEW_SIZE = ENV.fetch('PREVIEW_SIZE', '96x') def make_nonimage_preview(closed_file, basename) t = Time.now Attache.logger.info "[POOL] start nonimage preview" output_file = Tempfile.new(["preview", ".png"]).tap(&:close) cmd = case basename when /\.pdf$/i "convert -size #{PREVIEW_SIZE.inspect} #{closed_file.path.inspect}[0] -thumbnail #{PREVIEW_SIZE.inspect} -font #{BOLD_FONT_FILE.inspect}" else "convert -size #{PREVIEW_SIZE.inspect} \\( -gravity center -font #{BOLD_FONT_FILE.inspect} -fill #{EXT_COLOR.inspect} label:'#{make_safe_filename(basename).split(/\W+/).last}' \\)" end + " -bordercolor #{FG_COLOR.inspect} -border #{BORDER_SIZE} -background #{BG_COLOR.inspect} -gravity center -font #{THIN_FONT_FILE.inspect} -pointsize 12 -set caption #{basename.inspect} -polaroid 0 #{output_file.path.inspect}" Attache.logger.info cmd system cmd File.new(output_file.path) ensure Attache.logger.info "[POOL] done nonimage preview in #{Time.now - t}s" end def make_safe_filename(str) str.to_s.gsub(/[^\w\.]/, '_') end def thumbnail_for(closed_file:, target_geometry_string:, extension:, max: 2048) convert_options = '-interlace Plane' if %w(jpg jpeg).include?(extension.to_s.downcase) thumbnail = Paperclip::Thumbnail.new(closed_file, geometry: target_geometry_string, format: extension, convert_options: convert_options) current_geometry = current_geometry_for(thumbnail) target_geometry = Paperclip::GeometryParser.new(target_geometry_string).make if target_geometry.larger <= max && current_geometry.larger > max # optimization: # when users upload "super big files", we can speed things up # by working from a "reasonably large 2048x2048 thumbnail" (<2 seconds) # instead of operating on the original (>10 seconds) # we store this reusably in Attache.cache to persist reboot, but not uploaded to cloud working_geometry = "#{max}x#{max}>" working_file = Attache.cache.fetch(Digest::SHA1.hexdigest(working_geometry + closed_file.path)) do Attache.logger.info "[POOL] generate working_file" Paperclip::Thumbnail.new(closed_file, geometry: working_geometry, format: extension).make end Attache.logger.info "[POOL] use working_file #{working_file.path}" thumbnail = Paperclip::Thumbnail.new(working_file.tap(&:close), geometry: target_geometry_string, format: extension, convert_options: convert_options) end thumbnail end # allow stub in spec def current_geometry_for(thumbnail) thumbnail.current_geometry.tap(&:auto_orient) end end ================================================ FILE: lib/attache/tasks.rb ================================================ require "rake" namespace :attache do desc "Convert content of FILE to a JSON string; default FILE=config/vhost.yml" task :vhost do require 'yaml' require 'json' file = ENV.fetch("FILE") { "config/vhost.yml" } puts YAML.load(IO.read(file)).to_json end end ================================================ FILE: lib/attache/tus/upload.rb ================================================ class Attache::Tus::Upload < Attache::Base def initialize(app) @app = app end def _call(env, config) case env['PATH_INFO'] when '/tus/files' tus = ::Attache::Tus.new(env, config) params = params_of(env) # avoid unnecessary `invalid byte sequence in UTF-8` on `request.params` return config.unauthorized unless config.authorized?(params) case env['REQUEST_METHOD'] when 'POST' if positive_number?(tus.upload_length) relpath = generate_relpath(Attache::Upload.sanitize(tus.upload_metadata['filename'] || params['file'])) cachekey = File.join(request_hostname(env), relpath) bytes_wrote = Attache.cache.write(cachekey, StringIO.new) uri = URI.parse(Rack::Request.new(env).url) uri.query = (uri.query ? "#{uri.query}&" : '') + "relpath=#{CGI.escape relpath}" [201, tus.headers_with_cors('Location' => uri.to_s), []] else [400, tus.headers_with_cors('X-Exception' => "Bad upload length"), []] end when 'PATCH' relpath = params['relpath'] cachekey = File.join(request_hostname(env), relpath) http_offset = tus.upload_offset if positive_number?(env['CONTENT_LENGTH']) && positive_number?(http_offset) && (env['CONTENT_TYPE'] == 'application/offset+octet-stream') && tus.resumable_version.to_s == '1.0.0' && current_offset(cachekey, relpath, config) >= http_offset.to_i append_to(cachekey, http_offset, env['rack.input']) config.storage_create(relpath: relpath, cachekey: cachekey) if config.storage && config.bucket [200, tus.headers_with_cors({'Content-Type' => 'text/json'}, offset: current_offset(cachekey, relpath, config)), [json_of(relpath, cachekey, config)], ] else [400, tus.headers_with_cors('X-Exception' => 'Bad headers'), []] end when 'OPTIONS' [201, tus.headers_with_cors, []] when 'HEAD' relpath = params['relpath'] cachekey = File.join(request_hostname(env), relpath) [200, tus.headers_with_cors({'Content-Type' => 'text/json'}, offset: current_offset(cachekey, relpath, config)), [json_of(relpath, cachekey, config)], ] when 'GET' relpath = params['relpath'] uri = URI.parse(Rack::Request.new(env).url) uri.query = nil uri.path = File.join('/view', File.dirname(relpath), 'original', CGI.escape(File.basename(relpath))) [302, tus.headers_with_cors('Location' => uri.to_s), []] end else @app.call(env) end end private def current_offset(cachekey, relpath, config) file = Attache.cache.fetch(cachekey) do config.storage_get(relpath: relpath) if config.storage && config.bucket end file.size rescue Attache.cache.write(cachekey, StringIO.new) ensure file.tap(&:close) end def append_to(cachekey, offset, io) f = File.open(path_of(cachekey), 'r+b') f.sync = true f.seek(offset.to_i) f.write(io.read) ensure f.close end def positive_number?(value) (value.to_s == "0" || value.to_i > 0) end end ================================================ FILE: lib/attache/tus.rb ================================================ class Attache::Tus LENGTH_KEYS = %w[Upload-Length Entity-Length] OFFSET_KEYS = %w[Upload-Offset Offset] METADATA_KEYS = %w[Upload-Metadata Metadata] attr_accessor :env, :config def initialize(env, config) @env = env @config = config end def header_value(keys) value = nil keys.find {|k| value = env["HTTP_#{k.gsub('-', '_').upcase}"]} value end def upload_length header_value LENGTH_KEYS end def upload_offset header_value OFFSET_KEYS end def upload_metadata value = header_value METADATA_KEYS Hash[*value.split(/[, ]/)].inject({}) do |h, (k, v)| h.merge(k => Base64.decode64(v)) end end def resumable_version header_value ["Tus-Resumable"] end def headers_with_cors(headers = {}, offset: nil) tus_headers = { "Access-Control-Allow-Methods" => "PATCH", "Access-Control-Allow-Headers" => "Tus-Resumable, #{LENGTH_KEYS.join(', ')}, #{METADATA_KEYS.join(', ')}, #{OFFSET_KEYS.join(', ')}", "Access-Control-Expose-Headers" => "Location, #{OFFSET_KEYS.join(', ')}", } OFFSET_KEYS.each do |k| tus_headers[k] = offset end if offset # append tus_headers.inject(config.headers_with_cors.merge(headers)) do |sum, (k, v)| sum.merge(k => [*sum[k], v].join(', ')) end end end ================================================ FILE: lib/attache/upload.rb ================================================ class Attache::Upload < Attache::Base def initialize(app) @app = app end def _call(env, config) case env['PATH_INFO'] when '/upload' case env['REQUEST_METHOD'] when 'POST', 'PUT', 'PATCH' request = Rack::Request.new(env) params = request.GET # stay away from parsing body return config.unauthorized unless config.authorized?(params) relpath = generate_relpath(Attache::Upload.sanitize params['file']) cachekey = File.join(request_hostname(env), relpath) bytes_wrote = Attache.cache.write(cachekey, request.body) if bytes_wrote == 0 return [500, config.headers_with_cors.merge('X-Exception' => 'Local file failed'), []] else Attache.logger.info "[Upload] received #{bytes_wrote} #{cachekey}" end config.storage_create(relpath: relpath, cachekey: cachekey) if config.storage && config.bucket [200, config.headers_with_cors.merge('Content-Type' => 'text/json'), [json_of(relpath, cachekey, config)]] when 'OPTIONS' [200, config.headers_with_cors, []] else [400, config.headers_with_cors, []] end else @app.call(env) end end def self.sanitize(filename) filename.to_s.gsub(/\%/, '_') end end ================================================ FILE: lib/attache/upload_url.rb ================================================ class Attache::UploadUrl < Attache::Base def initialize(app) @app = app end def _call(env, config) case env['PATH_INFO'] when '/upload_url' # always pretend to be `POST /upload` env['PATH_INFO'] = '/upload' env['REQUEST_METHOD'] = 'POST' request = Rack::Request.new(env) params = request.params return config.unauthorized unless config.authorized?(params) if params['url'] file, filename, content_type = download_file(params['url']) filename = "index" if filename == '/' env['CONTENT_TYPE'] = content_type || content_type_of(file.path) env['rack.request.query_hash'] = (env['rack.request.query_hash'] || {}).merge('file' => filename) env['rack.input'] = file end end @app.call(env) end MAX_DEPTH = 30 def download_file(url, depth = 0) raise Net::HTTPError, "Too many redirects" if depth > MAX_DEPTH Attache.logger.info "Upload GET #{url}" if url.match /\Adata:([^;,]+|)(;base64|),/ # data:[][;base64], # http://tools.ietf.org/html/rfc2397 data = URI.decode(url[url.index(',')+1..-1]) data = Base64.decode64(data) if $2 == ';base64' content_type = ($1 == '' ? "text/plain" : $1) filename = "data.#{content_type.gsub(/\W+/, '.')}" return [StringIO.new(data), filename, content_type] end uri = uri.kind_of?(URI::Generic) ? url : URI.parse(url) http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true if uri.scheme == 'https' req = Net::HTTP::Get.new(uri.request_uri) req.initialize_http_header({"User-Agent" => ENV['USER_AGENT']}) if ENV['USER_AGENT'] req.basic_auth(uri.user, uri.password) if uri.user || uri.password res = http.request(req) case res.code when /\A30[1,2]\z/ download_file URI.join(url, res['Location']).to_s, depth + 1 when /\A2\d\d\z/ f = Tempfile.new(["upload_url", File.extname(uri.path)]) f.write(res.body) f.close [f.tap(&:open), File.basename(uri.path)] else raise Net::HTTPError, "Failed #{res.code}" end end end ================================================ FILE: lib/attache/version.rb ================================================ module Attache VERSION = "3.0.0" end ================================================ FILE: lib/attache/vhost.rb ================================================ class Attache::VHost attr_accessor :remotedir, :secret_key, :backup, :bucket, :storage, :download_headers, :headers_with_cors, :geometry_whitelist, :env def initialize(hash) self.env = hash || {} self.remotedir = env['REMOTE_DIR'] # nil means no fixed top level remote directory, and that's fine. self.secret_key = env['SECRET_KEY'] # nil means no auth check; anyone can upload a file self.geometry_whitelist = env['GEOMETRY_WHITELIST'] # nil means everything is acceptable if env['FOG_CONFIG'] self.bucket = env['FOG_CONFIG'].fetch('bucket') self.storage = Fog::Storage.new(env['FOG_CONFIG'].except('bucket').symbolize_keys) if env['BACKUP_CONFIG'] backup_fog = env['FOG_CONFIG'].merge(env['BACKUP_CONFIG']) self.backup = Attache::VHost.new(env.except('BACKUP_CONFIG').merge('FOG_CONFIG' => backup_fog)) end end self.download_headers = { "Cache-Control" => "public, max-age=31536000" }.merge(env['DOWNLOAD_HEADERS'] || {}) self.headers_with_cors = { 'Access-Control-Allow-Origin' => '*', 'Access-Control-Allow-Methods' => 'POST, PUT', 'Access-Control-Allow-Headers' => 'Content-Type', }.merge(env['UPLOAD_HEADERS'] || {}) end def hmac_for(content) OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha1'), secret_key, content) end def hmac_valid?(params) params['uuid'] && params['hmac'] && params['expiration'] && Time.at(params['expiration'].to_i) > Time.now && Rack::Utils.secure_compare(params['hmac'], hmac_for("#{params['uuid']}#{params['expiration']}")) end def storage_url(args) object = remote_api.new({ key: File.join(*remotedir, args[:relpath]), }) result = if object.respond_to?(:url) object.url(Time.now + 600) else object.public_url end ensure Attache.logger.info "storage_url: #{result}" end def storage_get(args) open storage_url(args) end def storage_create(args) Attache.logger.info "[JOB] uploading #{args[:cachekey].inspect}" body = begin Attache.cache.read(args[:cachekey]) rescue Errno::ENOENT :no_entry # upload file no longer exist; likely deleted immediately after upload end unless body == :no_entry remote_api.create({ key: File.join(*remotedir, args[:relpath]), body: body, }) Attache.logger.info "[JOB] uploaded #{args[:cachekey]}" end end def storage_destroy(args) Attache.logger.info "[JOB] deleting #{args[:relpath]}" remote_api.new({ key: File.join(*remotedir, args[:relpath]), }).destroy Attache.logger.info "[JOB] deleted #{args[:relpath]}" end def remote_api storage.directories.new(key: bucket).files end def async(method, args) ::Attache::Job.perform_async(method, env, args) end def authorized?(params) secret_key.blank? || hmac_valid?(params) end def unauthorized [401, headers_with_cors.merge('X-Exception' => 'Authorization failed'), []] end def backup_file(args) if backup key = File.join(*remotedir, args[:relpath]) storage.copy_object(bucket, key, backup.bucket, key) end end end ================================================ FILE: lib/attache.rb ================================================ require 'active_support/all' require 'sys/filesystem' require 'securerandom' require 'disk_store' require 'fileutils' require 'paperclip' require 'net/http' require 'tempfile' require 'sidekiq' require 'tmpdir' require 'logger' require 'base64' require 'rack' require 'json' require 'uri' require 'cgi' require 'fog' if ENV['REDIS_PROVIDER'] || ENV['REDIS_URL'] # default sidekiq elsif ENV['INLINE_JOB'] require 'sidekiq/testing/inline' else require 'sucker_punch' end module Attache class << self attr_accessor :localdir, :vhost, :cache, :logger, :publicdir end end Attache.logger = Logger.new(STDOUT) Attache.localdir = File.expand_path(ENV.fetch('LOCAL_DIR') { Dir.tmpdir }) Attache.vhost = JSON.parse(ENV.fetch('VHOST') { YAML.load(IO.read('config/vhost.yml')).to_json rescue '{}' }) Attache.cache = DiskStore.new(Attache.localdir, { cache_size: ENV.fetch('CACHE_SIZE_BYTES') { stat = Sys::Filesystem.stat("/") available = stat.block_size * stat.blocks_available (available * 0.8).floor # use 80% free disk by default }.to_i, reaper_interval: ENV.fetch('CACHE_EVICTION_INTERVAL_SECONDS') { 60 }.to_i, eviction_strategy: (Attache.vhost.empty? ? nil : :LRU), # lru eviction only when there is remote storage }) Attache.publicdir = ENV.fetch("PUBLIC_DIR") { File.expand_path("../public", File.dirname(__FILE__)) } require 'attache/job' require 'attache/resize_job' require 'attache/base' require 'attache/vhost' require 'attache/upload_url' require 'attache/upload' require 'attache/delete' require 'attache/backup' require 'attache/download' require 'attache/file_response_body' require 'attache/tus' require 'attache/tus/upload' ================================================ FILE: public/index.html ================================================ It works! ================================================ FILE: public/vendor/roboto/Apache License.txt ================================================ Font data copyright Google 2012 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: spec/fixtures/sample.txt ================================================ data:not a data uri ================================================ FILE: spec/lib/attache/backup_spec.rb ================================================ require 'spec_helper' describe Attache::Backup do let(:app) { ->(env) { [200, env, "app"] } } let(:middleware) { Attache::Backup.new(app) } let(:params) { {} } let(:filename) { "hello#{rand}.gif" } let(:reldirname) { "path#{rand}" } let(:file) { StringIO.new(IO.binread("spec/fixtures/transparent.gif"), 'rb') } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(nil) end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, env = middleware.call Rack::MockRequest.env_for('http://example.com', {}) expect(code).to eq 200 end context "backup file" do let(:params) { Hash(paths: ['image1.jpg', filename].join("\n")) } subject { proc { middleware.call Rack::MockRequest.env_for('http://example.com/backup?' + params.collect {|k,v| "#{CGI.escape k.to_s}=#{CGI.escape v.to_s}"}.join('&'), method: 'DELETE', "HTTP_HOST" => "example.com") } } it 'should respond with json' do end it 'should not touch local file' do expect(Attache).not_to receive(:cache) code, headers, body = subject.call expect(code).to eq(200) end context 'storage configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(double(:storage)) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(double(:bucket)) end it 'should backup file' do expect_any_instance_of(Attache::VHost).to receive(:backup_file).exactly(2).times subject.call end end context 'storage NOT configured' do it 'should backup file' do expect_any_instance_of(Attache::VHost).not_to receive(:backup_file) subject.call end end context 'with secret_key' do let(:secret_key) { "topsecret#{rand}" } before do allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(secret_key) end it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end context 'invalid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, "wrong#{secret_key}", "#{uuid}#{expiration}")) } it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end context 'valid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, secret_key, "#{uuid}#{expiration}")) } it 'should respond with success' do code, headers, body = subject.call expect(code).to eq(200) end context 'expired' do let(:expiration) { (Time.now - 1).to_i } # the past it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end end end end end ================================================ FILE: spec/lib/attache/delete_spec.rb ================================================ require 'spec_helper' describe Attache::Delete do let(:app) { ->(env) { [200, env, "app"] } } let(:middleware) { Attache::Delete.new(app) } let(:params) { {} } let(:filename) { "hello#{rand}.gif" } let(:reldirname) { "path#{rand}" } let(:file) { StringIO.new(IO.binread("spec/fixtures/transparent.gif"), 'rb') } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(nil) end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, env = middleware.call Rack::MockRequest.env_for('http://example.com', {}) expect(code).to eq 200 end context "deleting" do let(:params) { Hash(paths: ['image1.jpg', filename].join("\n")) } subject { proc { middleware.call Rack::MockRequest.env_for('http://example.com/delete?' + params.collect {|k,v| "#{CGI.escape k.to_s}=#{CGI.escape v.to_s}"}.join('&'), method: 'DELETE', "HTTP_HOST" => "example.com") } } it 'should respond with json' do end it 'should delete file locally' do expect(Attache.cache).to receive(:delete) do |path| expect(path).to start_with('example.com') end.exactly(2).times code, headers, body = subject.call expect(code).to eq(200) end context 'delete fail locally' do before do expect(Attache.cache).to receive(:delete) do raise Exception.new end.at_least(1).times end it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(500) end end context 'storage configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(double(:storage)) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(double(:bucket)) end it 'should delete file remotely' do expect_any_instance_of(Attache::VHost).to receive(:async) do |instance, method, path| expect(method).to eq(:storage_destroy) end.exactly(2).times subject.call end end context 'storage NOT configured' do it 'should NOT delete file remotely' do expect_any_instance_of(Attache::VHost).not_to receive(:async) subject.call end end context 'backup configured' do let(:backup) { double(:backup) } before do allow_any_instance_of(Attache::VHost).to receive(:backup).and_return(backup) end it 'should delete file in backup' do expect(backup).to receive(:async) do |method, path| expect(method).to eq(:storage_destroy) end.exactly(2).times subject.call end end context 'backup NOT configured' do it 'should NOT delete file in backup' do expect_any_instance_of(Attache::VHost).not_to receive(:async) subject.call end end context 'with secret_key' do let(:secret_key) { "topsecret#{rand}" } before do allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(secret_key) end it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end context 'invalid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, "wrong#{secret_key}", "#{uuid}#{expiration}")) } it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end context 'valid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, secret_key, "#{uuid}#{expiration}")) } it 'should respond with success' do code, headers, body = subject.call expect(code).to eq(200) end context 'expired' do let(:expiration) { (Time.now - 1).to_i } # the past it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end end end end end ================================================ FILE: spec/lib/attache/download_spec.rb ================================================ require 'spec_helper' describe Attache::Download do let(:app) { ->(env) { [200, env, "app"] } } let(:middleware) { Attache::Download.new(app) } let(:params) { {} } let(:filename) { "hello#{rand}.gif" } let(:reldirname) { "path#{rand}" } let(:geometry) { CGI.escape('2x2#') } let(:file) { StringIO.new(IO.binread("spec/fixtures/transparent.gif"), 'rb') } let(:remote_url) { "http://example.com/image.jpg" } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, env = middleware.call Rack::MockRequest.env_for('http://example.com', "HTTP_HOST" => "example.com") expect(code).to eq 200 end context 'downloading' do subject { proc { middleware.call Rack::MockRequest.env_for("http://example.com/view/#{reldirname}/#{geometry}/#{filename}", "HTTP_HOST" => "example.com") } } context 'not in local cache' do before do Attache.cache.delete("example.com/#{reldirname}/#{filename}") end context 'no cloud storage configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(nil) end it 'should respond not found' do code, headers, body = subject.call expect(code).to eq(404) end it 'should continue to respond not found' do code, headers, body = subject.call expect(code).to eq(404) code, headers, body = subject.call expect(code).to eq(404) end end context 'with cloud storage configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(double(:storage, directories: Struct.new(:key, :files))) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(double(:bucket)) end it 'should respond not found' do code, headers, body = subject.call expect(code).to eq(404) end context 'with backup configured' do it 'should respond not found' do allow_any_instance_of(Attache::VHost).to receive(:backup).and_return(double(:backup, storage_get: nil)) code, headers, body = subject.call expect(code).to eq(404) end it 'should respond found if in backup' do allow_any_instance_of(Attache::VHost).to receive(:backup).and_return(double(:backup, storage_get: file)) code, headers, body = subject.call expect(code).to eq(200) end end context 'available remotely' do before do allow_any_instance_of(Attache::VHost).to receive(:storage_get).and_return(file) allow_any_instance_of(Attache::VHost).to receive(:storage_url).and_return(remote_url) end it 'should proceed normally' do code, headers, body = subject.call expect(code).to eq(200) end context 'geometry is "remote"' do let(:geometry) { CGI.escape('remote') } it 'should send remote file' do expect(Attache.cache).not_to receive(:fetch) expect_any_instance_of(Attache::VHost).to receive(:storage_url) code, headers, body = subject.call response_content = '' body.each {|p| response_content += p } expect(response_content).to eq('') expect(code).to eq(302) expect(headers['Location']).to eq(remote_url) expect(headers['Cache-Control']).to eq("private, no-cache") end end end end end context 'in local cache' do before do Attache.cache.write("example.com/#{reldirname}/#{filename}", file) end context 'geometry is "original"' do let(:geometry) { CGI.escape('original') } it 'should send original file' do expect_any_instance_of(middleware.class).not_to receive(:get_thumbnail_file) code, headers, body = subject.call response_content = '' body.each {|p| response_content += p } original_content = file.tap(&:rewind).read expect(response_content).to eq(original_content) end end context 'geometry_whitelist is present' do let(:geometry_whitelist) { ['100x100'] } before do allow(middleware).to receive(:vhost_for).and_return(double(:vhost, geometry_whitelist: geometry_whitelist, storage: nil, backup: nil, download_headers: {})) end context 'geometry is whitelisted' do let(:geometry) { geometry_whitelist.sample } it 'should be allowed' do code, headers, body = subject.call expect(code).to eq(200) end end context 'geometry is NOT whitelisted' do let(:geometry) { '999x999' } it 'should NOT be allowed' do code, headers, body = subject.call expect(code).to eq(415) expect(body).to eq(["#{geometry} is not supported"]) end end end context 'rendering' do context 'non image' do let(:file) { StringIO.new(IO.binread("spec/fixtures/sample.txt"), 'rb') } let(:filename) { "hello#{rand}.txt" } it 'should output as png' do expect_any_instance_of(Attache::ResizeJob).to receive(:make_nonimage_preview).exactly(1).times.and_call_original code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq("image/png") end end context 'image' do it 'should output as gif' do expect_any_instance_of(Attache::ResizeJob).not_to receive(:make_nonimage_preview) code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq("image/gif") end end end end end end ================================================ FILE: spec/lib/attache/resize_job_spec.rb ================================================ require 'spec_helper' describe Attache::ResizeJob do describe '#thumbnail_for' do let(:max) { 2048 } let(:current) { [1, 1] } let(:current_w) { current.shift } let(:current_h) { current.shift } let(:job) { Attache::ResizeJob.new } let(:original_path) { "spec/fixtures/transparent.gif" } before { allow(job).to receive(:current_geometry_for).and_return(Paperclip::Geometry.new(current_w, current_h)) Attache.cache.delete(Digest::SHA1.hexdigest("#{max}x#{max}>" + original_path)) } subject { job.send(:thumbnail_for, closed_file: File.new(original_path), target_geometry_string: target, extension: "gif").file.path } context 'target > max' do let(:target) { ["#{max+1}x1>", "1x#{max+1}>"].sample } it { expect_any_instance_of(Paperclip::Thumbnail).not_to receive(:make) is_expected.to eq(original_path) } end context 'target <= max' do let(:target) { ["#{max}x1>", "1x#{max}>"].sample } context 'current > max' do let(:current) { [max+1, 1].shuffle } it { expect_any_instance_of(Paperclip::Thumbnail).to receive(:make) do |instance| expect(instance.target_geometry.to_s).to eq("#{max}x#{max}>") File.new(original_path) end is_expected.not_to eq(original_path) } end context 'current <= max' do let(:current) { [max-1, 1].shuffle } it { expect_any_instance_of(Paperclip::Thumbnail).not_to receive(:make) is_expected.to eq(original_path) } end end context 'convert_options for jpg, jpeg' do let(:jpg_path) { "spec/fixtures/landscape.jpg" } let(:thumbnail) { job.send(:thumbnail_for, closed_file: File.new(jpg_path), target_geometry_string: '1x1>', extension: %w(jpg jpeg).sample) } it { expect(thumbnail.convert_options).to eq(%w(-interlace Plane)) } end context 'convert_options for other file extensions' do let(:thumbnail) { job.send(:thumbnail_for, closed_file: File.new(original_path), target_geometry_string: '1x1>', extension: %w(png gif tiff bmp).sample) } it { expect(thumbnail.convert_options).to be_nil } end end end ================================================ FILE: spec/lib/attache/tus/upload_spec.rb ================================================ require 'spec_helper' describe Attache::Tus::Upload do let(:app) { ->(env) { [200, env, "app"] } } let(:middleware) { Attache::Tus::Upload.new(app) } let(:params) { Hash(file: filename) } let(:filename) { "Exãmple %#{rand} %20.gif" } let(:file) { StringIO.new(IO.binread("spec/fixtures/landscape.jpg"), 'rb') } let(:filesize) { File.size "spec/fixtures/landscape.jpg" } let(:hostname) { "example.com" } let(:create_path) { '/tus/files?' + params.collect {|k,v| "#{CGI.escape k.to_s}=#{CGI.escape v.to_s}"}.join('&') } let(:resume_path) { @location } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(nil) end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, headers, body = middleware.call Rack::MockRequest.env_for('http://' + hostname, "HTTP_HOST" => hostname) expect(code).to eq 200 end def make_request_to(request_uri, headers) middleware.call Rack::MockRequest.env_for('http://' + hostname + request_uri, Hash("HTTP_HOST" => hostname, 'HTTP_UPLOAD_METADATA' => "key #{Base64.encode64('value')},filename #{Base64.encode64(filename)}").merge(headers)) end context "tus creation" do it "must reject missing HTTP_ENTITY_LENGTH" do code, headers, body = make_request_to(create_path, method: 'POST', input: file) expect(code).to eq(400) end it "must reject invalid HTTP_ENTITY_LENGTH" do code, headers, body = make_request_to(create_path, method: 'POST', input: file, 'HTTP_ENTITY_LENGTH' => [-1, 'abc'].sample) expect(code).to eq(400) end it "must respond successfully with HTTP 201 + Location header" do code, headers, body = make_request_to(create_path, method: 'POST', input: file, 'HTTP_ENTITY_LENGTH' => filesize) expect(code).to eq(201) expect(headers['Location']).to be_present end end context "with uploaded file" do let(:relpath) { CGI.unescape @location.match(/relpath=([^&]+)/)[1] } let(:cachekey) { File.join(hostname, relpath) } let(:current_offset) { 3 + rand(10) } before do code, headers, body = make_request_to(create_path, method: 'POST', input: file, 'HTTP_ENTITY_LENGTH' => filesize) expect(code).to eq(201) @location = URI.parse(headers['Location']).request_uri open(middleware.path_of(cachekey), "a") {|f| f.write('a' * current_offset) } end context "tus patch" do it "must reject invalid HTTP_OFFSET" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file) expect(code).to eq(400) end it "must reject invalid HTTP_CONTENT_LENGTH" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => 0) expect(code).to eq(400) end it "must reject invalid Content-Type: application/offset+octet-stream" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => 0, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => ["application/octet-stream", nil].sample) expect(code).to eq(400) end it "must reject invalid Tus-Resumable version" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => 0, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => "application/offset+octet-stream", "HTTP_TUS_RESUMABLE" => ["0.9.9", "2.0.0"].sample) expect(code).to eq(400) end it "must respond successfully with HTTP 200" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => 0, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => "application/offset+octet-stream", "HTTP_TUS_RESUMABLE" => "1.0.0") expect(headers['X-Exception']).to be_nil expect(code).to eq(200) end it "must accept `offset` smaller or equal to current offset" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => current_offset - [0, 1].sample, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => "application/offset+octet-stream", "HTTP_TUS_RESUMABLE" => "1.0.0") expect(headers['X-Exception']).to be_nil expect(code).to eq(200) end it "must reject `offset` larger than current offset" do code, headers, body = make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => current_offset + 1, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => "application/offset+octet-stream", "HTTP_TUS_RESUMABLE" => "1.0.0") expect(code).to eq(400) end it "new current offset must be Offset + Content-Length" do expect { make_request_to(resume_path, method: 'PATCH', input: file, "HTTP_OFFSET" => current_offset, "HTTP_CONTENT_LENGTH" => filesize, "CONTENT_TYPE" => "application/offset+octet-stream", "HTTP_TUS_RESUMABLE" => "1.0.0") }.to change { middleware.send(:current_offset, cachekey, relpath, config = {}) }.by(filesize) end end context "tus head" do it "must respond successfully with HTTP 200 + Offset header" do code, headers, body = make_request_to(resume_path, method: 'HEAD') expect(code).to eq(200) expect(headers).to eq({ "Access-Control-Allow-Origin" => "*", "Access-Control-Allow-Methods" => "POST, PUT, PATCH", "Access-Control-Allow-Headers" => "Content-Type, Tus-Resumable, Upload-Length, Entity-Length, Upload-Metadata, Metadata, Upload-Offset, Offset", "Content-Type" => "text/json", "Access-Control-Expose-Headers" => "Location, Upload-Offset, Offset", "Offset" => current_offset.to_s, "Upload-Offset" => current_offset.to_s, }) end end context "tus get" do it "must redirec to attache download url for original geometry" do code, headers, body = make_request_to(resume_path, method: 'GET') expect(code).to eq(302) expect(File.basename headers['Location']).to eq(CGI.escape(Attache::Upload.sanitize filename)) end end end end ================================================ FILE: spec/lib/attache/tus_spec.rb ================================================ require 'spec_helper' describe Attache::Tus do let(:env) { @env } let(:config) { @config } let(:tus) { Attache::Tus.new(env, config) } it "should return Entity-Length for upload_length" do @env = { 'HTTP_ENTITY_LENGTH' => rand } expect(tus.upload_length).to eq(@env['HTTP_ENTITY_LENGTH']) end it "should prefer Upload-Length for upload_length" do @env = { 'HTTP_ENTITY_LENGTH' => rand, 'HTTP_UPLOAD_LENGTH' => rand } expect(tus.upload_length).to eq(@env['HTTP_UPLOAD_LENGTH']) end it "should return Offset for upload_offset" do @env = { 'HTTP_OFFSET' => rand } expect(tus.upload_offset).to eq(@env['HTTP_OFFSET']) end it "should prefer Upload-Offset for upload_offset" do @env = { 'HTTP_OFFSET' => rand, 'HTTP_UPLOAD_OFFSET' => rand } expect(tus.upload_offset).to eq(@env['HTTP_UPLOAD_OFFSET']) end it "should parse upload_metadata" do @env = { "HTTP_UPLOAD_METADATA" => "key dmFsdWU=,randkey0.87393016369783 cmFuZHZhbHVlMC44MzYxNjcyOTk3OTQyMTU2" } expect(tus.upload_metadata).to eq({ "key" => "value", "randkey0.87393016369783" => "randvalue0.8361672997942156", }) end end ================================================ FILE: spec/lib/attache/upload_spec.rb ================================================ require 'spec_helper' describe Attache::Upload do let(:app) { ->(env) { [200, env, "app"] } } let(:middleware) { Attache::Upload.new(app) } let(:params) { {} } let(:filename) { "Exãmple %#{rand} %20.gif" } let(:file) { StringIO.new(IO.binread("spec/fixtures/landscape.jpg"), 'rb') } let(:request_input) { file } let(:base64_data) { "data:image/gif;base64," + Base64.encode64(file.read) } let(:hostname) { "example.com" } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(nil) end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, headers, body = middleware.call Rack::MockRequest.env_for('http://' + hostname, "HTTP_HOST" => hostname) expect(code).to eq 200 end context "uploading" do let(:params) { Hash(file: filename) } subject { proc { middleware.call Rack::MockRequest.env_for('http://' + hostname + '/upload?' + params.collect {|k,v| "#{CGI.escape k.to_s}=#{CGI.escape v.to_s}"}.join('&'), method: 'PUT', input: request_input, "HTTP_HOST" => hostname) } } it 'should respond successfully with json' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['geometry']).to eq('4x3') expect(json['bytes']).to eq(425) expect(json['signature']).to eq(nil) end end it 'should wrote to cache with Attache::Upload.sanitize(params[:file]) as filename' do code, headers, body = subject.call json = JSON.parse(body.join('')) relpath = json['path'] expect(relpath).to end_with(Attache::Upload.sanitize params[:file]) expect(Attache.cache.read(hostname + '/' + relpath).tap(&:close)).to be_kind_of(File) end # does not support base64/data uri here # see upload_url.rb context 'base64' do context 'base64-encoded image' do let!(:request_input) { StringIO.new("data:image/gif;base64," + Base64.encode64(file.read)) } it 'should respond identically as when uploading binary' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['geometry']).to eq(nil) expect(json['content_type']).to eq('text/plain') end end end # various Data URI permutations # https://developer.mozilla.org/en-US/docs/Web/HTTP/data_URIs context 'simple text/plain data' do let!(:request_input) { StringIO.new "data:,Hello%2C%20World!" } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(23) end end end context "base64-encoded version of the above" do let!(:request_input) { StringIO.new "data:text/plain;base64,SGVsbG8sIFdvcmxkIQ%3D%3D" } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(47) end end end context "An HTML document with

Hello, World!

" do let!(:request_input) { StringIO.new "data:text/html,%3Chtml%3E%3Cbody%3E%3Ch1%3EHello,%20World!%3C/h1%3E%3C/body%3E%3C/html%3E" } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(89) end end end context "An HTML document that executes a JavaScript alert" do let!(:request_input) { StringIO.new "data:text/html," } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/html') expect(json['bytes']).to eq(44) end end end end context 'plain text with data: prefix' do let!(:file) { StringIO.new(IO.binread("spec/fixtures/sample.txt"), 'rb') } it 'should not be mangled by Base64 decoding' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(20) end end end context 'save fail locally' do before do allow(Attache.cache).to receive(:write).and_return(0) end it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(500) expect(headers['X-Exception']).to eq('Local file failed') end end context 'storage not configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(nil) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(nil) end it 'should NOT save file remotely' do expect_any_instance_of(Attache::VHost).not_to receive(:storage_create) subject.call end end context 'storage configured' do before do allow_any_instance_of(Attache::VHost).to receive(:storage).and_return(double(:storage)) allow_any_instance_of(Attache::VHost).to receive(:bucket).and_return(double(:bucket)) end it 'should save file remotely' do expect_any_instance_of(Attache::VHost).to receive(:storage_create).and_return(anything) subject.call end end context 'with secret_key' do let(:secret_key) { "topsecret#{rand}" } before do allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(secret_key) end it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end context 'invalid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, "wrong#{secret_key}", "#{uuid}#{expiration}")) } it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end context 'valid auth' do let(:expiration) { (Time.now + 10).to_i } let(:uuid) { "hi#{rand}" } let(:digest) { OpenSSL::Digest.new('sha1') } let(:params) { Hash(file: filename, expiration: expiration, uuid: uuid, hmac: OpenSSL::HMAC.hexdigest(digest, secret_key, "#{uuid}#{expiration}")) } it 'should respond with success' do code, headers, body = subject.call expect(code).to eq(200) end it 'should respond successfully with json with signature' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| json_without_signature = json.reject {|k,v| k == 'signature' } generated_signature = OpenSSL::HMAC.hexdigest(digest, secret_key, json_without_signature.sort.collect {|k,v| "#{k}=#{v}" }.join('&')) expect(json['signature']).to eq(generated_signature) end end context 'expired' do let(:expiration) { (Time.now - 1).to_i } # the past it 'should respond with error' do code, headers, body = subject.call expect(code).to eq(401) expect(headers['X-Exception']).to eq('Authorization failed') end end end end end end ================================================ FILE: spec/lib/attache/upload_url_spec.rb ================================================ require 'spec_helper' describe Attache::UploadUrl do let(:app) { ->(env) { [200, env, "app"] } } let(:uploader) { Attache::Upload.new(app) } let(:middleware) { Attache::UploadUrl.new(uploader) } let(:hostname) { "example.com" } before do allow(Attache).to receive(:localdir).and_return(Dir.tmpdir) # forced, for safety allow_any_instance_of(Attache::VHost).to receive(:secret_key).and_return(nil) end after do FileUtils.rm_rf(Attache.localdir) end it "should passthrough irrelevant request" do code, headers, body = middleware.call Rack::MockRequest.env_for('http://' + hostname, "HTTP_HOST" => hostname) expect(code).to eq 200 end context 'upload as url' do subject { proc { middleware.call Rack::MockRequest.env_for('http://' + hostname + '/upload_url?' + params.collect {|k,v| "#{CGI.escape k.to_s}=#{CGI.escape v.to_s}"}.join('&'), method: 'PUT', "HTTP_HOST" => hostname) } } context 'to image' do let(:params) { Hash(url: "https://raw.githubusercontent.com/choonkeat/attache/master/spec/fixtures/landscape.jpg") } it 'should respond successfully with json' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['geometry']).to eq('4x3') end end end context 'follow redirect; works with non image too' do let(:params) { Hash(url: "http://google.com") } it 'should respond successfully with json' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['path']).not_to end_with('/') expect(json['content_type']).to eq('text/html') end end end context 'data uri' do context 'base64-encoded image' do let(:file) { StringIO.new(IO.binread("spec/fixtures/landscape.jpg"), 'rb') } let(:params) { Hash(url: "data:image/gif;base64," + Base64.encode64(file.read)) } it 'should respond identically as when uploading binary' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['geometry']).to eq('4x3') expect(json['bytes']).to eq(425) end end end # various Data URI permutations # https://developer.mozilla.org/en-US/docs/Web/HTTP/data_URIs context 'simple text/plain data' do let(:params) { Hash(url: "data:,Hello%2C%20World!") } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(13) end end end context "base64-encoded version of the above" do let(:params) { Hash(url: "data:text/plain;base64,SGVsbG8sIFdvcmxkIQ%3D%3D") } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/plain') expect(json['bytes']).to eq(13) end end end context "An HTML document with

Hello, World!

" do let(:params) { Hash(url: "data:text/html,%3Chtml%3E%3Cbody%3E%3Ch1%3EHello,%20World!%3C/h1%3E%3C/body%3E%3C/html%3E") } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/html') expect(json['bytes']).to eq(48) end end end context "An HTML document that executes a JavaScript alert" do let(:params) { Hash(url: "data:text/html,") } it 'should decode' do code, headers, body = subject.call expect(code).to eq(200) expect(headers['Content-Type']).to eq('text/json') JSON.parse(body.join('')).tap do |json| expect(json).to be_has_key('path') expect(json['content_type']).to eq('text/html') expect(json['bytes']).to eq(29) end end end end end end ================================================ FILE: spec/lib/attache/vhost_spec.rb ================================================ require 'spec_helper' require 'fog/storage/local/models/file' require 'fog/aws/models/storage/file' describe Attache::VHost do let(:config) { { 'REMOTE_DIR' => remotedir } } let(:config_with_backup) { YAML.load_file('config/vhost.example.yml').fetch("aws.example.com").merge('REMOTE_DIR' => remotedir) } let(:vhost) { Attache::VHost.new(config) } let(:remote_api) { double(:remote_api) } let(:file_io) { StringIO.new("") } let(:relpath) { 'relpath' } let(:cachekey) { 'hostname/relpath' } let(:remotedir) { 'remote_directory' } before do allow(vhost).to receive(:remote_api).and_return(remote_api) end describe '#storage_url' do let(:url) { 'http://example.com/a/b/c' } before do allow(remote_api).to receive(:new).and_return(files) allow_any_instance_of(Fog::Storage::Local::File).to receive(:public_url).and_return(url) allow_any_instance_of(Fog::Storage::AWS::File).to receive(:url).and_return(url) end context 'fog local storage' do let(:files) { return Fog::Storage::Local::File.new } it 'should return' do expect(vhost.storage_url(relpath: relpath)).to eq(url) end end context 'fog s3 storage' do let(:files) { return Fog::Storage::AWS::File.new } it 'should return' do expect(vhost.storage_url(relpath: relpath)).to eq(url) end end end describe '#storage_create' do it 'should read with cachekey, write with remotedir prefix' do expect(Attache.cache).to receive(:read).with(cachekey).and_return(file_io) expect(remote_api).to receive(:create).with(key: "#{remotedir}/#{relpath}", body: file_io) vhost.storage_create(relpath: relpath, cachekey: cachekey) end it 'should raise on other errors' do allow(Attache.cache).to receive(:read) { raise Exception.new } expect(remote_api).not_to receive(:create) expect { vhost.storage_create(relpath: relpath, cachekey: cachekey) }.to raise_error(Exception) end end describe '#storage' do it { expect(vhost.storage).to be_nil } context 'configured' do let(:config) { config_with_backup } it { expect(vhost.storage).to be_kind_of(Fog::Storage::AWS::Real) } it { expect(vhost.storage.region).to eq('us-west-1') } end end describe '#bucket' do it { expect(vhost.bucket).to be_nil } context 'configured' do let(:config) { config_with_backup } it { expect(vhost.bucket).to eq("CHANGEME") } end end describe '#backup' do it { expect(vhost.backup).to be_nil } describe '#backup_file' do it 'should not do anything' do allow_message_expectations_on_nil expect(vhost.storage).not_to receive(:copy_object) vhost.backup_file(relpath: relpath) end end context 'configured' do let(:config) { config_with_backup } it { expect(vhost.backup).to be_kind_of(Attache::VHost) } it { expect(vhost.backup.storage).to be_kind_of(Fog::Storage::AWS::Real) } it { expect(vhost.backup.storage.region).to eq('us-west-1') } it { expect(vhost.backup.bucket).to eq("CHANGEME_BAK") } describe '#backup_file' do it 'should not do anything' do expect(vhost.storage).to receive(:copy_object).with( vhost.bucket, "#{remotedir}/#{relpath}", vhost.backup.bucket, "#{remotedir}/#{relpath}" ) vhost.backup_file(relpath: relpath) end end end end end ================================================ FILE: spec/spec_helper.rb ================================================ ENV['VHOST'] = '{"0.0.0.0":{}}' require 'attache.rb' require 'sucker_punch/testing/inline' # This file was generated by the `rspec --init` command. Conventionally, all # specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`. # The generated `.rspec` file contains `--require spec_helper` which will cause # this file to always be loaded, without a need to explicitly require it in any # files. # # Given that it is always loaded, you are encouraged to keep this file as # light-weight as possible. Requiring heavyweight dependencies from this file # will add to the boot time of your test suite on EVERY test run, even for an # individual file that may not need all of that loaded. Instead, consider making # a separate helper file that requires the additional dependencies and performs # the additional setup, and require it from the spec files that actually need # it. # # The `.rspec` file also contains a few flags that are not defaults but that # users commonly want. # # See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration RSpec.configure do |config| # rspec-expectations config goes here. You can use an alternate # assertion/expectation library such as wrong or the stdlib/minitest # assertions if you prefer. config.expect_with :rspec do |expectations| # This option will default to `true` in RSpec 4. It makes the `description` # and `failure_message` of custom matchers include text for helper methods # defined using `chain`, e.g.: # be_bigger_than(2).and_smaller_than(4).description # # => "be bigger than 2 and smaller than 4" # ...rather than: # # => "be bigger than 2" expectations.include_chain_clauses_in_custom_matcher_descriptions = true end # rspec-mocks config goes here. You can use an alternate test double # library (such as bogus or mocha) by changing the `mock_with` option here. config.mock_with :rspec do |mocks| # Prevents you from mocking or stubbing a method that does not exist on # a real object. This is generally recommended, and will default to # `true` in RSpec 4. mocks.verify_partial_doubles = true end # The settings below are suggested to provide a good initial experience # with RSpec, but feel free to customize to your heart's content. # These two settings work together to allow you to limit a spec run # to individual examples or groups you care about by tagging them with # `:focus` metadata. When nothing is tagged with `:focus`, all examples # get run. config.filter_run :focus config.run_all_when_everything_filtered = true # Run specs in random order to surface order dependencies. If you find an # order dependency and want to debug it, you can fix the order by providing # the seed, which is printed after each run. # --seed 1234 config.order = :random =begin # Limits the available syntax to the non-monkey patched syntax that is # recommended. For more details, see: # - http://myronmars.to/n/dev-blog/2012/06/rspecs-new-expectation-syntax # - http://teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/ # - http://myronmars.to/n/dev-blog/2014/05/notable-changes-in-rspec-3#new__config_option_to_disable_rspeccore_monkey_patching config.disable_monkey_patching! # This setting enables warnings. It's recommended, but in some cases may # be too noisy due to issues in dependencies. config.warnings = true # Many RSpec users commonly either run the entire suite or an individual # file, and it's useful to allow more verbose output when running an # individual spec file. if config.files_to_run.one? # Use the documentation formatter for detailed output, # unless a formatter has already been configured # (e.g. via a command-line flag). config.default_formatter = 'doc' end # Print the 10 slowest examples and example groups at the # end of the spec run, to help surface which specs are running # particularly slow. config.profile_examples = 10 # Seed global randomization in this process using the `--seed` CLI option. # Setting this allows you to use `--seed` to deterministically reproduce # test failures related to randomization by passing the same `--seed` value # as the one that triggered the failure. Kernel.srand config.seed =end end Attache.logger = Logger.new("/dev/null") Paperclip.options[:log] = false Sidekiq::Logging.logger = nil SuckerPunch.logger = nil