Image uploading has always been a problem for me at my current job. This is because we use paperclip to process our image into many different sizes (thumbnails, small squares, large image, etc). Processing these images take some time and if there’s enough users trying it at the same time, the request will timeout. This was a big issue for us because as our company scales, the timeouts were happening more frequently. So this was a critical issue that needed to be addressed immediately. So our solution was to use Amazon S3 and delayed jobs (https://github.com/collectiveidea/delayed_job).
The idea was to upload the image up to Amazon S3, which will then return an image URL. So this uploading part does not touch our backend at all so even large images should be fine. It will not eat up any web threads. Once we get the URL from the cloud storage, we will send that to our backend. This will create an image with this URL and will use delayed jobs to process that image. Delayed jobs will basically put it in a queue and use workers to process the image in the background. This way, the user can get quick feedback and it will not hold up a web thread. The drawback to this is that the image might take long time to process if there are hundreds of jobs in the queue. The user will have a poor experience if they wonder why their image isn’t showing. But there are ways around that by using the temporary image URL from Amazon S3.
So first things first. We need to create the image model. It only really needs a two fields for now. Just the file attachment and the unprocessed image URL from Amazon S3.
class Image < Asset
field :unprocessed_image_url
has_mongoid_attached_file :image, {:styles => IMAGE_STYLES, :processors => :thumbnail}
def process_image
self.image = open(self.unprocessed_image_url)
self.save_image
end
endThe IMAGE_STYLES are just all of the different sizes of the images we need to cut. Then we have a process_image method that will open up the URL. By putting it into the image field, Paperclip will process that image for us.
The next step is to create a signed URLs controller. This controller will be used to create the Amazon S3 upload policy document and signature. This will be called when you upload an image. It will generate a json with these info and pass this along with the image to Amazon S3. It’s basically our credentials to make sure we have the rights to upload an image to our Amazon S3.
class SignedUrlsController < ApplicationController
def index
render json: {
policy: s3_upload_policy_document,
signature: s3_upload_signature,
key: "uploads/#{SecureRandom.uuid}/#{params[:doc][:title]}",
success_action_redirect: "/"
}
end
private
# generate the policy document that amazon is expecting.
def s3_upload_policy_document
Base64.encode64(
{
expiration: 30.minutes.from_now.utc.strftime('%Y-%m-%dT%H:%M:%S.000Z'),
conditions: [
{ bucket: S3['bucket'] },
{ acl: 'public-read' },
["starts-with", "$key", "uploads/"],
{ success_action_status: '201' }
]
}.to_json
).gsub(/\n|\r/, '')
end
# sign our request by Base64 encoding the policy document.
def s3_upload_signature
Base64.encode64(
OpenSSL::HMAC.digest(
OpenSSL::Digest::Digest.new('sha1'),
S3['secret_access_key'],
s3_upload_policy_document
)
).gsub(/\n/, '')
end
end
Next is a controller to actually upload our image URL we get from Amazon S3. It will create the image model and then use delayed jobs to put the processing into a queue. It’s pretty straight forward the the json_for_image just either checks if the image is there or not. If the image is processed, it will grab the processed image and if not, it will use the unprocessed_image_url. This is just for preview. If you don’t need that, you can just pass a success.
class ImagesController < ApplicationController
def create
image = Image.create(:unprocessed_image_url => params[:url])
image.delay.process_image
render :json => json_for_image(image)
end
end
Now that our backend is complete, we need to work on the image uploader form. We need a good image uploader and there is a really nice one online (https://github.com/blueimp/jQuery-File-Upload). It has tons of neat features such as progress bar, chunked uploads, client-side image resizing, and the list goes on. But I’m using it mostly for the drag and drop support as well as the multiple file upload at once. So here is the HTML part in HAML:
%form.image-form{:action => "#{S3['protocol'] + '://' + S3['bucket'] + '.' + S3['domain']}", :method => "post"}
%input{:type => "hidden", :name => "key"}
%input{:type => "hidden", :name => "AWSAccessKeyId", :value => "#{S3['access_key_id']}"}
%input{:type => "hidden", :name => "acl", :value => "public-read"}
%input{:type => "hidden", :name => "policy"}
%input{:type => "hidden", :name => "signature"}
%input{:type => "hidden", :name => "success_action_status", :value => "201"}
%input{:type => "file", :multiple => false, :name => "file", :accept => "image/*", :class => "image-upload"}
You will need to have the Amazon S3 protocal, bucket, domain, and access_key_id key ready. I also have created a drop zone which you can point to any div where you can drag and drop images to. After you set this up, you will need to initialize the plugin in the Javascript (this will be in coffeescript):
initializeUploadField: ->
jQuery =>
@$(".image-upload").fileupload
dropZone: @$(".image-dropzone")
add: (e, data) =>
$.ajax
url: "/signed_urls"
type: "GET"
dataType: "json"
data:
doc:
title: data.files[0].name
async: false
success: (retdata) =>
@$("input[name=key]").val retdata.key
@$("input[name=policy]").val retdata.policy
@$("input[name=signature]").val retdata.signature
data.submit()
success: (data) =>
image_url = decodeURIComponent($(data).find("Location").text())
image = new Woahlag.Models.Image
url: image_url
image.save null,
success: (model, response) =>
console.log "Success"
error: ->
console.log "ERROR SAVING IMAGE"
As you can see in the add block, it will hit our signed_urls controller to fetch that json we set up earlier. If it succeeds, it sets the key, policy, and signature for Amazon S3 and then submit the data. When that’s done, Amazon will pass back a bunch of data but we only want the location of that image. So we grab that and created a Backbone model with that URL to submit. Though I only printed out “Success” in this example, you can use the image json to show the preview.
And that concludes how to upload mass amount of pictures without locking down your site. I will have posts later on about how we had weird critical bugs with this and how we managed cropping. Thanks!
I Don't Know What I'm Doing 