• Migrate to object storage
  • All-in-one Rake task
  • Individual Rake tasks
• Migrate to local storage
  • All-in-one Rake task

Uploads migrate Rake tasks

There is a Rake task for migrating uploads between different storage types.

• Migrate all uploads with gitlab:uploads:migrate:all or
• To only migrate specific upload types, use gitlab:uploads:migrate.

Migrate to object storage

After configuring the object storage for uploads to GitLab, use this task to migrate existing uploads from the local storage to the remote storage.

All of the processing is done in a background worker and requires no downtime.

Read more about using object storage with GitLab.

All-in-one Rake task

GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos, attachments, and favicon) to object storage in one step. The wrapper task invokes individual Rake tasks to migrate files falling under each of these categories one by one.

These individual Rake tasks are described in the next section.

To migrate all uploads from local storage to object storage, run:

• Linux package installations:

  gitlab-rake "gitlab:uploads:migrate:all"
  

• Self-compiled installations:

  sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all
  

You can optionally track progress and verify that all uploads migrated successfully using the PostgreSQL console:

• sudo gitlab-rails dbconsole --database main for Linux package installations.
• sudo -u git -H psql -d gitlabhq_production for self-compiled installations.

Verify objectstg below (where store=2) has count of all artifacts:

gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads;

total | filesystem | objectstg
------+------------+-----------
   2409 |          0 |      2409

Verify that there are no files on disk in the uploads folder:

sudo find /var/opt/gitlab/gitlab-rails/uploads -type f | grep -v tmp | wc -l

Individual Rake tasks

If you already ran the all-in-one Rake task, there is no need to run these individual tasks.

The Rake task uses three parameters to find uploads to migrate:

This task also accepts an environment variable which you can use to override the default batch size:

The following shows how to run gitlab:uploads:migrate for individual types of uploads.

# gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point]

# Avatars
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"

# Attachments
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"

# Favicon
gitlab-rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"

# Markdown
gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"
gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"

# Design Management design thumbnails
gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action, :image_v432x230]"

# sudo -u git -H bundle exec rake gitlab:uploads:migrate

# Avatars
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"

# Attachments
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"

# Favicon
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"

# Markdown
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"

# Design Management design thumbnails
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]"

Migrate to local storage

If you need to disable object storage for any reason, you must first migrate your data out of object storage and back into your local storage.

All-in-one Rake task

GitLab provides a wrapper Rake task that migrates all uploaded files (for example, avatars, logos, attachments, and favicon) to local storage in one step. The wrapper task invokes individual Rake tasks to migrate files falling under each of these categories one by one.

For details on these Rake tasks, refer to Individual Rake tasks. Keep in mind the task name in this case is gitlab:uploads:migrate_to_local.

To migrate uploads from object storage to local storage, run the following Rake task:

• Linux package installations:

  gitlab-rake "gitlab:uploads:migrate_to_local:all"
  

• Self-compiled installations:

  sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all
  

After running the Rake task, you can disable object storage by undoing the changes described in the instructions to configure object storage.