-
-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve migrations to support migrating milestones/labels/issues/comm…
…ents/pullrequests (#6290) * add migrations * fix package dependency * fix lints * implements migrations except pull requests * add releases * migrating releases * fix bug * fix lint * fix migrate releases * fix tests * add rollback * pull request migtations * fix import * fix go module vendor * add tests for upload to gitea * more migrate options * fix swagger-check * fix misspell * add options on migration UI * fix log error * improve UI options on migrating * add support for username password when migrating from github * fix tests * remove comments and fix migrate limitation * improve error handles * migrate API will also support migrate milestones/labels/issues/pulls/releases * fix tests and remove unused codes * add DownloaderFactory and docs about how to create a new Downloader * fix misspell * fix migration docs * Add hints about migrate options on migration page * fix tests
- Loading branch information
Showing
128 changed files
with
33,540 additions
and
75 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| date: "2019-04-15T17:29:00+08:00" | ||
| title: "Advanced: Migrations Interfaces" | ||
| slug: "migrations-interfaces" | ||
| weight: 30 | ||
| toc: true | ||
| draft: false | ||
| menu: | ||
| sidebar: | ||
| parent: "advanced" | ||
| name: "Migrations Interfaces" | ||
| weight: 55 | ||
| identifier: "migrations-interfaces" | ||
| --- | ||
|
|
||
| # Migration Features | ||
|
|
||
| The new migration features were introduced in Gitea 1.9.0. It defines two interfaces to support migrating | ||
| repositories data from other git host platforms to gitea or, in the future migrating gitea data to other | ||
| git host platforms. Currently, only the migrations from github via APIv3 to Gitea is implemented. | ||
|
|
||
| First of all, Gitea defines some standard objects in packages `modules/migrations/base`. They are | ||
| `Repository`, `Milestone`, `Release`, `Label`, `Issue`, `Comment`, `PullRequest`. | ||
|
|
||
| ## Downloader Interfaces | ||
|
|
||
| To migrate from a new git host platform, there are two steps to be updated. | ||
|
|
||
| - You should implement a `Downloader` which will get all kinds of repository informations. | ||
| - You should implement a `DownloaderFactory` which is used to detect if the URL matches and | ||
| create a Downloader. | ||
| - You'll need to register the `DownloaderFactory` via `RegisterDownloaderFactory` on init. | ||
|
|
||
| ```Go | ||
| type Downloader interface { | ||
| GetRepoInfo() (*Repository, error) | ||
| GetMilestones() ([]*Milestone, error) | ||
| GetReleases() ([]*Release, error) | ||
| GetLabels() ([]*Label, error) | ||
| GetIssues(start, limit int) ([]*Issue, error) | ||
| GetComments(issueNumber int64) ([]*Comment, error) | ||
| GetPullRequests(start, limit int) ([]*PullRequest, error) | ||
| } | ||
| ``` | ||
|
|
||
| ```Go | ||
| type DownloaderFactory interface { | ||
| Match(opts MigrateOptions) (bool, error) | ||
| New(opts MigrateOptions) (Downloader, error) | ||
| } | ||
| ``` | ||
|
|
||
| ## Uploader Interface | ||
|
|
||
| Currently, only a `GiteaLocalUploader` is implemented, so we only save downloaded | ||
| data via this `Uploader` on the local Gitea instance. Other uploaders are not supported | ||
| and will be implemented in future. | ||
|
|
||
| ```Go | ||
| // Uploader uploads all the informations | ||
| type Uploader interface { | ||
| CreateRepo(repo *Repository, includeWiki bool) error | ||
| CreateMilestone(milestone *Milestone) error | ||
| CreateRelease(release *Release) error | ||
| CreateLabel(label *Label) error | ||
| CreateIssue(issue *Issue) error | ||
| CreateComment(issueNumber int64, comment *Comment) error | ||
| CreatePullRequest(pr *PullRequest) error | ||
| Rollback() error | ||
| } | ||
|
|
||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| // Copyright 2019 The Gitea Authors. All rights reserved. | ||
| // Use of this source code is governed by a MIT-style | ||
| // license that can be found in the LICENSE file. | ||
|
|
||
| package models | ||
|
|
||
| import "github.com/go-xorm/xorm" | ||
|
|
||
| // InsertIssue insert one issue to database | ||
| func InsertIssue(issue *Issue, labelIDs []int64) error { | ||
| sess := x.NewSession() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if err := insertIssue(sess, issue, labelIDs); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| func insertIssue(sess *xorm.Session, issue *Issue, labelIDs []int64) error { | ||
| if issue.MilestoneID > 0 { | ||
| sess.Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| if _, err := sess.ID(issue.MilestoneID).NoAutoTime().Update(new(Milestone)); err != nil { | ||
| return err | ||
| } | ||
| } | ||
| if _, err := sess.NoAutoTime().Insert(issue); err != nil { | ||
| return err | ||
| } | ||
| var issueLabels = make([]IssueLabel, 0, len(labelIDs)) | ||
| for _, labelID := range labelIDs { | ||
| issueLabels = append(issueLabels, IssueLabel{ | ||
| IssueID: issue.ID, | ||
| LabelID: labelID, | ||
| }) | ||
| } | ||
| if _, err := sess.Insert(issueLabels); err != nil { | ||
| return err | ||
| } | ||
| if !issue.IsPull { | ||
| sess.ID(issue.RepoID).Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| } else { | ||
| sess.ID(issue.RepoID).Incr("num_pulls") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_pulls") | ||
| } | ||
| } | ||
| if _, err := sess.NoAutoTime().Update(issue.Repo); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| sess.Incr("num_issues") | ||
| if issue.IsClosed { | ||
| sess.Incr("num_closed_issues") | ||
| } | ||
| if _, err := sess.In("id", labelIDs).Update(new(Label)); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if issue.MilestoneID > 0 { | ||
| if _, err := sess.ID(issue.MilestoneID).SetExpr("completeness", "num_closed_issues * 100 / num_issues").Update(new(Milestone)); err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| return nil | ||
| } | ||
|
|
||
| // InsertComment inserted a comment | ||
| func InsertComment(comment *Comment) error { | ||
| sess := x.NewSession() | ||
| defer sess.Close() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
| if _, err := sess.NoAutoTime().Insert(comment); err != nil { | ||
| return err | ||
| } | ||
| if _, err := sess.ID(comment.IssueID).Incr("num_comments").Update(new(Issue)); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| // InsertPullRequest inserted a pull request | ||
| func InsertPullRequest(pr *PullRequest, labelIDs []int64) error { | ||
| sess := x.NewSession() | ||
| defer sess.Close() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
| if err := insertIssue(sess, pr.Issue, labelIDs); err != nil { | ||
| return err | ||
| } | ||
| pr.IssueID = pr.Issue.ID | ||
| if _, err := sess.NoAutoTime().Insert(pr); err != nil { | ||
| return err | ||
| } | ||
| return sess.Commit() | ||
| } | ||
|
|
||
| // MigrateRelease migrates release | ||
| func MigrateRelease(rel *Release) error { | ||
| sess := x.NewSession() | ||
| if err := sess.Begin(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| var oriRel = Release{ | ||
| RepoID: rel.RepoID, | ||
| TagName: rel.TagName, | ||
| } | ||
| exist, err := sess.Get(&oriRel) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if !exist { | ||
| if _, err := sess.NoAutoTime().Insert(rel); err != nil { | ||
| return err | ||
| } | ||
| } else { | ||
| rel.ID = oriRel.ID | ||
| if _, err := sess.ID(rel.ID).Cols("target, title, note, is_tag, num_commits").Update(rel); err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| for i := 0; i < len(rel.Attachments); i++ { | ||
| rel.Attachments[i].ReleaseID = rel.ID | ||
| } | ||
|
|
||
| if _, err := sess.NoAutoTime().Insert(rel.Attachments); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| return sess.Commit() | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| // Copyright 2019 The Gitea Authors. All rights reserved. | ||
| // Copyright 2018 Jonas Franz. All rights reserved. | ||
| // Use of this source code is governed by a MIT-style | ||
| // license that can be found in the LICENSE file. | ||
|
|
||
| package base | ||
|
|
||
| import "time" | ||
|
|
||
| // Comment is a standard comment information | ||
| type Comment struct { | ||
| PosterName string | ||
| PosterEmail string | ||
| Created time.Time | ||
| Content string | ||
| Reactions *Reactions | ||
| } |
Oops, something went wrong.