Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DistGB] enable dist partition pipeline to save FusedCSCSamplingGraph partition directly #7728

Merged
merged 45 commits into from
Sep 19, 2024
Merged

[DistGB] enable dist partition pipeline to save FusedCSCSamplingGraph partition directly #7728

merged 45 commits into from
Sep 19, 2024

Conversation

CfromBU
Copy link
Collaborator

@CfromBU CfromBU commented Aug 21, 2024

implement dist partition in FusedCSCSamplingGraph format.

@dgl-bot
Copy link
Collaborator

dgl-bot commented Aug 21, 2024

To trigger regression tests:

  • @dgl-bot run [instance-type] [which tests] [compare-with-branch];
    For example: @dgl-bot run g4dn.4xlarge all dmlc/master or @dgl-bot run c5.9xlarge kernel,api dmlc/master

@CfromBU
Copy link
Collaborator Author

CfromBU commented Aug 21, 2024 •
edited
Loading

order:
#7724 partition test case
#7727 partition
#7728 dist partition
#7690 dist partition test case

This was referenced Aug 21, 2024
Merged
Merged
Open
@Rhett-Ying Rhett-Ying marked this pull request as draft August 21, 2024 07:30
@Rhett-Ying Rhett-Ying marked this pull request as ready for review August 21, 2024 07:30
@dgl-bot
Copy link
Collaborator

dgl-bot commented Aug 21, 2024

Commit ID: 9e1caadb796d43634de793bf46211c284340d1b7

Build ID: 1

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

Rhett-Ying
Rhett-Ying reviewed Aug 23, 2024
View reviewed changes
tools/distpartitioning/data_shuffle.py Outdated Show resolved Hide resolved
tools/distpartitioning/data_shuffle.py Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 5, 2024

Commit ID: 4fa7d71f3451d441d46e10f307ba62dbadf9061d

Build ID: 3

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 5, 2024

Commit ID: dfb10aaabf0c961d1a9fb7978d6c03d08f822801

Build ID: 4

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@Rhett-Ying Rhett-Ying changed the title [DistGB] use distribute partition to save FusedCSCSamplingGraph [DistGB] enable dist partition pipeline to save FusedCSCSamplingGraph partition directly Sep 6, 2024
@Rhett-Ying Rhett-Ying self-requested a review September 6, 2024 01:33
Rhett-Ying
Rhett-Ying requested changes Sep 6, 2024
View reviewed changes
Copy link
Collaborator

@Rhett-Ying Rhett-Ying left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dedicated tests are required.

tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Outdated Show resolved Hide resolved
@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 6, 2024

Commit ID: f91e8c2

Build ID: 5

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

Rhett-Ying
Rhett-Ying reviewed Sep 11, 2024
View reviewed changes
tools/distpartitioning/convert_partition.py Show resolved Hide resolved
tools/distpartitioning/utils.py Show resolved Hide resolved
tools/distpartitioning/utils.py Outdated Show resolved Hide resolved
tools/distpartitioning/convert_partition.py Show resolved Hide resolved
tests/tools/test_dist_partition_graphbolt.py Show resolved Hide resolved
@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: ab35628159cbc2a985de66dbb90658f4395ac03a

Build ID: 22

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: 3b5dd4aa446544a8e5fcda8bedc4b303c997ec38

Build ID: 23

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: c3c477d5e164c8b6a951abc3ccdf9d812003039f

Build ID: 24

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: 008138cabaee0d06c98314a0f2a3b664b1c8406f

Build ID: 25

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: 2476f55cfea6d7278f0c4be9520048fc4128523d

Build ID: 26

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@Rhett-Ying Rhett-Ying self-requested a review September 12, 2024 08:55
@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 12, 2024

Commit ID: fe751b1

Build ID: 27

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 13, 2024

Commit ID: 44e5beaa096d9c1aca9e27adcc758be5dd055e7f

Build ID: 28

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 18, 2024

Commit ID: 19e2147ef13a7ee0c5bee917405a57347f1f034f

Build ID: 29

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 18, 2024

Commit ID: cc481391475f397d494192c7e0aebc1d74a9b6e9

Build ID: 30

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@dgl-bot
Copy link
Collaborator

dgl-bot commented Sep 18, 2024

Commit ID: 8d84d71466f6cc3d5670facc1018606ad9e4886b

Build ID: 31

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

@Rhett-Ying
Copy link
Collaborator

@thvasilo FYI. could you try with this new feature on your side?

@thvasilo
Copy link
Contributor

Hi @Rhett-Ying, sure I can try this out. Is there documentation on how to use this new capability? Is passing --use-graphbolt to our call to dispatch data enough in this case?

@Rhett-Ying
Copy link
Collaborator

Hi @Rhett-Ying, sure I can try this out. Is there documentation on how to use this new capability? Is passing --use-graphbolt to our call to dispatch data enough in this case?

Yes

@CfromBU CfromBU merged commit 3bc8e22 into dmlc:master Sep 19, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Rhett-Ying Rhett-Ying Rhett-Ying approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@CfromBU @dgl-bot @Rhett-Ying @thvasilo