Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing Distributed Loader #48

Merged
merged 11 commits into from
Mar 3, 2016
Merged

Testing Distributed Loader #48

merged 11 commits into from
Mar 3, 2016

Commits on Mar 2, 2016

  1. Rename flag in uidassigner binary

    @manishrjain
    manishrjain committed Mar 2, 2016
    Configuration menu
    Copy the full SHA
    2cae836 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2016

  1. Merge branch 'dmuts' of github.com:dgraph-io/dgraph into dmuts

    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    033acc9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dmuts

    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    9b9aa68 View commit details
    Browse the repository at this point in the history

  3. Optimizations based on uidassigner. Also, pick elements from dirtymap… 

    … randomly.
    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    fa631d8 View commit details
    Browse the repository at this point in the history

  4. Change how many keys we merge. Also, run gentlyMerge in a goroutine, … 

    …so our memory monitor can call aggressivelyEvict in time. Also, with this change, the memory growth is a heck of a lot more controlled.
    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    2678fd4 View commit details
    Browse the repository at this point in the history

  5. Decrease memory allocations or buffer growth.

    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    9982d13 View commit details
    Browse the repository at this point in the history

  6. Loader shouldn't cause uid assignment. Add a Get() function in uid, a… 

    …nd avoid calling rdf.GetUid. Plus, other changes across the entire code base to accommodate this.
    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    4e6355f View commit details
    Browse the repository at this point in the history

  7. Store any errors encountered while running the pipeline. If any error… 

    … is found, stop processing immediately. Switch the system to sync.WaitGroup completely, instead of an error channel. Log fatal on error in loader and uidassigner
    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    93e35d1 View commit details
    Browse the repository at this point in the history

  8. Keep the number of goroutines running gentlyMerge function in check.

    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    15f13e5 View commit details
    Browse the repository at this point in the history

  9. It's faster to let the aggressiveMerge run, than to run gentle merge.

    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    ed9e6d2 View commit details
    Browse the repository at this point in the history

  10. Set the number of goroutines for gentle merging to 18. It is little s… 

    …lower to run these goroutines, but they keep the memory usage in check.
    @manishrjain
    manishrjain committed Mar 3, 2016
    Configuration menu
    Copy the full SHA
    37d6146 View commit details
    Browse the repository at this point in the history