-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NEW] Schedule workers on consolidated offers #154
[NEW] Schedule workers on consolidated offers #154
Commits on Jul 28, 2016
-
New package for Resource Collection to address the following problems…
… and use it to schedule and assign workers with consolidated offers per node. Adresses the following problems. Problem 1: Types of OfferResources was limited - CPU, MEM and PORTS. Each of them where either RESERVED or UNRESERVED. We treated RESERVED or UNRESERVED resources the same way. So the logic in OfferResources was simple. But we now have three different types of RESERVED, STATICALLY_RESERVED and DYNAMICALLY_RESERVED and we are starting to have special handling on each of the types of resources. One example of such code is https://github.com/mesos/storm/pull/111/files#diff-9e83ab25db3f6d262627383d8aa8f815. Problem 2: Currently, offers related logic is spread across OfferResources and RotatingMap. What we ultimately need is an interface which could be used across various parts of the framework. This commit introduces a 'storm.mesos.resources' package which is the first step in achieving the aforementioned goal. This package enables us to define an interface like 'storm.mesos.resources.Aggregator' and have various aggregators implement 'storm.mesos.resources.Aggregator' thereby abstracting the aggegation logic(including locking) from the rest of the packages.
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 5dcd80c - Browse repository at this point
Copy the full SHA 5dcd80cView commit details -
Trio coding changes for improving allSlotsAvailableForScheduling
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 42e5bc8 - Browse repository at this point
Copy the full SHA 42e5bc8View commit details -
Remove additional logs for testing plus and optimize import statements
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 8c5ac41 - Browse repository at this point
Copy the full SHA 8c5ac41View commit details -
Using java8 specific function String.join causes travis to fail with the
following error while building java7 binary. ``` testGetTasksToLaunchForOneTopologyWithOneOffer(storm.mesos.MesosNimbusTest) Time elapsed: 0.158 sec <<< ERROR! java.lang.NoSuchMethodError: java.lang.String.join(Ljava/lang/CharSequence;Ljava/lang/Iterable;)Ljava/lang/String; ``` Fixing ^^ by using org.apache.commons.lang3.StringUtils.join instead of java.lang.String.join
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for dfade0a - Browse repository at this point
Copy the full SHA dfade0aView commit details -
Addressing First Round of code review comments from Erik Weathers and…
… Jessica Hartog. Still TODO: 1. Refactor MesosNimbus.getTasksToLaunch as per Erik's comment 2. Refactor SchedulerUtils.getPorts as per Jessica's comment
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for d7a0274 - Browse repository at this point
Copy the full SHA d7a0274View commit details -
Addressed the following comments:
1. Refactor MesosNimbus.getTasksToLaunch as per Erik's comment 2. Refactor SchedulerUtils.getPorts as per Jessica's comment
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for aa5ed42 - Browse repository at this point
Copy the full SHA aa5ed42View commit details -
Missed a couple of comments on MesosNimbusTest
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 42e5738 - Browse repository at this point
Copy the full SHA 42e5738View commit details -
Addressing Erik's comments - Second round
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 1392a95 - Browse repository at this point
Copy the full SHA 1392a95View commit details -
Karthick Duraisamy Soundararaj authored and Jessica Hartog committed
Jul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 1d212fd - Browse repository at this point
Copy the full SHA 1d212fdView commit details -
Fixing range addition as per Jessica's comment
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for b1c875d - Browse repository at this point
Copy the full SHA b1c875dView commit details -
Renaming OfferResources -> AggregatedOffers
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for ea55da6 - Browse repository at this point
Copy the full SHA ea55da6View commit details -
Addressing Erik's comment regarding 'avaliable' typo and removing TODO
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for a65ce91 - Browse repository at this point
Copy the full SHA a65ce91View commit details -
make important log about aggregated resources more readable
Also improve readabiltiy to removing unnecessary verbosity of ReservationType enum identifiers, and then defining their `toString()` method to use the lowercase version of their id. These new logs retain the same information, but in half the characters. We can do even better by stripping out unnecessary trailing 0s from the resources. Example of original log: ``` 2016-06-19T07:55:05.924+0000 s.m.u.MesosCommon [INFO] Available resources at workerhostname: cpu : Resource cpus - Total available : 19.500000 Total available by reservation type : [ , [DYNAMICALLY_RESERVED : 0.0, STATICALLY_RESERVED : 0.0, UNRESERVED : 19.5] ], memory: Resource mem - Total available : 88561.000000 Total available by reservation type : [ , [DYNAMICALLY_RESERVED : 0.0, STATICALLY_RESERVED : 0.0, UNRESERVED : 88561.0] ], ports : [31003-31003,31005-32000] ``` Example of new log: ``` 2016-06-19T09:21:43.075+0000 s.m.u.MesosCommon [INFO] Available resources at workerhostname: cpus: 19.500000 (dynamic: 0.0, static: 0.0, unreserved: 19.5), mem: 72402.000000 (dynamic: 0.0, static: 0.0, unreserved: 72402.0), ports: [31000-31001,31006-32000] ```
Configuration menu - View commit details
-
Copy full SHA for 531c46a - Browse repository at this point
Copy the full SHA 531c46aView commit details -
Including worker ports into log lines related to fit and failure to l…
…aunch
Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for bfe032a - Browse repository at this point
Copy the full SHA bfe032aView commit details -
Adding in some comments for clarity
Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for 782e489 - Browse repository at this point
Copy the full SHA 782e489View commit details -
Adding in an assert for combining offers to make sure they're from th…
…e same host
Jessica Hartog committedJul 28, 2016 Configuration menu - View commit details
-
Copy full SHA for b0f8af9 - Browse repository at this point
Copy the full SHA b0f8af9View commit details
Commits on Jul 29, 2016
-
Fix removeAndGet to no longer modify a list we are iterating over
Jessica Hartog committedJul 29, 2016 Configuration menu - View commit details
-
Copy full SHA for b239dd2 - Browse repository at this point
Copy the full SHA b239dd2View commit details -
Scheduling improvements for spreading work across executors:
* avoid rescheduling executors when all workers are already running * throw RuntimeException for nonsensical case where all workers are running but we have unassigned executors * improve variable naming to be more concise without losing fidelity, and thus improve readability * removing a call from schedule, since we're pretty sure we don't need it * fixing the number of slots available for each topology * spread executors by fixing an unnecessary assignment to executors * spread executors by not scaling workers down when there are less slots available than assigned slots
Configuration menu - View commit details
-
Copy full SHA for 69d63b9 - Browse repository at this point
Copy the full SHA 69d63b9View commit details -
Improve few existing comments, add more comments for existing functio…
…ns, delete few unnecessary/irrelevant methods
Karthick Duraisamy Soundararaj authored and Jessica Hartog committedJul 29, 2016 Configuration menu - View commit details
-
Copy full SHA for b5cd2cf - Browse repository at this point
Copy the full SHA b5cd2cfView commit details -
adjusts tests to account for some corner cases that exist for the tim…
…e being 1. Our floating point tabulations in AggregatedOffers can lead to confusing results where a 1.0 becomes 0.999999999996 for example. We add a fudge factor by adding 0.01 more CPU resources in one test to work around this until the following issue is fixed: * mesos#161 2. Since we are disabling the ability of allSlotsAvailableForScheduling to account for supervisors already existing when calculating resource needs, we disable the test that is validating that behavior. That will be reenabled once we fix this issue: * mesos#160 Also a few cosmetic-ish changes: 1. Use more standard hostnames with a domain of "example.org", a standard domain for documentation. See http://example.org 2. Rearrange/renumber some of the offers to prevent confusion.
Configuration menu - View commit details
-
Copy full SHA for 1032e58 - Browse repository at this point
Copy the full SHA 1032e58View commit details